Forum: Ruby on Rails Rails and background tasks/threads

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Cde4ce7f2f5caf468c0a59c308368b9b?d=identicon&s=25 Eric Ching (eching)
on 2006-02-14 21:55
I am just getting into web servers/web applications and rails as well
so bear with me.   I am trying to write a web app that, based on a
users input from the browser, will perform some task, and update the
browser (ala ajax style) as needed and/or provide a way for the user to
control the background task.

Now, I have more experience with java servlets, which makes this easy
enough for a beginner.  The user inputs some info and the server spawns
a new thread to perform the task in the background while the user can
continue using the app, until it is updated with the results.  (in my
case the background task is putting or getting messages on/from an MQ
Series queue)   I keep a reference to the background task in the
session object for access as needed.

What I am wondering is if the same concepts can be applied to a rails
application.  I started researching rails off and on recently and it
looks like this is mostly possible, but the vague area I am wondering
about is spawning a background thread and then maintaining a reference
to it so the user can get the data back and/or manipulate it in various
ways(start/stop/put to a different queue etc)

I saw RailsCron mentioned in another thread, but am not sure that is
exactly what I am looking for (maybe I'm wrong).  Anyone with some
experience with this sort of scenario that can provide some guidance
would be greatly appreciated.  I've asked the question a few times in
various forums with little to no response.

Thanks in advance,
Eric
Ff82af3238a57fbd1212832ec1a19f28?d=identicon&s=25 Dylan Stamat (Guest)
on 2006-02-14 22:32
(Received via mailing list)
link_to_remote ?
6dab365a82517fb694650a57ee88e4a4?d=identicon&s=25 joey__ (Guest)
on 2006-02-14 23:30
You could start a new thread and put it in the session, which they could
then stop or start.

Like:
session[:thread] = Thread.new{process}

Look at http://www.ruby-doc.org/core/classes/Thread.html &
http://www.ruby-doc.org/stdlib/libdoc/thread/rdoc/index.html
Or threads in the Ruby PicAxe book.


Joey
132a94ca65959bda6c74fae54bff2425?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2006-02-14 23:42
(Received via mailing list)
On Feb 14, 2006, at 2:30 PM, joey__ wrote:

>
>
> Joey


You will find that you cannot put a thread in the session. To put an
object in the session it needs to be serializable. This means no
threads, singletons or procs/lambdas can go in the session.

Cheers-
  -Ezra Zygmuntowicz
Yakima Herald-Republic
WebMaster
http://yakimaherald.com
509-577-7732
ezra@yakima-herald.com
Cde4ce7f2f5caf468c0a59c308368b9b?d=identicon&s=25 Eric Ching (eching)
on 2006-02-14 23:53
If it's as simple as that I'll be very happy.

Initially I'd just be using WEBrick (for internal purposes), does that
pose any problems?

For some reason I thought this was not possible, or practical.  I'll
have to try and remember why I thought so.
Cde4ce7f2f5caf468c0a59c308368b9b?d=identicon&s=25 Eric Ching (eching)
on 2006-02-15 01:59
Bummer, didn't see the last post.

So what's a good solution to this problem?  Seems like it would be a
common need, but then I'm kind of green to web programming.

Any suggestions?
4005a47a8f2ceee49670b920593c1d52?d=identicon&s=25 Ben Munat (Guest)
on 2006-02-15 05:17
(Received via mailing list)
Ezra Zygmuntowicz wrote:
> You will find that you cannot put a thread in the session. To put an
> object in the session it needs to be serializable. This means no
> threads, singletons or procs/lambdas can go in the session.

So you know, I've seen this information (and I think I've even told
someone this
information), but it's just now ocurred to me that I should ask if
there's an equivalent
of java's "transient" in ruby? In java, making a variable transient
means that it won't be
serialized.

If Ruby's serialization pulls the whole graph, then there must be some
way to tell it to
skip certain children... or be very careful what you create references
to in your objects.

As for the OP's question though, I would think you could store your
threads or procs in a
hash and put the key in the session.

b
B3260ee62969961010117e21e9872a3a?d=identicon&s=25 Kenneth Lee (Guest)
on 2006-02-15 05:29
(Received via mailing list)
>From my experience, at least when running on Apache, putting work into
a background thread did not give me the results that I wanted.  It
appears that the thread will suspend itself unless the server gets
activity.  I ended up having to use a timer() and Ajax.Request to keep
polling for whether the thread was done to, among other things, keep
the thread running.

I had better luck forking off a heavyweight process to do the
background work.  I've heard that the background processes work better
in LigHTTP.
Ad7805c9fcc1f13efc6ed11251a6c4d2?d=identicon&s=25 Alex Young (Guest)
on 2006-02-15 05:44
(Received via mailing list)
Ben Munat wrote:
> As for the OP's question though, I would think you could store your
> threads or procs in a hash and put the key in the session.

Given that the session *is* a hash, how does that help?  It just won't
work.  Without being able to serialize the threads, what does the hash
key reference?  Bear in mind that the lookup might not happen on the
same machine, let alone in the same process...
8e44c65ac5b896da534ef2440121c953?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2006-02-15 08:00
(Received via mailing list)
On Feb 14, 2006, at 8:40 PM, Alex Young wrote:

>
> Alex
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>


	One way I handle longer running background processes in my rails
apps is to use distributed ruby(drb). Drb is dead simple to use so
its easy to make a drb server that contains the classes needed to do
the long running work. Then you can kick off the long running task to
the drb server from an ajax request. Then you can use a periodically
call remote to poll the drb server for its percent done on the job
you submitted. Once its 100% you can return the results. Doing it
this way divorces the long running task from the rails request/
response cycle all together and makes it easy to add progress bars to
the view for these long tasks. I have this functionality almost in a
state to refactor it into a plugin if any one was interested in it.

Cheers-
-Ezra Zygmuntowicz
WebMaster
Yakima Herald-Republic Newspaper
ezra@yakima-herald.com
509-577-7732
7f71224fad4b2b28809650165bd37836?d=identicon&s=25 Greg Edwards (other box) (Guest)
on 2006-02-15 08:30
(Received via mailing list)
I've been solving this by using Rinda (it's in the pick-ax book and is
the
Ruby version of Linda which is a robust scalable method) so that RoR
posts
messages for things it wants done, and then "worker agents" process the
messages and post back another one giving updates or a "I'm done"
message.
If you investigate Rinda and are interested in taking this approach,
then I
can share some code that will save you a lot of time (the documentation
for
getting started on Rinda is a little sparse).

I'd love to hear how other people are solving the problem.
-Greg
4005a47a8f2ceee49670b920593c1d52?d=identicon&s=25 Ben Munat (Guest)
on 2006-02-15 09:34
(Received via mailing list)
Alex Young wrote:
>
The key can just be a string, so the session is once again serializable.
You keep your
threads in a hash in memory. When the thread is needed again, the code
that handles this
looks for the appropriate session value to get it's key, and then uses
that to get the
thread from the hash.

It's basically just keeping the non-serializable stuff in your own hash
that won't be
serialized.

Hmm, although, I guess this wouldn't work without a daemon process in
which to keep the
hash... there is a way to do that in rails, right? Well, still... one
could start a
separate ruby process for this storage... would need to hook into that
from the rails code
somehow.

b
4005a47a8f2ceee49670b920593c1d52?d=identicon&s=25 Ben Munat (Guest)
on 2006-02-15 09:43
(Received via mailing list)
Ezra Zygmuntowicz wrote:
> refactor it into a plugin if any one was interested in it.
>

Ezra, you're just a fount of useful information! I should have read this
before I replied
to Alex. And I would defintely like to hear about that plugin!

b

PS: I reaaaallllly think it's important that mongrel (or something like
it) turn into a
servlet-container-style daemonized process into which one can hook and
unhook apps while
running, spin off background threads, pool db connections, etc. I don't
think the java
world will take rails seriously without that. Then again, maybe drb can
grow into that, or
hook into mongrel, etc. etc.
4005a47a8f2ceee49670b920593c1d52?d=identicon&s=25 Ben Munat (Guest)
on 2006-02-15 09:43
(Received via mailing list)
Greg Edwards (other box) wrote:
>
It'd be cool to see that code.... of course, I should probably learn
some more ruby and
get my rails training app into a respectable state first... but this
would be more fun!

b
Ad7805c9fcc1f13efc6ed11251a6c4d2?d=identicon&s=25 Alex Young (Guest)
on 2006-02-15 10:13
(Received via mailing list)
Ben Munat wrote:
>> same machine, let alone in the same process...
>>
>
> The key can just be a string, so the session is once again serializable.
> You keep your threads in a hash in memory. When the thread is needed
> again, the code that handles this looks for the appropriate session
> value to get it's key, and then uses that to get the thread from the hash.
>
The session's serializable, but that's got you nothing because when you
unserialize it on the next request, what your key points to may well not
exist any more, or exist on another machine, or just generally not be
visible.

> It's basically just keeping the non-serializable stuff in your own hash
> that won't be serialized.
If that worked, you wouldn't need a session *at all*, you could just
keep everything in memory all the time.

> Hmm, although, I guess this wouldn't work without a daemon process in
> which to keep the hash... there is a way to do that in rails, right?
> Well, still... one could start a separate ruby process for this
> storage... would need to hook into that from the rails code somehow.
Yay!  DRb!
Ad7805c9fcc1f13efc6ed11251a6c4d2?d=identicon&s=25 Alex Young (Guest)
on 2006-02-15 10:19
(Received via mailing list)
Ezra Zygmuntowicz wrote:
> into a plugin if any one was interested in it.
I'd love to see that.  I'm bodging that approach together piecewise at
the moment, and a plugin would make it *much* simpler.  Rather than
long-running tasks, though, I'm actually talking to permanent daemons
that happen to live in the same object space, so the concept of 'percent
done' doesn't really apply.

I've spotted where I would pluginize this, but I haven't had the impetus
to fully work it through.  If you're closer to it than I am, please,
please release!  :-)
E3c79c779c0b390049289cdfe7cb9705?d=identicon&s=25 Bob Hutchison (Guest)
on 2006-02-15 14:36
(Received via mailing list)
On Feb 15, 2006, at 2:29 AM, Greg Edwards (other box) wrote:

> can share some code that will save you a lot of time (the
> documentation for
> getting started on Rinda is a little sparse).

I'd thank you if you did share. I'd like to use Rinda but keep
putting it off until I have more time (as if that'll ever happen).

If you have a reliable way of starting/restarting the Rinda servers,
or any ideas, I think that'd be very well received too :-)

Cheers,
Bob

>
> I'd love to hear how other people are solving the problem.
> -Greg

----
Bob Hutchison                  -- blogs at <http://www.recursive.ca/
hutch/>
Recursive Design Inc.          -- <http://www.recursive.ca/>
Raconteur                      -- <http://www.raconteur.info/>
xampl for Ruby                 -- <http://rubyforge.org/projects/xampl/>
Cde4ce7f2f5caf468c0a59c308368b9b?d=identicon&s=25 Eric Ching (eching)
on 2006-02-15 15:32
Wow, thanks for all the great responses.  I think I need to plug away at
some more Rails and give the sugestions a try (rinda/drb).  Any example
would be greatly appreciated.

I want to use Rails because there is so much about it I do like, but
without being able to tightly control background threads/tasks I really
cannot jump into it 100% - that's the make or break for me.
8e44c65ac5b896da534ef2440121c953?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2006-02-15 17:12
(Received via mailing list)
Greg-

	I would love to see your rinda code. I have a great grasp of drb and
have been using it heavily but I haven't really dug into rinda yet
and the docs are sparse. So please do share.

Thanks
-Ezra

On Feb 14, 2006, at 11:29 PM, Greg Edwards (other box) wrote:

> can share some code that will save you a lot of time (the
> [mailto:rails-bounces@lists.rubyonrails.org] On Behalf Of Eric Ching
>
> application.  I started researching rails off and on recently and it
> various forums with little to no response.
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>

-Ezra Zygmuntowicz
WebMaster
Yakima Herald-Republic Newspaper
ezra@yakima-herald.com
509-577-7732
4bd34a2216dc8bdbf1f017f64e4d59e8?d=identicon&s=25 Kyle Maxwell (Guest)
on 2006-02-15 18:04
(Received via mailing list)
It sounds like RailsCron might be a good fit, as it's a good way to
run rails background tasks in general.  You might be able to get away
with a simple daemonized script/runner.

Assuming that you have figured out the best way to have a background
task running satisfactorily, I would write it as a method that traps a
signal, and whenever that signal is sent, it looks in the database (or
a flat file) to see what has changed.  Alternately, you could poll the
database.

-Kyle
4005a47a8f2ceee49670b920593c1d52?d=identicon&s=25 Ben Munat (Guest)
on 2006-02-15 18:41
(Received via mailing list)
Alex Young wrote:
>> It's basically just keeping the non-serializable stuff in your own
>> hash that won't be serialized.
>
> If that worked, you wouldn't need a session *at all*, you could just
> keep everything in memory all the time.

Well, yeah, but all the other crap that you can serialize you might as
well have in a
session and let it get serialized. This whole thread was more or less a
thought experiment
anyway. I don't personally currently have a need to stick threads,
procs, or singletons in
the session... but I expect I might some day. I'm just working through
ruby's rough spots
mentally...

Ultimately, my point is that ruby/rails needs a way to keep
application-wide data in
memory. This is very useful in the Java world. In fact, I'd say that a
big chunk of the
reason why Java did so well in the web world was that they went straight
for the container
model. Plopping a multi-threaded, long-running server process into a
world of
start-process-stop CGI insanity made people go "whoa... cool".

b
132a94ca65959bda6c74fae54bff2425?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2006-02-15 21:28
(Received via mailing list)
On Feb 15, 2006, at 6:32 AM, Eric Ching wrote:

> Wow, thanks for all the great responses.  I think I need to plug
> away at
> some more Rails and give the sugestions a try (rinda/drb).  Any
> example
> would be greatly appreciated.
>
> I want to use Rails because there is so much about it I do like, but
> without being able to tightly control background threads/tasks I
> really
> cannot jump into it 100% - that's the make or break for me.


Eric-

	Here is a simple drb server/client for you to play with. This is an
example of exposing an ActiveRecord model over drb. You can actually
publish any object you like over drb. this example is a bit contrived
but it can show you how drb works a bit. Run the server in a terminal
and then open irb and paste in the client part and play with that. I
am going to work on getting my drb+ajax progress bars plugin into
releaseable shape this weekend so stay tuned.

#!/usr/local/bin/ruby
#server
require 'rubygems'
require 'drb/drb'
require_gem 'activerecord'

ActiveRecord::Base.establish_connection(
     :adapter  => "mysql",
     :username => "root",
     :host     => "localhost",
     :password => "xxxxxxx",
     :database => "remote_drb"
)

ActiveRecord::Schema.define(:version => 1) do
   create_table "remote_records", :force => true do |t|
     t.column :title, :string
     t.column :desc, :string
   end
end

class RemoteRecord < ActiveRecord::Base
    include DRb::DRbUndumped
end

DRb.start_service("druby://127.0.0.1:3500", RemoteRecord)
puts DRb.uri
DRb.thread.join

__END__
#client run in irb
require 'rubygems'
require 'drb/drb'
require_gem 'activerecord'

DRb.start_service
RemoteRecord = DRbObject.new(nil, 'druby://127.0.0.1:3500')
[
   ['foo','bar'],
   ['qux', 'baz'],
   ['nik', 'nuk']
].each do |record|
    RemoteRecord.create :title => record[0], :desc => record[1]
end

test = RemoteRecord.find :all
test.each{|o| p "#{o.title} : #{o.desc}" }

# results in:
#  "foo : bar"
#  "qux : baz"
#  "nik : nuk"



Cheers-

-Ezra Zygmuntowicz
Yakima Herald-Republic
WebMaster
http://yakimaherald.com
509-577-7732
ezra@yakima-herald.com
7f71224fad4b2b28809650165bd37836?d=identicon&s=25 Greg Edwards (other box) (Guest)
on 2006-02-16 01:46
(Received via mailing list)
Ezra (and everyone else :-)

Ok, here's how I am using Linda (cleaned up as best I can to make things
more clear). Basically, put all of these files into one directory to get
started, and then after things are working then you can start moving
different pieces to different servers.

First of all, the background that you should read (or, the background
that I
read that was helpful):
http://en.wikipedia.org/wiki/Linda_%28coordination... (good
background on Linda (and why you'd want to use it), which Rinda
implements)
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/...
http://segment7.net/projects/ruby/drb/rinda/ringserver.html
http://www.ruby-doc.org/stdlib/libdoc/rinda/rdoc/c...
ml
http://www.ntecs.de/blog/Tech/ComputerScience

and, of course, the pickax has a page on Rinda.

And finally, my code (not very well documented, since it assumes that
you've
read all of the other URLs I pasted above and that now you just want an
example of how it works all together as well as an example of several
things
that I couldn't really find easily in the documentation).

(there are 4 files below)

--- the port where everyone talks to each other needs to be defined ---
In file, rinda_port_uri.rb
--------------------------
RINDA_PORT_URI = "druby://your.hostname.com:12345"
	#(you can also use "localhost" if you are all running on the same
machine, but as soon as you put ANY process on a different server, then
they
have to all have the exact same druby:// line (as far as I can tell...
:-)
#ruby -rsocket -e 'p Socket.gethostname'
	# this command was helpful for me to find out what ruby thinks my
server was named.


File: rinda_blackboard.rb  # this is the main "server" that enables
agents
to get/send messages for things they want someone to do.
-------------------------
require 'drb/drb'
require 'drb/acl'
require 'rinda/tuplespace'
require 'rinda_port_uri' # defines the constant used for the Port

acl = ACL.new(%w(deny all    # block out other servers from accessing
your
Linda blackboard
								 allow
10.0.0.7
								 allow
10.0.0.4
								 allow
60.59.145.219  #the documentation says that you can put wildcards
'60.59.145.*' but that didn't work for me...)
								 allow
60.59.145.220
								 allow
localhost))

DRb.install_acl(acl)

#ruby -rsocket -e 'p Socket.gethostname'
DRb.start_service(RINDA_PORT_URI, Rinda::TupleSpace.new(10))
#puts DRb.uri
#(0..5).each {|x|  # this block will start up 5 agents
#	puts "Starting agent #{x}"
#	IO.popen("ruby rinda_agent.rb")
#}
puts "blackboard is open on #{RINDA_PORT_URI}"
DRb.thread.join
#puts '[return] to exit'
#gets



File: rinda_agent.rb  # this is where you do the work... you can have
multiple agents running, and on different servers all pointing to the
same
rinda_blackboard
--------------------
# this is a "example wrapper" thing... If I want an agent to do some
work
# for me, then I (personally) want to pass:
# - a message,
# - a hash containing variables and stuff
# - a reference ID that uniquely identifies this message request
# - the "return message" I will be looking for that I'd like the agent
to
#   reply using (so that I know what to look for)
# - a return hash containing results from the agent
# (that's just me, what I want...)

require 'drb/drb'
require 'rinda/tuplespace'
require 'rinda_port_uri' # defines the constant used for the Port
require 'json/lexer'
require 'json/objects'
	# I like using JSON to tranfer info between agents, but you can use
a simple hash if you wish.

DRb.start_service
ts = Rinda::TupleSpaceProxy.new(DRbObject.new(nil,RINDA_PORT_URI))

class Gen

	def worker1(var_in, ref_id, msg_out)
		... do something here
		an_amazing_result = ...
		#return the results of the work
		[msg_out,an_amazing_result]
	end
end

loop do
	puts
	puts
	puts
	puts
	puts "waiting...(for PPT stuff)"

	# Read the next command
	op, var_in, ref_id,msg_out,var_out = ts.take(

[%r{^(msg_1_I_listen_for|msg_2_I_listen_for)$},nil,nil,nil,nil],55) #
time
out after 55 seconds if no activity... good for debugging initially (can
be
omitted in production)

	puts "Received! #{op} #{ref_id} #{var_in}"

	gen = Gen.new
	msg_out,var_out = gen.send(op, var_in, ref_id)

#--- DONE --------------------
	ts.write([msg_out,ref_id, var_out]) unless (msg_out.nil? or
msg_out.length == 0)
	puts "done! #{msg_out} #{ref_id} #{var_out}"
end


file: rinda_watcher.rb
-----------------------
#finally, once you've got your blackboard where you can post messages to
get
work done, I wanted to be able to SEE what the heck was being put on
there
:-) so this file is a easy to implement, hacky, fast way of doing
that...

require 'drb/drb'
require 'rinda/tuplespace'
require 'rinda_port_uri' # defines the constant used for the Port
require 'pp'

DRb.start_service
puts "trying to connect to #{RINDA_PORT_URI}..."
ts = Rinda::TupleSpaceProxy.new(DRbObject.new(nil,RINDA_PORT_URI))
puts "---------------------Watching"

option = 1
case option
	when 1
		result = ts.read_all([nil,nil])
		pp result unless result.nil?
		result = ts.read_all([nil,nil,nil])
		pp result unless result.nil?
		result = ts.read_all([nil,nil,nil,nil])
		pp result unless result.nil?
		result = ts.read_all([nil,nil,nil,nil,nil])
		pp result unless result.nil?
		result = ts.read_all([nil,nil,nil,nil,nil,nil])
		pp result unless result.nil?
		puts
	when 2
			result =
ts.read_all([%r{prepare_to_generate_ppt_.*},
				nil, #session[:username],
				nil, #@current_ppt_name,
				nil, #@new_ppt_name,
				nil, #@path_to_template,
				nil, #json_msg
				nil, #user password
				])

			pp result unless result.nil?
	when 3
			0.upto(10) {|ignore|
				hit = 0
				result = ts.read_all([nil,nil])
				unless result.nil? or result.size ==0
					pp result
					ts.take([nil,nil])
					hit +=1
				end

				result = ts.read_all([nil,nil,nil])
				unless result.nil? or result.size ==0
					pp result
					ts.take([nil,nil,nil])
					hit +=1
				end

				result = ts.read_all([nil,nil,nil,nil])
				unless result.nil? or result.size ==0
					pp result
					ts.take([nil,nil,nil,nil])
					hit +=1
				end

				result = ts.read_all([nil,nil,nil,nil,nil])
				unless result.nil? or result.size ==0
					pp result
					ts.take([nil,nil,nil,nil,nil])
					hit +=1
				end

				result =
ts.read_all([nil,nil,nil,nil,nil,nil])
				unless result.nil? or result.size ==0
					pp result
					ts.take([nil,nil,nil,nil,nil,nil])
					hit +=1
				end

				result =
ts.read_all([nil,nil,nil,nil,nil,nil,nil])
				unless result.nil? or result.size ==0
					pp result

ts.take([nil,nil,nil,nil,nil,nil,nil])
					hit +=1
				end

				if hit > 0
					puts '----------------------------'
					puts
				end
			}

End


-Greg

Greg.railslist@eyetools.com
916 792 4538
4005a47a8f2ceee49670b920593c1d52?d=identicon&s=25 Ben Munat (Guest)
on 2006-02-16 07:57
(Received via mailing list)
Sweet... moving this to my "saved" folder!

b
Db303dc84d03a992b33cd3ac978f89ae?d=identicon&s=25 Benjamin Curtis (Guest)
on 2006-02-16 18:08
(Received via mailing list)
Add me to the list of people eagerly awaiting the arrival of this
plugin. :)
Cde4ce7f2f5caf468c0a59c308368b9b?d=identicon&s=25 Eric Ching (eching)
on 2006-02-16 23:23
Thanks for the code samples, much appreciated. :-)
This topic is locked and can not be replied to.