Queuing work in a web app (sinatra/webrick)

Ruby
1

Hi

I want to make a webapp that allows users to queue bit of work
(converting video for example). They are also able to see the queue at
any time and check the progress (queued|processing|done) and are able to
REORDER items in the queue.
The queue can be stopped (ie. paused) and resumed in between conversions
of course.

I’m using sintatra and the LoopingThread example from here
http://codeidol.com/other/rubyckbk/Multitasking-and-Multithreading/Terminating-a-Thread/

Im sure there’s ppl out there who’ve done something similar. I can do
this in java no problem (synchronized lists etc) but not sure how to do
this in ruby as im sort of a newbie.

I have a working bit of code but im sure im missing something (memory
leak, synchorizing issues etc). Can post more code (very small) if
someone’s willing to help.

Thanks!

2

Gurpal 2000 wrote:

Hi

I want to make a webapp that allows users to queue bit of work
(converting video for example). They are also able to see the queue at
any time and check the progress (queued|processing|done) and are able to
REORDER items in the queue.
The queue can be stopped (ie. paused) and resumed in between conversions
of course.

I’m using sintatra and the LoopingThread example from here
http://codeidol.com/other/rubyckbk/Multitasking-and-Multithreading/Terminating-a-Thread/

Im sure there’s ppl out there who’ve done something similar. I can do
this in java no problem (synchronized lists etc) but not sure how to do
this in ruby as im sort of a newbie.

I have a working bit of code but im sure im missing something (memory
leak, synchorizing issues etc). Can post more code (very small) if
someone’s willing to help.

Thanks!

Oh and i forgot to mention that the queue can be persisted if u want - i
dont care at this stage. But im essentially keeping the queue in a flat
file on disk to hopefully get some ‘cheap locking’. I’d like to do this
without a DB so to speak.

thx

3

snip…

OK i altered my code to use Sequel.sqlite. So now i have my “list” in
memory. So now the queue implementation is the issue. thx

4

On Thu, Nov 12, 2009 at 3:46 PM, Gurpal 2000 [email protected] wrote:

OK i altered my code to use Sequel.sqlite. So now i have my “list” in
memory. So now the queue implementation is the issue. thx

Are you wanting to farm out work among a distributed network of agents?
Nanite + RabbitMQ comes to mind:

5

Tony A. wrote:

On Thu, Nov 12, 2009 at 3:46 PM, Gurpal 2000 [email protected] wrote:

OK i altered my code to use Sequel.sqlite. So now i have my “list” in
memory. So now the queue implementation is the issue. thx

Are you wanting to farm out work among a distributed network of agents?
Nanite + RabbitMQ comes to mind:

GitHub - ezmobius/nanite: self assembling fabric of ruby daemons

not really farm out, i’ll check this out but mq sounds over the top for
what’s essentially a small home project (for now)

6

On Thu, Nov 12, 2009 at 10:45 PM, Gurpal 2000 [email protected] wrote:

I’m using sintatra and the LoopingThread example from here

Thanks!

Oh and i forgot to mention that the queue can be persisted if u want - i
dont care at this stage. But im essentially keeping the queue in a flat
file on disk to hopefully get some ‘cheap locking’. I’d like to do this
without a DB so to speak.

There are numerous solutions for this kind of task: BackgroundJob,
DelayedJob,
Workling etc. The list is quite extensive.

My personal opinion is that spinning off threads to do this work is not
the
way to
go about it. To my mind this is an activity that should be spun off in a
separate
process.

While you want to avoid using the DB for this, I think you should
reconsider:
DB backed scheduling gets your locking and monitoring via
transactions/persistence and
it doesn’t have to be difficult either. I would look for a background
scheduling tool
that implicitly uses your DB so that you don’t get your hands dirty.

7

Hi,

I would guess your culprit for the spike to 100% CPU usage may be your
infinite loop:

 until @stopped
   yield
   Thread.stop if @paused
 end

Look at using sleep() if you want to decrease CPU usage. Even sleeping
for one second should decrease your CPU usage dramatically in my
experience(maybe with a rare spike).

8

Robert G. wrote:

Hi,

I would guess your culprit for the spike to 100% CPU usage may be your
infinite loop:

 until @stopped
   yield
   Thread.stop if @paused
 end

Look at using sleep() if you want to decrease CPU usage. Even sleeping
for one second should decrease your CPU usage dramatically in my
experience(maybe with a rare spike).

Sure, i got this code from the Oreilly Ruby Cookbook. Is it bad code
then?
Again this doesn’t happen when i use similar code in Java…

9

OK i managed to use Sequel DB. But the backgroundjob api etc seems
overkill. This LoopingThread works below, but i noticed 100% usgae on
ruby 1.8.6. Is this normal?

class LoopingThread < Thread
def initialize
@stopped = false
@paused = false
super do
before_loop
until @stopped
yield
Thread.stop if @paused
end
after_loop
end
end

def before_loop
end

def after_loop
end

def stop
@stopped = true
end

def paused=(paused)
@paused = paused
run if !paused
end
end

10

Sure, i got this code from the Oreilly Ruby Cookbook. Is it bad code
then?
The book seems to be a little outdated - for example, i rarely see
people defining setters themselves nowadays, especially when you can use
attr_writer :pause . I’m not sure why they didn’t pick up on what could
be an infinite loop, or at least a loop that can repeat itself many,
many times.

Again this doesn’t happen when i use similar code in Java…
I don’t know why that is in Java - in C, Ruby, an infinite loop is going
to consume a lot of CPU power unless you take precautions.