Forum: Ruby Real-time image processing in Ruby

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
944f59a432ec8f148eb04767389285c5?d=identicon&s=25 John Koschwanez (johnk)
on 2005-12-29 00:08
I'm a Ruby newbie - "Programming Ruby" was great Xmas break reading!
I write image analysis and processing code to analyze single cells under
a microscope in real-time and to automate lab equipment.  I now write
solely in C++ for PC, but I would love to use Ruby to quickly try new
ideas without plowing through C++ code.  I have read through the forums
and I see Ara Howard is using C extensions to process images - I'll try
the same thing.  I have some questions for the group:
- I capture images from a camera, process and analyze them, and display
them at ~10 fps.  Ruby-v4l (capture) is for Linux only.  Tk is probably
too slow, and someone recommended using OpenGL for display.  Has anyone
had success with image capture or display using Ruby?  What worked?  Any
advice?
- Is anyone using pure Ruby to process images real-time?  RMagick is too
limited for my applications.  I doubt Ruby would match the speed of C,
but it might be fun to try.
Thanks, John
E20e89d58211a3631842daecc1245de2?d=identicon&s=25 Ilmari Heikkinen (Guest)
on 2005-12-29 01:13
(Received via mailing list)
On 12/29/05, John Koschwanez <ishkaprog@gmail.com> wrote:
> had success with image capture or display using Ruby?  What worked?  Any
> advice?

I wrote a small slideshow program with Ruby, imlib2, and OpenGL. With
images around 800x600 in size, it achieves ~20 images per second on my
Athlon XP 1733MHz (read jpeg, decode jpeg into gl texture format, send
the texture to the vidcard, draw quad with texture on screen, swap
frame.) If you have a ready stream of decoded image data coming, I'd
guess the main limiting factor would be RAM->vidcard bus speed.

Don't know about Tk, but would find it sort of strange if it couldn't
display 10fps video. Anyone tried?

> - Is anyone using pure Ruby to process images real-time?  RMagick is too
> limited for my applications.  I doubt Ruby would match the speed of C,
> but it might be fun to try.

Correct me if I'm wrong, but image processing is one of those things
that plays directly into the weaknesses of the current Ruby
implementation. Lots of iteration, lots of method calls -> very slow
in pure Ruby.  It's okay for prototyping algorithms on small images,
though. Now, if there was something like Regexps, but for image
editing...

> Thanks, John
>

Hope this helps,
Ilmari
D36eff3004b39abc4b93fe8a410d8bd3?d=identicon&s=25 Ron M (Guest)
on 2005-12-29 02:07
(Received via mailing list)
Ilmari Heikkinen wrote:
> On 12/29/05, John Koschwanez <ishkaprog@gmail.com> wrote:
  >
>>- Is anyone using pure Ruby to process images real-time?  RMagick is too
>>limited for my applications.  I doubt Ruby would match the speed of C,
>>but it might be fun to try.
>
> Correct me if I'm wrong, but image processing is one of those things
> that plays directly into the weaknesses of the current Ruby
> implementation.

I disagree.   Image processing is an area where even in Java/C#/C/etc
you're probably better off using highly tuned matrix libraries than
rewriting such basic operations from scratch; and Ruby has a pretty
good matrix library (NArray).

And under the assumption that regardless of the language most of
the CPU time will be used inside such a matrix library, the
overhead of the language wrapping it shouldn't matter much.



If you do find yourself writing ruby loops to touch large sets
of individual pixels, however, you should probably consider
extending NArray in C instead - hopefully with some pretty
general purpose function so many ruby algorithms could make
use of the same extensions.  (for example, I'm not sure if
NArray has a nice way of specifying a mask to only process
part of an image).

- R
Eb9493c94d8db9887e5f15284d2c767f?d=identicon&s=25 unknown (Guest)
on 2005-12-29 02:55
(Received via mailing list)
In article <2a14ed641166a177977c5f6a33bdc7b5@ruby-forum.com>,
John Koschwanez  <ishkaprog@gmail.com> wrote:
>had success with image capture or display using Ruby?  What worked?  Any
>advice?
>- Is anyone using pure Ruby to process images real-time?  RMagick is too
>limited for my applications.  I doubt Ruby would match the speed of C,
>but it might be fun to try.

Ruby will probably be too slow for your needs (I doubt you'll be able to
make even 10fps unless you write a lot of your code in a C extension).
What
if you take a C or C++ image processing lib (like CImg) and wrap it for
use
from Ruby?  Well, CImg might be a tough one as it relies (too much) on
macros
- that could make it tough to wrap, I suspect.

Another option might be to use Ruby as a code generator.  I was working
on a
project last summer where we were using a Matrox card for image
capture/processing.  We
were using the Matrox image processing API which is a bit of a pain to
work
with (you end up having to violate DRY all the time the way it's set
up).  So
what I started doing was to create a DSL (domain specific langauge)
which
could generate the C code using the Matrox image processing API.  I
didn't
finish it, but I think the appraoch had potential.  Basically, you could
write
a bit of Ruby code that would then generate a whole lot more C code
(something
like 4 to 8X the number of lines; and you didnt' need to violate the DRY
principle).  So the description of what you wanted to do was in fairly
compact, easier to understand Ruby code which generated the C code and
compiled it.  While Ruby itself isn't fast enough to do realtime image
processing, you really only need to describe how you want the image
processed
and then generate C code which runs many, many times faster.  And you
wouldn't
even need to write any C code (after you've worked out the code
generation
details, that is).

I think image processing is very  amenable to this sort of highlevel
description, because most image processing is just a matter of a series
of
transfromations:

Captured Image -> transform A -> transform B ... transform N -> output
image.

In our case what we needed was to be able to easily plug in different
transforms to find out which one(s) would give us the most desireable
results.
Perhaps in your case you already know what transforms you need and what
order
they should be in, if that's the case you may not find much benefit from
this
approach.

Phil
3bb23e7770680ea44a2d79e6d10daaed?d=identicon&s=25 M. Edward (Ed) Borasky (Guest)
on 2005-12-29 03:10
(Received via mailing list)
Have a look at the Perl Data Language (PDL). This was designed to do
high-performance image processing with a Perl user interface. I suspect
the underlying C code could be interfaced to Ruby as easily as it was to
Perl. If you're going to do serious number crunching of any kind in
Ruby, you probably want to use C code to do the work.

Ron M wrote:

>>
>
> use of the same extensions.  (for example, I'm not sure if
> NArray has a nice way of specifying a mask to only process
> part of an image).
>
> - R
>
>
>

--
M. Edward (Ed) Borasky

http://linuxcapacityplanning.com
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2005-12-29 03:13
(Received via mailing list)
On Thu, 29 Dec 2005, John Koschwanez wrote:

> had success with image capture or display using Ruby?  What worked?  Any
> advice?
> - Is anyone using pure Ruby to process images real-time?  RMagick is too
> limited for my applications.  I doubt Ruby would match the speed of C,
> but it might be fun to try.
> Thanks, John



i'm not doing anything near-realtime, but you may want to check this
out:

   http://artengine.ca/gridflow/

my understanding is that it's used for realtime video processing, so
images
might be possible too.

i must say, however, that it seems like doing what you are doing,
especially
in linux only, might be possible with rmagick only - just dump the
frames in
/dev/shm and go...

another option, quite suitable for prototyping only, is just using
narray : i
use it together with a hack that allows the grids to be mmap'd
quantities so
'image processing' is nothing but matrix ops... narray ships with a
simple
method of displaying images under x.  at 10 fps this might just hack it.

regards.


-a
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2005-12-29 03:19
(Received via mailing list)
On Thu, 29 Dec 2005, Ron M wrote:

> rewriting such basic operations from scratch; and Ruby has a pretty
> good matrix library (NArray).
>
> And under the assumption that regardless of the language most of
> the CPU time will be used inside such a matrix library, the
> overhead of the language wrapping it shouldn't matter much.

absolutely.  unless you start looping... this is the tradeoff for
high-level
languages.

>
> If you do find yourself writing ruby loops to touch large sets
> of individual pixels, however, you should probably consider
> extending NArray in C instead - hopefully with some pretty
> general purpose function so many ruby algorithms could make
> use of the same extensions.  (for example, I'm not sure if
> NArray has a nice way of specifying a mask to only process
> part of an image).

and a combination of NArray and RubyInline and/or ruby dl would make for
very
rapid prototyping indeed.

-a
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2005-12-29 03:35
(Received via mailing list)
On Thu, 29 Dec 2005, John Koschwanez wrote:

> - Is anyone using pure Ruby to process images real-time?  RMagick is too
> limited for my applications.

i missed this the first time through...

this is totally uncharted territory, but see

   http://raa.ruby-lang.org/project/ruby-vtk/
   http://public.kitware.com/VTK/

i'm guessing you need something like that if rmagick is too limited...
vtk
certainly has tons of algorithms... what beast to compile though!

cheers.

-a
944f59a432ec8f148eb04767389285c5?d=identicon&s=25 John Koschwanez (johnk)
on 2005-12-29 18:11
>> If you do find yourself writing ruby loops to touch large sets
>> of individual pixels, however, you should probably consider
>> extending NArray in C instead - hopefully with some pretty
>> general purpose function so many ruby algorithms could make
>> use of the same extensions.  (for example, I'm not sure if
>> NArray has a nice way of specifying a mask to only process
>> part of an image).
>
> and a combination of NArray and RubyInline and/or ruby dl would make for
> very
> rapid prototyping indeed.
>
> -a

Thanks - I'll try NArray with RubyInline.  I was about to complain that
something so useful is not included in the stdlib, but I then saw the
recent thread covering that subject.
Someday...RubyLab!
4913bcb685792f601ae9813b4713abb9?d=identicon&s=25 Detlef Reichl (Guest)
on 2005-12-29 20:24
(Received via mailing list)
Am Donnerstag, den 29.12.2005, 08:08 +0900 schrieb John Koschwanez:
> - I capture images from a camera, process and analyze them, and display
> them at ~10 fps.  Ruby-v4l (capture) is for Linux only.  Tk is probably
> too slow, and someone recommended using OpenGL for display.  Has anyone
> had success with image capture or display using Ruby?  What worked?  Any
> advice?

Hi,

cause ruby-v4l was to limited for me i've started to wirte Wc, a webcam
library (I think you can use it also with other camseras, but i didn't
tested it). For now i've tested it with six different cams (and kernel
modules) and v4l (version 1). At this moment i'm on add 4vl2 support and
will release a preview in the next days. If *bsd has something like v4l
(i don't know), suport for this will also be added some day... Support
for windoze will not come.

Simple image prossesing linke rotation, fliping, croping, mixing, ... is
provided. Nearly the complete library is written in C and to add other
functions is very easy.

For displaying i use the ruby bindings of Gtk+. The overhead of the
grapics library for the output is very low. On my 1.23 MHz Mac Mini a
viewer at ~10 fps generates ~20-40% system load. Higher frame rates are
not possible for me, cause i have connected several cams at once and
they have to share the bandwidth (and it is not possible to use USB2 on
an external hub, due of the limitions of the usb-implimentation of
linux).

You can find it at: http://dr.dhis.org/webcam/index.rbx
With the package you will get some examples like the camera viewer.

i'm open for sugestions to extend the library.

Cheers
detlef
This topic is locked and can not be replied to.