Native gem roundup!

Ruby
1

I’m curious what native gems/extensions people are typically using. In
general it seems like most native extensions fall into two categories:

  • They are wrappers around a C API/library, as in zlib, rmagick,
    nokogiri
  • They are written for performance reasons, to implement a particular
    algorithm in a native language or call a library for the same reasons

And there’s a lot of grey area, with some extensions falling in both
categories.

Wrappers can now largely be handled by FFI, and I hope more and more of
them will use FFI as needed to access those libraries. But I’m concerned
about extensions written for performance, since Ruby 1.9 and JRuby do
almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby’s dependence on extensions for
things FFI or “faster Ruby” could do, since it will improve the future
for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or
why do they exist?

  • Charlie
2

So, what native gems or extensions do you use? Why do you use them or
why do they exist?

I use mysql, mysqlplus, eventmachine, ruby debug, hitimes, win32-api,
ruby prof, mongrel, and ferret. Most wrappers to C libs [which is for
speed].
Cheers!
-=r

3

Charles Oliver N. writes:

I’m curious what native gems/extensions people are typically using.
In
general it seems like most native extensions fall into two
categories:

  • They are wrappers around a C API/library, as in zlib, rmagick,
    nokogiri
  • They are written for performance reasons, to implement a particular
    algorithm in a native language or call a library for the same reasons

And there’s a lot of grey area, with some extensions falling in both
categories.

Wrappers can now largely be handled by FFI, and I hope more and more
of
them will use FFI as needed to access those libraries. But I’m
concerned
about extensions written for performance, since Ruby 1.9 and JRuby do
almost as much to speed Ruby up.

Ultimately, my quest is to eliminate Ruby’s dependence on extensions
for
things FFI or “faster Ruby” could do, since it will improve the
future
for both the standard and alternative implementations.

So, what native gems or extensions do you use? Why do you use them or
why do they exist?

  • Charlie

For me, they would be FastCGI and ruby-postgres, two more examples of
wrapping a C library. (Technically, I currently run FastCGI as
installed — in ‘site_ruby’ — rather than a gem; there was always a
hiccup if I tried to install it as a gem, so I went with the quickest
way to get it working.)

As you may guess, this is a Rails application. FastCGI is used for
Apache deployment; ruby-postgres is for accessing the PostgreSQL
database. I know there are other options for Apache/web deployment
(e.g. Mongrel), but this install is stable and other priorities take
precedence over investigating alternative deployment configurations.
For PostgreSQL, even if I moved to ruby-pg, that is still a C library
wrapper.

Anyways, just trying to answer your query.

4

Charles Oliver N. wrote:

I’m curious what native gems/extensions people are typically using. In
general it seems like most native extensions fall into two categories:

  • They are wrappers around a C API/library, as in zlib, rmagick,
    nokogiri
  • They are written for performance reasons, to implement a particular
    algorithm in a native language or call a library for the same reasons
  • They interact directly with the Ruby interpreter (e.g. rcov,
    ruby-debug)

Here’s the set on my home machine:

$ cd /usr/local/lib/ruby/gems/1.8/gems/; find . -name ‘*.so’
./json-1.1.3/ext/json/ext/parser/parser.so
./json-1.1.3/ext/json/ext/generator/generator.so
./json-1.1.3/ext/json/ext/parser.so
./json-1.1.3/ext/json/ext/generator.so
./fastthread-1.0.1/ext/fastthread/fastthread.so
./fastthread-1.0.1/lib/fastthread.so
./rcov-0.8.1.2.0/lib/rcovrt.so
./rcov-0.8.1.2.0/ext/rcovrt/rcovrt.so
./linecache-0.43/ext/trace_nums.so
./linecache-0.43/lib/trace_nums.so
./mongrel-1.1.5/ext/http11/http11.so
./mongrel-1.1.5/lib/http11.so
./sqlite3-ruby-1.2.4/ext/sqlite3_api/sqlite3_api.so
./sqlite3-ruby-1.2.4/lib/sqlite3_api.so
./ruby-debug-base-0.10.3/ext/win32/ruby_debug.so
./ruby-debug-base-0.10.3/ext/ruby_debug.so
./ruby-debug-base-0.10.3/lib/ruby_debug.so
./ruby-gpgme-1.0.3/lib/gpgme_n.so
./ruby-gpgme-1.0.3/gpgme_n.so

On another machine I also have termios, rubywmq, ruby-postgres,
sys-filesystem, RedCloth

5

On Feb 2, 2009, at 07:15 , Charles Oliver N. wrote:

  • They are wrappers around a C API/library, as in zlib, rmagick,
    nokogiri
  • They are written for performance reasons, to implement a
    particular algorithm in a native language or call a library for the
    same reasons

or they’re written to get into ruby internals… I don’t quite
consider that a wrapper around C API and it looks like you don’t either.

6

Charles Oliver N. wrote:

Wrappers can now largely be handled by FFI, and I hope more and more

  • Charlie

Charlie,

As someone with a social survey research background, I want to advise
you that this is an extremely poor way to get your question answered. If
you’re not serious about getting a good answer, your request is very
close to list-noise. If you are, then you need a decent sample OR all
the parametric data (i.e., don’t sample it - get it all).

I wonder why you don’t go after the latter? Is there someway to get
download counts for various gems? I realize there are hundreds, but THAT
would be useful data.

Alternatively, you could use the gems themselves as your population.
Draw a sample of them, as samples of serious Ruby code (and get at least
35, and preferably much more than that), then scrape from them the gems
THEY use, and get a frequency distribution from that sample.

In terms of bang for buck, I’d go with the latter alternative, 'cause
THAT data would actually be something from which you might reasonably
infer something.

What you’re doing with this list-post nonsense is akin to putting a box
of surveys on the sidewalk, with a sign “please fill one out”, then
taking your results and thinking they actually MEAN something. Believe
me, they do not.

Hope this helps the “cause”.

t.

–

Tom C., MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< [email protected] >> (email)
<< TomCloyd.com >> (website)
<< sleightmind.wordpress.com >> (mental health weblog)
7

Wrappers can now largely be handled by FFI, and I hope more and more of

  • Charlie

I use RUDL or Rubygame (SDL wrappers) to make games;
ruby-prof for profiling of these games, because pure Ruby profiler from
the standard library slows programs too much down.

Jakub

–
“Configure complete, now type ‘make’ and PRAY.”

            (configure script of zsnes - www.zsnes.com)
8

Tom C. wrote:

And there’s a lot of grey area, with some extensions falling in both

If you’re not serious about getting a good answer, your request is
the gems THEY use, and get a frequency distribution from that sample.
Hope this helps the “cause”.

t.

There is always this:

http://gems.rubyforge.org/stats.html

-Justin

9

Justin C. wrote:

same reasons
for things FFI or “faster Ruby” could do, since it will improve the
As someone with a social survey research background, I want to advise
Draw a sample of them, as samples of serious Ruby code (and get at
Believe me, they do not.

-Justin

Now, THAT’s outright cheating. Worse yet, you probably Googled to find
this (or could have, if you didn’t know about it). What is this world
coming too?

t.

–

Tom C., MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< [email protected] >> (email)
<< TomCloyd.com >> (website)
<< sleightmind.wordpress.com >> (mental health weblog)
10

Tom C. wrote:

As someone with a social survey research background, I want to advise
you that this is an extremely poor way to get your question answered. If
you’re not serious about getting a good answer, your request is very
close to list-noise. If you are, then you need a decent sample OR all
the parametric data (i.e., don’t sample it - get it all).

I disagree. I think the people most likely to respond to a mailing-list
request are the exact people I’m looking to reach. Did it occur to you
that perhaps I know there are download stats for all gems that I could
mine? I’m looking to gauge the importance of gems to vocal mailing-list
participants directly, by posting a general message here and seeing who
responds. And I’ve gotten a lot of great responses, both on-list and
off. If I wanted an accurate count of gem downloads (no doubt inflated
by older gems including dev-time libraries as runtime dependencies) I
certainly could get that. But people coming out and saying “I really
need X” is far more interesting.

So please, others, feel free to reply with what X you find important.

And Tom, thanks for your input. What native extensions do you find
useful?

  • Charlie
11

Roger P. wrote:

So, what native gems or extensions do you use? Why do you use them or
why do they exist?

I use mysql, mysqlplus, eventmachine, ruby debug, hitimes, win32-api,
ruby prof, mongrel, and ferret. Most wrappers to C libs [which is for
speed].
Cheers!

Thanks for the response!

Is that the win32 that ships with MRI or something else? I don’t use MRI
much, so I’m not familiar with that library.

eventmachine and mongrel…I assume we’re talking about evented mongrel?

ruby_debug and ruby_prof…I see ruby_prof coming up a lot. I think
we’ll simply need to provide something similar out of the box for JRuby.
Hmmm.

One last question…why mysql or mysqlplus gems instead of the “nice”
DB wrapper libraries like Sequel, DataMapper, and so on?

  • Charlie
12

Coey M. wrote:

As you may guess, this is a Rails application. FastCGI is used for
Apache deployment; ruby-postgres is for accessing the PostgreSQL
database. I know there are other options for Apache/web deployment
(e.g. Mongrel), but this install is stable and other priorities take
precedence over investigating alternative deployment configurations.
For PostgreSQL, even if I moved to ruby-pg, that is still a C library
wrapper.

Anyways, just trying to answer your query.

Cool, thanks. Are you using ruby-postgres through ActiveRecord itself or
standalone in some way

  • Charlie
13

strongtyping, used by html-table. I sent the author a patch that makes
it work
with 1.8.6 and 1.9.1 but haven’t heard back.

14

Brian C. wrote:

  • They are wrappers around a C API/library, as in zlib, rmagick,
    nokogiri
  • They are written for performance reasons, to implement a particular
    algorithm in a native language or call a library for the same reasons
  • They interact directly with the Ruby interpreter (e.g. rcov,
    ruby-debug)

Damn good point…you totally trumped me there. So there’s three cases,
with this new one perhaps being truly the most impl-specific case.

Here’s the set on my home machine:

$ cd /usr/local/lib/ruby/gems/1.8/gems/; find . -name ‘*.so’
./json-1.1.3/ext/json/ext/parser/parser.so
./json-1.1.3/ext/json/ext/generator/generator.so
./json-1.1.3/ext/json/ext/parser.so
./json-1.1.3/ext/json/ext/generator.so

Do people generally use the native json gem for performance reasons?
Does a pure-ruby version not cut it? Are there benchmarks?

./fastthread-1.0.1/ext/fastthread/fastthread.so
./fastthread-1.0.1/lib/fastthread.so

Ok, not needed under JRuby, recent Ruby 1.8.6 or 1.8.7+.

./rcov-0.8.1.2.0/lib/rcovrt.so
./rcov-0.8.1.2.0/ext/rcovrt/rcovrt.so

Tooling…blasted tooling. JVM has so much good tooling, if I could only
harness it well for Ruby stuff.

./ruby-gpgme-1.0.3/gpgme_n.so

On another machine I also have termios, rubywmq, ruby-postgres,
sys-filesystem, RedCloth

Thanks for the response. Obviously I’m interested because of JRuby, and
I think most of these have equivalents already, but there are obviously
gaps. I think dev-time tooling may actually be where we’re weakest at
the moment, which is ironic given the history of JVM/Java-based tooling.

  • Charlie
15

Ryan D. wrote:

or they’re written to get into ruby internals… I don’t quite consider
that a wrapper around C API and it looks like you don’t either.

Yeah, that’s definitely a third category I totally missed. It’s an
unusual one, though, since it doesn’t necessarily have to be “native”
(it could be in Ruby in Rubinius, or Java in JRuby…)

And my failure to recognize it could be why we don’t have a ruby-prof
for JRuby yet. Definitely noted.

  • Charlie
16

Jakub Pavlík jn. wrote:

Wrappers can now largely be handled by FFI, and I hope more and more of

  • Charlie

I use RUDL or Rubygame (SDL wrappers) to make games;
ruby-prof for profiling of these games, because pure Ruby profiler from the standard library slows programs too much down.

I think someone was working on SDL over FFI recently, but I don’t
remember who and I don’t remember how far they got. I’d love to see it
myself.

ruby-prof…noted. Seems like an essential tool for most folks, we’ll
just have to provide something.

  • Charlie
17

On Feb 3, 2009, at 01:24 , Charles Oliver N. wrote:

Ryan D. wrote:

or they’re written to get into ruby internals… I don’t quite
consider that a wrapper around C API and it looks like you don’t
either.

Yeah, that’s definitely a third category I totally missed. It’s an
unusual one, though, since it doesn’t necessarily have to be
“native” (it could be in Ruby in Rubinius, or Java in JRuby…)

Yeah. just depends on the impl’s API into itself. ParseTree simply
isn’t necessary on rubinius.

18

Jos B. wrote:

strongtyping, used by html-table. I sent the author a patch that makes it work
with 1.8.6 and 1.9.1 but haven’t heard back.

I was unaware of strongtyping…very glad you brought it to my
attention. So many projects, so little time…but this warrants some
time.

Thank you!

  • Charlie
19

$ cd /usr/local/lib/ruby/gems/1.8/gems/; find . -name ‘*.so’
./json-1.1.3/ext/json/ext/parser/parser.so
./json-1.1.3/ext/json/ext/generator/generator.so
./json-1.1.3/ext/json/ext/parser.so
./json-1.1.3/ext/json/ext/generator.so

Do people generally use the native json gem for performance reasons?
Does a pure-ruby version not cut it? Are there benchmarks?

I only use it by default, because there are other gems which have
dependencies on “json” rather than “json_pure”. e.g.

$ gem dependency couchrest
Gem couchrest-0.12.4
json (>= 1.1.2, runtime)
rest-client (>= 0.5, runtime)
mime-types (>= 1.15, runtime)

Regards,

Brian.

20

Ryan D. wrote:

Yeah. just depends on the impl’s API into itself. ParseTree simply isn’t
necessary on rubinius.

I’m still interested in producing a PT-compatible sexp from JRuby’s AST,
but it becomes harder the more we customize that AST (and the more we
pre-compile code and dump the memory-expensive AST).

The other stuff, though, we definitely need. Profiling, debugging
(including into Java code, if necessary), code coverage. It’s
“interesting” to be reinventing all these tools again atop the JVM, but
it’s also frustrating that they weren’t created language-agnostic by JVM
advocates in the beginning.

Myopic idiots.

  • Charlie