# \[ANN\] Rubydoop 1.0, write Hadoop jobs in JRuby

**URL:** <https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381>\
**Category:** JRuby\
**Created:** [October 2, 2012, 7:46am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381 "2012-10-02T07:46:08Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![aris](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/258eb7/32.png) [@aris](https://www.ruby-forum.com/u/aris)\
**Post date:** [October 2, 2012, 7:46am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/1 "2012-10-02T07:46:08Z")

</div>

Hi,

I’ve just released v1.0.0 of Rubydoop. It’s a gem that makes it possible  
to write Hadoop jobs in Ruby without using the streaming APIs. It  
configures the Hadoop runtime to run your Ruby code in an embedded JRuby  
runtime, and it provides a configuration DSL that’s way nicer to use  
than Hadoop’s ToolRunner.

Check out the code, some instructions and examples at

> **[GitHub - iconara/rubydoop: Write Hadoop jobs in JRuby](https://github.com/iconara/rubydoop)**
>
> Write Hadoop jobs in JRuby. Contribute to iconara/rubydoop development by creating an account on GitHub.

If you’re into Hadoop and JRuby, I really hope this will help you, and  
if you have any suggestions or feature requests, please let me know.

yours,  
Theo, @iconara

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 17, 2012, 9:33pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/2 "2012-12-17T21:33:34Z")

</div>

hi –

i’ve been looking at amazon elastic mapreduce + jruby myself. how does  
your approach differ from using something like warbler to compile ruby  
sources + gem dependencies into a standalone jar then using that jar  
when invoking hadoop?

thanks.

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 19, 2012, 2:41am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/3 "2012-12-19T02:41:42Z")

</div>

thanks. i’ve been playing around with rubydoop today, and have a  
question: it looks to me like the packaging won’t honor gemspecs, it  
only scans each gem’s lib/ directory and packages up whatever it finds  
in there. is that correct, or am i misconfiguring something? (one of my  
local gems happens to have something in app/ as well as lib/)

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 18, 2012, 7:33pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/4 "2012-12-18T19:33:46Z")

</div>

Rubydoop tries to solve the problem of how to make Hadoop talk to your  
Ruby  
code, without forcing you to write lots of annotations to make your Ruby  
classes look like Java classes. You could say that it’s more or less the  
same thing as using Warbler and all that, but doing that is a lot of  
work.

I’m currently running a Rubydoop-based job on EMR on 640 CPUs, it works  
like a charm. If you’re going to run it on lots of data, you should use  
the  
v1.1.0.pre2, it contains a few things I’ve had to fix when scaling up  
from  
small jobs to massive jobs.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 19, 2012, 3:50am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/5 "2012-12-19T03:50:04Z")

</div>

i think i see what’s going on: it’s respecting the require\_paths  
directive, but not the files directive.

also it looks like it flattens out the folder hierarchy such that if i  
have:

my\_gem  
|  
|- lib/  
|-one.rb  
|-vendor/  
|-two.rb

the jar will contain

/one.rb  
/two.rb

a problem with this is if one.rb has a relative path, e.g.

here=File.expand\_path(File.dirname( **FILE** ))  
require “#{here}/…/vendor/two.rb”

this is now broken. also if you have two gems containing the same  
filename, it seems like you’d have a problem.

i should mention that this is for a locally installed gem (i.e.  
gem ‘blah’, :path =\> ‘…/…/’  
not sure if it makes a difference)

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 19, 2012, 9:09am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/6 "2012-12-19T09:09:44Z")

</div>

“I’d love to find a safer way of packaging the dependencies, but short  
of  
puting the whole gem directories into the jar and doing some magic to  
make sure that they all are on the load path I think that this works  
reasonably well.”

that doesn’t sound so bad, i was in the middle of playing around with  
something like:

```
def load_gem_files
  Bundler.definition.specs_for(@options[:gem_groups]).flat_map do

```

|spec|  
if spec.full\_name !~ /^(?:bundler|rubydoop|jruby-openssl)-\d+/  
spec.files.select {|f| f =~ /.rb$/}.map do |f|  
[spec.full\_gem\_path, f]  
end  
else  
[]  
end  
end  
end

the in the ant jar definition, adding:

```
      bundled_gem_files.each { |path| fileset :dir => path[0],

```

:includes =\> “#{path[1]}” }

it’s a little ugly since we could unify files with common parent  
folders, but it works in that the correct files with the correct folder  
structure are added to the jar.

the thing i haven’t figured out yet is how to ensure that all those  
files are in the load path. i’m probably missing something though,  
because that seems like it should be straightforward – can’t you just  
add all those directories to the manifest file’s classpath (since my  
understanding is that the jruby load\_path and classpath are unified)?

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 19, 2012, 9:27am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/7 "2012-12-19T09:27:22Z")

</div>

if you get something working it would be something I’d love to merge in.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 19, 2012, 7:17am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/8 "2012-12-19T07:17:41Z")

</div>

yes, it only looks at the require paths. there are a lot of gems that  
are  
set up very differently, I tried a few things and it looked like using  
require paths worked the best. there are a lot of gems that don’t have  
anything, or very little, in the files list. many gems are packaged  
really  
badly, and one of the worst offenders is jruby-openssl, which has caused  
me  
no end of trouble (it went so far that I had to put in special handling  
for  
it, basically rewriting its structure before adding it to the jar).

the require path thing works because these are the paths put on the load  
path, and even if you have two files with the same name on the load  
path,  
require will only find the first of them anyway – so it doesn’t matter  
that only one gets packaged into the jar. that is, unless your gem tries  
to  
be clever, like jruby-openssl, and loads things that are neither in any  
require path, or in the files list.

it gets really hairy when the code is running from inside a jar and you  
get  
paths like blah/blah/job.jar!/some/gem/…/…/…/doblidoo/file.rb, since  
one  
of those … takes you outside of the jar file.

I’d love to find a safer way of packaging the dependencies, but short of  
puting the whole gem directories into the jar and doing some magic to  
make  
sure that they all are on the load path I think that this works  
reasonably  
well. I did some work on the packaging in redstorm and it wasn’t very  
pretty. I wanted to get away from the complicated things that happen  
when  
you just stuff all the gems into the jar and try to set up the correct  
load  
path. putting all the require paths in the root of the jar makes all  
well-behaved gems, and most slightly weird gems work out of the box.

maybe bundler can be coaxed into helping, but it behaves strangely when  
it’s not in control of things.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 20, 2012, 8:13am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/9 "2012-12-20T08:13:19Z")

</div>

which version of Hadoop are you running? I’ve targeted 1.0.3.

from the error it looks like it’s trying to run another job after the  
first  
has finished and that it’s that job that is complaining.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 20, 2012, 4:01am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/10 "2012-12-20T04:01:07Z")

</div>

i ended up adding this line to my MR:

$LOAD\_PATH \<\< File.expand\_path(File.dirname( **FILE** )) + ‘/lib’

as a way of forcing the load path to be what i want. i couldn’t get  
jruby to respect the manifest classpath.

my mapreduce actually completes now (i get the reducer output file), but  
at the end it complains about the output path existing (which it  
definitely does not before i run the command). have you run into this?

Ilyas-MacBook-Pro:word\_count ilya$ hadoop jar build/word\_count.jar  
word\_count Gemfile /tmp/hadoop3  
12/12/19 18:03:55 INFO mapred.Task: Task:attempt\_local\_0001\_r\_000000\_0  
is done. And is in the process of commiting  
12/12/19 18:03:55 INFO mapred.LocalJobRunner:  
12/12/19 18:03:55 INFO mapred.Task: Task attempt\_local\_0001\_r\_000000\_0  
is allowed to commit now  
12/12/19 18:03:55 INFO output.FileOutputCommitter: Saved output of task  
‘attempt\_local\_0001\_r\_000000\_0’ to /tmp/hadoop3  
12/12/19 18:03:55 INFO mapred.LocalJobRunner: reduce \> reduce  
12/12/19 18:03:55 INFO mapred.Task: Task ‘attempt\_local\_0001\_r\_000000\_0’  
done.  
12/12/19 18:03:56 INFO mapred.JobClient: map 100% reduce 100%  
12/12/19 18:03:56 INFO mapred.JobClient: Job complete: job\_local\_0001  
12/12/19 18:03:56 INFO mapred.JobClient: Counters: 18  
12/12/19 18:03:56 INFO mapred.JobClient: Rubydoop  
12/12/19 18:03:56 INFO mapred.JobClient: JRuby runtimes created=2  
12/12/19 18:03:56 INFO mapred.JobClient: File Output Format Counters  
12/12/19 18:03:56 INFO mapred.JobClient: Bytes Written=217  
12/12/19 18:03:56 INFO mapred.JobClient: FileSystemCounters  
12/12/19 18:03:56 INFO mapred.JobClient: FILE\_BYTES\_READ=47005944  
12/12/19 18:03:56 INFO mapred.JobClient: FILE\_BYTES\_WRITTEN=47444829  
12/12/19 18:03:56 INFO mapred.JobClient: File Input Format Counters  
12/12/19 18:03:56 INFO mapred.JobClient: Bytes Read=258  
12/12/19 18:03:56 INFO mapred.JobClient: Map-Reduce Framework  
12/12/19 18:03:56 INFO mapred.JobClient: Map output materialized  
bytes=342  
12/12/19 18:03:56 INFO mapred.JobClient: Map input records=10  
12/12/19 18:03:56 INFO mapred.JobClient: Reduce shuffle bytes=0  
12/12/19 18:03:56 INFO mapred.JobClient: Spilled Records=50  
12/12/19 18:03:56 INFO mapred.JobClient: Map output bytes=286  
12/12/19 18:03:56 INFO mapred.JobClient: Total committed heap usage  
(bytes)=507133952  
12/12/19 18:03:56 INFO mapred.JobClient: SPLIT\_RAW\_BYTES=125  
12/12/19 18:03:56 INFO mapred.JobClient: Combine input records=0  
12/12/19 18:03:56 INFO mapred.JobClient: Reduce input records=25  
12/12/19 18:03:56 INFO mapred.JobClient: Reduce input groups=20  
12/12/19 18:03:56 INFO mapred.JobClient: Combine output records=0  
12/12/19 18:03:56 INFO mapred.JobClient: Reduce output records=20  
12/12/19 18:03:56 INFO mapred.JobClient: Map output records=25  
12/12/19 18:03:56 WARN mapred.JobClient: No job jar file set. User  
classes may not be found. See JobConf(Class) or JobConf#setJar(String).  
12/12/19 18:03:56 INFO mapred.JobClient: Cleaning up the staging area  
file:/tmp/hadoop-ilya/mapred/staging/ilya1409933347/.staging/job\_local\_0002  
12/12/19 18:03:56 ERROR security.UserGroupInformation:  
PriviledgedActionException as:ilya  
cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output  
directory /tmp/hadoop3 already exists  
Exception in thread “main”  
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory  
/tmp/hadoop3 already exists  
at  
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)  
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:949)  
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:396)  
at  
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)  
at  
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)  
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)  
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)  
at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:31)  
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)  
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)  
at rubydoop.RubydoopJobRunner.main(RubydoopJobRunner.java:82)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at  
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  
at  
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  
at java.lang.reflect.Method.invoke(Method.java:597)  
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 20, 2012, 10:39am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/11 "2012-12-20T10:39:37Z")

</div>

i was running 1.1.1. via a combination of downgrading to 1.0.3 plus  
some other changes i’ve been making, i successfully ran a mapreduce  
locally using my entire rails app packaged up in a single jar (a few  
hundred gems, roughly 4000 files).

trying the same jar with elastic mapreduce, however, gives me an error  
right now:

Exception in thread “main” rubydoop.RubydoopRunnerException: Could not  
load job setup script “word\_count”  
at rubydoop.RubydoopJobRunner.configureJobs(RubydoopJobRunner.java:63)  
at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:30)  
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)  
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)  
at rubydoop.RubydoopJobRunner.main(RubydoopJobRunner.java:82)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at  
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  
at  
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  
at java.lang.reflect.Method.invoke(Method.java:597)  
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)  
Caused by: org.jruby.exceptions.RaiseException: (LoadError) no such file  
to load – rubygems/path\_support

the $LOAD\_PATH as reported by my mapper is:

[“file:/home/hadoop/lib/jruby-complete-no-joda-1.6.5.jar!/META-INF/jruby.home/lib/ruby/site\_ruby/1.9”,  
“file:/home/hadoop/lib/jruby-complete-no-joda-1.6.5.jar!/META-INF/jruby.home/lib/ruby/site\_ruby/shared”,  
“file:/home/hadoop/lib/jruby-complete-no-joda-1.6.5.jar!/META-INF/jruby.home/lib/ruby/1.9”,  
“/mnt/var/lib/hadoop/tmp/hadoop-unjar1366878146839692659”,  
“/mnt/var/lib/hadoop/tmp/hadoop-unjar1366878146839692659/lib”]

some quick googling led me to this:  
[http://techpolesen.blogspot.com/2007/07/jruby-runtime-initialization-and.html](http://techpolesen.blogspot.com/2007/07/jruby-runtime-initialization-and.html),  
but replacing

```
    Ruby runtime = Ruby.newInstance(config);

```

with  
Ruby runtime = JavaEmbedUtils.initialize(new ArrayList(),  
config)

in InstanceContainer didn’t seem to make any difference. i plan on  
tackling this next, any pointers would be appreciated.

Theo Hultberg wrote in post #1089694:

> which version of Hadoop are you running? I’ve targeted 1.0.3.
> 
> from the error it looks like it’s trying to run another job after the  
> first  
> has finished and that it’s that job that is complaining.
> 
> T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 20, 2012, 3:43pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/12 "2012-12-20T15:43:56Z")

</div>

good catch, i didn’t notice hadoop’s jruby 1.6 jar had snuck in there.  
you mentioned that you’re running rubydoop on EMR – is there a setting  
that allows you to use 1.7.x there? i assume you have to change  
hadoop’s classpath before starting up, via something like this  
[Forums | AWS re:Post](https://forums.aws.amazon.com/thread.jspa?messageID=347690)?

Theo Hultberg wrote in post #1089725:

> JRuby 1.6.x has some bugs in the embedding that I’ve run into. That  
> particular error will resolve if you upgrade to JRuby 1.7.x, but if you  
> can’t you might have to fiddle with the load path a bit (I forget  
> exactly  
> where rubygems/path\_support is located).
> 
> Rubydoop removes the 1.8 stdlib from the load path, since JRuby 1.6.x  
> adds  
> both the 1.8 and 1.9 stdlibs for some reason, even when the runtime is  
> explicitly running in 1.9 mode.
> 
> T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 20, 2012, 1:04pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/13 "2012-12-20T13:04:50Z")

</div>

JRuby 1.6.x has some bugs in the embedding that I’ve run into. That  
particular error will resolve if you upgrade to JRuby 1.7.x, but if you  
can’t you might have to fiddle with the load path a bit (I forget  
exactly  
where rubygems/path\_support is located).

Rubydoop removes the 1.8 stdlib from the load path, since JRuby 1.6.x  
adds  
both the 1.8 and 1.9 stdlibs for some reason, even when the runtime is  
explicitly running in 1.9 mode.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 20, 2012, 4:15pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/14 "2012-12-20T16:15:15Z")

</div>

I have no idea why there’s a jruby-complete.jar on the default EMR  
installation, but so far I haven’t noticed any bad effects from removing  
it.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 20, 2012, 4:12pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/15 "2012-12-20T16:12:20Z")

</div>

oooooh, now I remember. when you run on EMR you need to run this as a  
bootstrap action:

```
#!/bin/bash
rm /home/hadoop/lib/jruby-complete-no-joda-1.6.5.jar

```

I’ve just copy and pasted that so many times now I’ve forgotten about  
it. I  
should write a guide and stick all these things there for posterity.

if you launch your flow using the aws-sdk gem you can add this:

```
remove_old_jruby_action = {
    :name => 'remove_old_jruby',
    :script_bootstrap_action => {
      :path => "s3://your-bucket/remove_old_jruby.sh"
    }
}

```

and then add that to the :bootstrap\_actions option to  
JobFlowCollection#create. don’t forget to upload the script to s3 before  
you run the flow.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 21, 2012, 11:24pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/16 "2012-12-21T23:24:11Z")

</div>

thanks, that helped me make progress. the thing i’m struggling with now  
is that it seems like EMR does not unjar my jar before running it, which  
is what my local hadoop does (i.e. everything gets loaded from a  
/tmp/… directory that my jar gets uncompressed to locally). in EMR, i  
get stacktraces like:

## org.jruby.exceptions.RaiseException: (ENOENT) No such file or directory

jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!  
at org.jruby.RubyFile.realpath(org/jruby/RubyFile.java:760)  
at  
RUBY.realpath(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/lib/jruby-complete-1.7.1.jar!/META-INF/jruby.home/lib/ruby/1.9/pathname.rb:446)  
at  
RUBY.find\_root\_with\_flag(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!/lib/rails/engine.rb:637)  
at  
RUBY.config(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!/lib/rails/engine.rb:511)  
at  
org.jruby.RubyBasicObject. **send** (org/jruby/RubyBasicObject.java:1659)  
at RUBY.config(jar:0)  
at  
RUBY.Engine(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!/lib/cs-base/engine.rb:11)

from which it looks like EMR hadoop is loading directly from that jar  
file, which makes any attempts to call Pathname.new().realpath (for  
example) fail. some gems try to do that (to load their configs, etc).  
i’m trying to figure out how to get EMR to stage my jar similarly to  
what my local hadoop installation does.

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 24, 2012, 8:13am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/17 "2012-12-24T08:13:28Z")

</div>

okay, after wrestling with rails, jruby and hadoop for a few days, i’m  
finally able to use my rails app in the mapper class. the key sticking  
point turned out to be the fact that rails searches for its  
configuration by walking up the directory tree by using File.dirname.  
but if running from a jar, jruby’s RubyFile implementation refuses to  
treat the “root” directory as a valid path entry. the easy solution was  
to modify radoop’s package.rb to jar up my rails app code under a  
/classes directory instead of at the top level. i then needed to add  
/classes and /classes/lib to the LOAD\_PATH. (it has be /classes due to  
hadoop’s classpath when the job jar is run by the workers.)

i’m attaching my patched up version package.rb, which is the only change  
to rubydoop.rb i needed to make. if it looks interesting, let me know  
and i can send you a pull request on github.

thanks for all your help.

Ilya K. wrote in post #1089906:

> thanks, that helped me make progress. the thing i’m struggling with now  
> is that it seems like EMR does not unjar my jar before running it, which  
> is what my local hadoop does (i.e. everything gets loaded from a  
> /tmp/… directory that my jar gets uncompressed to locally). in EMR, i  
> get stacktraces like:
> 
> ## org.jruby.exceptions.RaiseException: (ENOENT) No such file or directory

jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!

> at org.jruby.RubyFile.realpath(org/jruby/RubyFile.java:760)  
> at

RUBY.realpath(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/lib/jruby-complete-1.7.1.jar!/META-INF/jruby.home/lib/ruby/1.9/pathname.rb:446)

> at

RUBY.find\_root\_with\_flag(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!/lib/rails/engine.rb:637)

> at

RUBY.config(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!/lib/rails/engine.rb:511)

> at  
> org.jruby.RubyBasicObject. **send** (org/jruby/RubyBasicObject.java:1659)  
> at RUBY.config(jar:0)  
> at

RUBY.Engine(jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job\_201212212148\_0001/jars/job.jar!/lib/cs-base/engine.rb:11)

> from which it looks like EMR hadoop is loading directly from that jar  
> file, which makes any attempts to call Pathname.new().realpath (for  
> example) fail. some gems try to do that (to load their configs, etc).  
> i’m trying to figure out how to get EMR to stage my jar similarly to  
> what my local hadoop installation does.

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 25, 2012, 11:41am UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/18 "2012-12-25T11:41:55Z")

</div>

Thanks for the patch, not near my computer at the moment so I haven’t  
looked at it, but I’d rather that you submitted a pull request anyway,  
it’s much easier to work with and discuss.

T#

---

<div class="post-metadata">

**Author:** ![theo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a87d85/32.png) [@theo](https://www.ruby-forum.com/u/theo)\
**Post date:** [December 27, 2012, 8:54pm UTC](https://www.ruby-forum.com/t/ann-rubydoop-1-0-write-hadoop-jobs-in-jruby/223381/19 "2012-12-27T20:54:58Z")

</div>

sure

Theo Hultberg wrote in post #1090144:

> Thanks for the patch, not near my computer at the moment so I haven’t  
> looked at it, but I’d rather that you submitted a pull request anyway,  
> it’s much easier to work with and discuss.
> 
> T#