Forum: JRuby Minimizing memory usage

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Martijn S (Guest)
on 2009-04-16 12:46
(Received via mailing list)
Hi all,

As you might know we run a small Rails hosting business in the
Netherlands.
I'm currently planning to offer shared JRuby hosting through the
Glassfish
gem or possibly mongrel (no 'real' app server just yet), proxied behind
Apache.

Unfortunately an average Rails application uses between 250-350MB of
resident memory this way (JDK 1.6 64-bit, Server VM, Linux 2.6), which
is a
bit much compared to the REE+Passenger apps we normally run. Is there
anything easy we can do to minimize the memory usage of these separate
JRuby
applications without impacting performance to much? All small
improvements
are welcome. I looked around a bit, but I don't know what effect heap
sizes
etc. have on memory usage and performance.

Regards,

Martijn

P.S. The performance of JRuby 1.2.0 with the above setup on a Quad Core
Xeon
with 8GB RAM is amazing :)
--
View this message in context:
http://www.nabble.com/Minimizing-memory-usage-tp23...
Sent from the JRuby - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
marc (Guest)
on 2009-04-16 14:05
(Received via mailing list)
Martijn S said...
> applications without impacting performance to much? All small improvements
> are welcome. I looked around a bit, but I don't know what effect heap sizes
> etc. have on memory usage and performance.
>
> P.S. The performance of JRuby 1.2.0 with the above setup on a Quad Core Xeon
> with 8GB RAM is amazing :)

I know virtually nothing about the detail of this, but I found,
empirically, that I got far better memory usage out of running Redmine -
a bug/problem-tracking Rail app - using the glassfish gem.

Running under glassfish app server, usage would spiral to over 0.5Gb and
even when gc kicked in - it took hours - it would not reduce the
footprint below 200Mb.

Running as:

  jruby -J-Xmx512M -S glassfish --runtimes-min 2 --runtimes-min 8 -e
production

The idle size is around 12Mb, and even when three or four folk are
working on it, it rarely peaks over 40Mb. Performance is better too.

I have no idea whether this is useful info or not, especially in your
case. This whole area seems to be a complex black art, and I think it
would be time well-spent simplifying it.

--
Cheers,
Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Jacob K. (Guest)
on 2009-04-16 21:46
(Received via mailing list)
First, I feel compelled to point out that Glassfish gem is Glassfish
server stripped down to just the stuff that you need to serve Jruby
apps, so it's at least closer to a real app server than mongrel is. It
will handle multiple concurrent users, pool non-threadsafe rails
applications for you, etc. That little rant out of the way...

The first thing to check in any case where Java is using more memory
than you expect it to is your heap settings. If you run Java with
-Xms512m, for example, then Java will set the minimum heap size to
512MB, and it will reserve half a gig of memory, whether it needs it or
not. The GC will only run when it's filled the entire 512MB with
garbage, and it will look like Java is hogging memory, even if it could
easily be happy living in 64MB. If it's taking hours for the GC to have
to run, that's a good sign that you've given it too much heap.

I'd suggest launching a rails app and then pointing jconsole (included
with JDK 5.0+) at it, so that you can see how much of your allocated
heap your applications are actually using. I'm somewhat curious as to
how marc was able to get it to idle at 12mb with two runtimes, but my
generic test application idles at around 40MB per runtime, which is
about what I'm used to seeing. 250-350MB definitely seems high, to me.

You can play around with the -Xmx and -Xms (maximum and minimum heap
sizes, respectively) settings a bit if you want to, but I find that in
general setting the max to the maximum that you are willing to let it
take and leaving the VM's memory allocator to figure out the rest works
best.

Setting -Xms higher (probably with a maximum of your expected idle size)
will improve startup (since resizing the heap forces a full garbage
collection), but otherwise won't affect things. Setting -Xmx higher will
reduce time between garbage collections, but will make the app take up
more memory. To a point, that's good: If you have 8GB RAM and it's a
dedicated server, you want to be using as much of that 8GB of RAM as
possible (with a bit of free space, maybe 512MB, since you _never_ want
to swap to disk). However, RAM beyond what the application needs to only
be doing a young generation collection every few minutes won't improve
performance at all, so more RAM generally means more apps running on the
server... Until you become CPU-bound.

<Java resource usage and GC lecture/rant>
Especially in rails apps, where most objects are extremely short-lived,
all the garbage collections that you run are going to be essentially
instant. The Hotspot GC young generation (objects less than one
collection old) collections scale based on how many objects survive, and
with rails that increases with the number of currently active requests
when the GC goes off. I actually have GC logs from Glassfish running a
Rails app (with images, AJAX, etc.) at a few thousand requests per
second, and it only has about 12MB of new live data per collection,
which means GCs in the hundredths of seconds. Those collections were
happening every three seconds, but they don't have to stop the
application and .02 seconds really isn't very long.  Eventually, the
surviving objects will pile up and a full collection will need to be run
to clean them out, and that will take about a second. The full GCs need
to stop everything for a bit, but not for the full second. The stress
test needed a full collection roughly every 45 minutes on a 1G heap, so
a second of GC every 45 minutes is not bad at all. This is also on an
app serving a few thousand requests per second, so you'll likely have
much lower traffic.

I had a point, I promise (that's why I labeled this as a rant). When you
are sizing your server instances, you'll want to figure out where the
best compromise between number of runtimes and frequency of garbage
collections is. If you set the heap too small, you'll be doing full
collections more often, which will eventually noticeably impact
performance: If you're running a 1-second collection every minute, it's
not that big a deal. If you're running a 1-second collection every three
seconds, it's a problem. If you actually run out of heap, you can be
doing a 1-second collection every second, which will grind your app to a
halt. But, if you set the heap too large, you won't be able to fit as
many runtimes into the space that you have, and thus won't be able to
use the machine's resources optimally. If, for example, you had a choice
between 128MB heaps with GCs every minute and 256MB heaps with GCs every
two minutes, you'd probably want to go with the 128MB heaps as long as
you had idle CPU time, since that means (potentially) twice as many apps
on the machine at once. Once you become CPU-bound (CPUs at 100% under
your "heavy" load), then you might as well increase the heaps to fill
the available memory, since adding more apps would decrease overall
performance.

Always remember that the processors and memory on your computers are
there to be used, and if you can go to maximal load and still have idle
CPU and free memory, you can add more load and thus get more out of the
computer.
</rant>


rant tl;dr: You want the largest heaps that you can get on as many apps
as it takes to use all of the available processor when fully loaded, as
long as your full GC times are reasonable in those conditions. If they
aren't, reduce the number of apps and increase the heap size on the
remaining ones until full GC times are reasonable (or add more memory to
the machine, if possible).

marc wrote:
>> anything easy we can do to minimize the memory usage of these separate JRuby
> a bug/problem-tracking Rail app - using the glassfish gem.
> The idle size is around 12Mb, and even when three or four folk are
> working on it, it rarely peaks over 40Mb. Performance is better too.
>
> I have no idea whether this is useful info or not, especially in your
> case. This whole area seems to be a complex black art, and I think it
> would be time well-spent simplifying it.
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ikai L. (Guest)
on 2009-04-16 22:07
(Received via mailing list)
Jacob, that was a really good email. It'd be really helpful in a blog
post!

Ikai


On 4/16/09 10:48 AM, "Jacob K." <removed_email_address@domain.invalid> wrote:

> not. The GC will only run when it's filled the entire 512MB with
>
> more memory. To a point, that's good: If you have 8GB RAM and it's a
> instant. The Hotspot GC young generation (objects less than one
> to stop everything for a bit, but not for the full second. The stress
> performance: If you're running a 1-second collection every minute, it's
> your "heavy" load), then you might as well increase the heaps to fill
> rant tl;dr: You want the largest heaps that you can get on as many apps
>>> I'm currently planning to offer shared JRuby hosting through the Glassfish
>>>
>> footprint below 200Mb.
>> case. This whole area seems to be a complex black art, and I think it
>
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Martijn S (Guest)
on 2009-04-16 22:35
(Received via mailing list)
Wow, thanks a lot for this extensive reply! Things are a lot clearer to
me
now.

So if I understand correctly, (part of) the tradeoff between memory
usage
and performance lies in the heap size vs GC runs. I'll try fiddling with
jconsole and some different apps as you recommended to set some
reasonable
boundries. If I find out anything interesting I'll report back here.


Jacob K. wrote:
> 512MB, and it will reserve half a gig of memory, whether it needs it or
> about what I'm used to seeing. 250-350MB definitely seems high, to me.
> reduce time between garbage collections, but will make the app take up
> all the garbage collections that you run are going to be essentially
> to clean them out, and that will take about a second. The full GCs need
> collections more often, which will eventually noticeably impact
> on the machine at once. Once you become CPU-bound (CPUs at 100% under
>
>>> As you might know we run a small Rails hosting business in the
>>> anything easy we can do to minimize the memory usage of these separate
>>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>
>
>

--
View this message in context:
http://www.nabble.com/Minimizing-memory-usage-tp23...
Sent from the JRuby - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Jacob K. (Guest)
on 2009-04-16 22:48
(Received via mailing list)
Sure, I should be able to make it more blog-postish and get it up in a
few days.

Ikai L. wrote:
>> apps, so it's at least closer to a real app server than mongrel is. It
>> to run, that's a good sign that you've given it too much heap.
>> general setting the max to the maximum that you are willing to let it
>> to swap to disk). However, RAM beyond what the application needs to only
>> when the GC goes off. I actually have GC logs from Glassfish running a
>> app serving a few thousand requests per second, so you'll likely have
>> doing a 1-second collection every second, which will grind your app to a
>>
>> aren't, reduce the number of apps and increase the heap size on the
>>>> gem or possibly mongrel (no 'real' app server just yet), proxied behind
>>>> P.S. The performance of JRuby 1.2.0 with the above setup on a Quad Core Xeon
>>>
>>> would be time well-spent simplifying it.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Alex D. (Guest)
on 2009-04-16 23:06
(Received via mailing list)
In the case of multiple applications, would multiple glassfish gem
instances
be more efficient than a single glassfish server instance?
Jacob K. (Guest)
on 2009-04-16 23:17
(Received via mailing list)
I'm not actually sure. On the one hand, with gem you're reloading
Grizzly (the web server) and the infrastructure, which the full server
would only have a single copy of. On the other hand, with gem you've got
a lot less infrastructure than Glassfish server has, and while OSGI
helps with not loading what you aren't using, you've still got OSGI and
the other sniffers in there taking up space.

I suspect that the answer is that given N applications, multiple gems
are smaller than a single server for small N, and a single server is
better for large N. What is "large" or "small" is going to depend on the
size of the applications, though. I'd suggest experimentation with your
application and workload to find the answer.

Alex D. wrote:
> In the case of multiple applications, would multiple glassfish gem instances
> be more efficient than a single glassfish server instance?
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
marc (Guest)
on 2009-04-17 00:56
(Received via mailing list)
Jacob K. said...
> First, I feel compelled to point out that Glassfish gem is Glassfish
> server stripped down to just the stuff that you need to serve Jruby
> apps, so it's at least closer to a real app server than mongrel is. It
> will handle multiple concurrent users, pool non-threadsafe rails
> applications for you, etc. That little rant out of the way...

Not sure why you consider that a rant, it's useful info. More power to
the glassfish gem I say, it works wonderfully.

> The first thing to check in any case where Java is using more memory
> than you expect it to is your heap settings. If you run Java with
> -Xms512m, for example, then Java will set the minimum heap size to
> 512MB, and it will reserve half a gig of memory, whether it needs it or
> not. The GC will only run when it's filled the entire 512MB with
> garbage, and it will look like Java is hogging memory, even if it could
> easily be happy living in 64MB. If it's taking hours for the GC to have
> to run, that's a good sign that you've given it too much heap.

You've not quoted me, but I guess this is in response to what I wrote.
Thing is, I simply installed glassfish - as part of netbeans - and
shoved redmine into it. It worked okay, but for the memory grab. I
looked around a bit and thought that there must be a better way to do
things. Don't get me wrong, I think glassfish is the nuts, but you don't
always need a hammer, etc.

Things is, it's not at all obvious how to configure glassfish - the
basics, not becoming a glassfish guru. That's my point, really, it needs
to be made a bit simpler for simple cases, while, of course, retaining
the standard Javaworld you-can-tweak-every-freaking-bit via XML for the
hardcore element. My use case was to throw up a helpdesk system el
pronto/ That's obviously different to deploying a site that might get
slashdotted.

> I'd suggest launching a rails app and then pointing jconsole (included
> with JDK 5.0+) at it, so that you can see how much of your allocated
> heap your applications are actually using. I'm somewhat curious as to
> how marc was able to get it to idle at 12mb with two runtimes, but my
> generic test application idles at around 40MB per runtime, which is
> about what I'm used to seeing. 250-350MB definitely seems high, to me.

The deployed redmine installation is on a Windows box. That might make a
difference. But that is absolutely its idle state. And it's very
impressive.

> <Java resource usage and GC lecture/rant>

> </rant>

Very useful info. Thanks.

--
Cheers,
Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
marc (Guest)
on 2009-04-17 01:01
(Received via mailing list)
Ikai L. said...
> Jacob, that was a really good email. It'd be really helpful in a blog post!

+1 And if you could devise some simple code to demonstrate jconsole
usage demonstrating your points that would be fantastic. Appreciate that
might be too much work. Y'know, but it doesn't hurt to ask.

--
Cheers,
Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2009-04-17 01:53
(Received via mailing list)
I'm going to have to read this in detail, but a related point is that by
default JRuby uses -Xmx512M, which means that it can easily grow to 512M
in cases where it doesn't *really* need that much space. You can choke
it down by passing your own size:

-J-Xmx200M

for a size that seems comfortable for your app. It's worth playing with
a bit.

- Charlie

Jacob K. wrote:
> not. The GC will only run when it's filled the entire 512MB with
>
> more memory. To a point, that's good: If you have 8GB RAM and it's a
> instant. The Hotspot GC young generation (objects less than one
> to stop everything for a bit, but not for the full second. The stress
> performance: If you're running a 1-second collection every minute, it's
> your "heavy" load), then you might as well increase the heaps to fill
> rant tl;dr: You want the largest heaps that you can get on as many apps
>>> Netherlands.
>>> separate JRuby
>>
>>   jruby -J-Xmx512M -S glassfish --runtimes-min 2 --runtimes-min 8 -e
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2009-04-17 01:55
(Received via mailing list)
Jacob K. wrote:
> size of the applications, though. I'd suggest experimentation with your
> application and workload to find the answer.

Yeah, that's about what I'd say too. I've been recommending that hosting
providers use separate JVM instances, since that's the best isolation.
And in some cases, small N of separate JRuby+GlassfishGem will be better
for single app scaling too. Obviously the ideal is if you can run Rails
in threadsafe mode or any of the other threadsafe web frameworks and
only have a single GF instance with a single JRuby instance. By far the
best bang for your memory buck, and it will warm up a lot faster than N
instances too.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2009-04-17 02:00
(Received via mailing list)
marc wrote:
> the standard Javaworld you-can-tweak-every-freaking-bit via XML for the
> hardcore element. My use case was to throw up a helpdesk system el
> pronto/ That's obviously different to deploying a site that might get
> slashdotted.

This is the number one thing we need to improve in the JRuby world. The
JVM, the app servers, and even JRuby itself are tweakable in an infinite
number of ways, and it's just too complicated to expect anyone to know
how to come up with the right combination. So we need to find ways to
improve it.

I encourage *everyone* to take a look at the GF gem codebase and see if
you can help out. Most of the logic you're interested in is just Ruby
code...there's very little Java. And with a few pointers from the GF
guys, it would be really easy to improve the command line, add some
simple options and improve the config file (0.9.3 new feature), and
generally just make it "ready" for what Rubyists expect.

We really need help making all the great libraries (and JRuby itself)
more Rubyist friendly, and I think this is probably one of the easiest
areas for Rubyists to contribute right now.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Jacob K. (Guest)
on 2009-04-17 02:17
(Received via mailing list)
Charles Oliver N. wrote:
> simple options and improve the config file (0.9.3 new feature), and
> generally just make it "ready" for what Rubyists expect.
+1 there. We're really trying to make the gem as good about being
friendly as we can while still letting it be tweakable. I suspect that
we suffer from knowing what all of the knobs do, and so miss out on the
parts that people who aren't familiar with it don't understand. More
feedback is always good, and we can certainly make changes to it to make
things make more sense.

My personal solution to finding the right combination is constrained
auto-configuration, which the pooled runtime system uses as much as it
can. The basic idea is that you fill in the values you know (or care
about), and the program fills in the rest of the configuration based on
what you've given it. I'm not sure if it's the right way to do things,
but it's definitely fun.
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Christian S. (Guest)
on 2009-04-17 18:14
(Received via mailing list)
I don't know if that already has been mentioned in this thread: I use
-X-C to reduce memory usage a bit and for faster warmup, too. Strange
enough Rails runs even faster with -X-C on my Ubuntu box (while it's a
bit slower on OS X)

Jacob K. wrote:
>> the GF guys, it would be really easy to improve the command line, add
> auto-configuration, which the pooled runtime system uses as much as it
>>
>
>    http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Pablo F. (Guest)
on 2009-04-17 18:48
(Received via mailing list)
Hi

How about -Xmn? I have found that when working large memory chunks
(think 2Gb+ of allocated memory) for a small time, jruby tends to grow
the size of the heap even after the memory chunk is niled. Is there any
way to cause the GC to shrink the heap size after the memory is emptied
using -Xmn?

This is the scenario:

+ Program starts, heap size is about 70 MB (that includes the virtual
machine).
+ Program does its magic (which is quite quick but needs to alloc a big
chunk), heap grows to 2.5 GB+
+ Program sits there waiting for more orders.

When reaching the last step, even though the memory was niled the heap
size remains at the largest number and when reiterating the second step
it goes back to 100 MB for a few moments.

Is there any way to change this behavior with some magic flag?

Best regards,

Pablo


On Thu, 2009-04-16 at 10:48 -0700, Jacob K. wrote:
> not. The GC will only run when it's filled the entire 512MB with
>
> more memory. To a point, that's good: If you have 8GB RAM and it's a
> instant. The Hotspot GC young generation (objects less than one
> to stop everything for a bit, but not for the full second. The stress
> performance: If you're running a 1-second collection every minute, it's
> your "heavy" load), then you might as well increase the heaps to fill
> rant tl;dr: You want the largest heaps that you can get on as many apps
> >> I'm currently planning to offer shared JRuby hosting through the Glassfish
> >>
> > footprint below 200Mb.
> > case. This whole area seems to be a complex black art, and I think it
>
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2009-04-18 02:12
(Received via mailing list)
We've noticed that too, and never really identified what it is. And it's
not consistent either. Sometimes it helps to jit, sometimes not. One
thing that is certain, jitted has less memory churn, so that's one thing
in its favor.

Christian S. wrote:
>>> anyone to know how to come up with the right combination. So we need
>> we suffer from knowing what all of the knobs do, and so miss out on
>>> We really need help making all the great libraries (and JRuby itself)
>>>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2009-04-18 02:14
(Received via mailing list)
There is no way on Hotspot to shrink the heap after it grows, which is
unfortunate. It means you need to decide on a good upper bound and
accept that your app is going to hold that much memory. But of course
for you to be making best use of the system, you wouldn't have a lot of
idle time. It's manageable, but I certainly would love to see a future
version of Hotspot that's able to shrink the heap back down and release
memory.

Pablo F. wrote:
> + Program starts, heap size is about 70 MB (that includes the virtual
>
>> applications for you, etc. That little rant out of the way...
>> I'd suggest launching a rails app and then pointing jconsole (included
>> best.
>> performance at all, so more RAM generally means more apps running on the
>> second, and it only has about 12MB of new live data per collection,
>>
>> many runtimes into the space that you have, and thus won't be able to
>> there to be used, and if you can go to maximal load and still have idle
>> the machine, if possible).
>>>> resident memory this way (JDK 1.6 64-bit, Server VM, Linux 2.6), which is a
>>> empirically, that I got far better memory usage out of running Redmine -
>>>
>> To unsubscribe from this list, please visit:
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Peter K Chan (Guest)
on 2009-04-19 22:56
(Received via mailing list)
I posted this before but it didn't seem to have made it to the mailing
list.

Consider the XX:MaxHeapFreeRatio option. It directs the JVM to return
unused memory to the OS. The only thing is that it cannot be used for
the concurrent GC.

Peter
Jacob K. (Guest)
on 2009-04-22 20:44
(Received via mailing list)
I've thrown things together into a blog at
http://blogs.sun.com/Jacobkessler/entry/four_guide...

Comments and such are welcome!

Jacob K. wrote:
>> On 4/16/09 10:48 AM, "Jacob K." <removed_email_address@domain.invalid> wrote:
>>> -Xms512m, for example, then Java will set the minimum heap size to
>>> generic test application idles at around 40MB per runtime, which is
>>> will improve startup (since resizing the heap forces a full garbage
>>> the
>>> Rails app (with images, AJAX, etc.) at a few thousand requests per
>>> app serving a few thousand requests per second, so you'll likely have
>>> three
>>> you had idle CPU time, since that means (potentially) twice as many
>>> </rant>
>>> marc wrote:
>>>>>
>>>>> etc. have on memory usage and performance.
>>>> Running under glassfish app server, usage would spiral to over
>>>> working on it, it rarely peaks over 40Mb. Performance is better too.
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
marc (Guest)
on 2009-04-24 16:57
(Received via mailing list)
Jacob K. said...
> I've thrown things together into a blog at
> http://blogs.sun.com/Jacobkessler/entry/four_guide...
>
> Comments and such are welcome!

Thank a lot. I'll give it a good read over the weekend.

--
Cheers,
Marc


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.