Packet : A Pure Ruby Library for Event Driven Network Programming

Hi Folks,

I am pleased to announce “Packet”, A pure ruby library for Event
Driven Network Programming.

Packet is a pure ruby library for writing network applications in Ruby.
It follows Evented Model of network programming and implements almost
all the
features provided by EventMachine.

It also provides real easy to user UNIX workers for concurrent
programming.

Its best to have some examples going:

== Examples
=== A Simple Echo Server:
require “rubygems”
require “packet”

class Foo
def receive_data p_data
send_data(p_data)
end

def post_init
puts “Client connected”
end

def connection_completed
puts “Whoa man”
end

def unbind
puts “Client Disconnected”
end
end

Packet::Reactor.run do |t_reactor|
t_reactor.start_server(“localhost”,11006,Foo)
end

Those new to network programming with events and callbacks, will note
that,
each time a new client connects an instance of class Foo is
instantiated.
When client writes some data to the socket, receive_data method is
invoked.

Although Packet implements an API similar to EventMachine, but it
differs
slightly because of the fact that, for a packet app, there can be more
than one
reactor loop running and hence, we don’t use Packet.start_server(…).

=== A Simple Http Client
class WikiHandler
def receive_data p_data
p p_data
end

def post_init
end

def unbind
end

def connection_completed
send_data(“GET / \r\n”)
end
end

Packet::Reactor.run do |t_reactor|
t_reactor.connect(“en.wikipedia.org”,80,WikiHandler)
end

=== Using Callbacks and Deferables
Documentation to come.

=== Using Workers
Packet enables you to write simple workers, which will run in
different process and gives you nice
evented handle for concurrent execution of various tasks.

When, you are writing a scalable networking application
using Event Model of network programming,
sometimes when processing of certain events take time,
your event loop is stuck there. With green
threads, you don’t really have a way of paralleling
your request processing. Packet library, allows
you to write simple workers, for executing long
running tasks. You can pass data and callbacks as an
argument.

When you are going to use workers in
your application, you need to define
constant WORKER_ROOT,
which is the directory location, where
your workers are located. All the workers defined in that directory
will be automatically, picked and forked in a
new process when your packet app starts. So, a typical
packet_app, that wants to use workers, will look like this:

packet_app_root
|
|__ lib
|
|___ worker
|
|___ config
|
|___ log

You would define WORKER_ROOT = PACKET_APP_ROOT/worker

All the workers must inherit class Packet::Worker, and hence a
general skeleton of worker will look like:

class FooWorker < Packet::Worker
  set_worker_name :foo_worker #=> This is necessary.
  def receive_data p_data
  end

  def connection_completed
  end

  def unbind
  end

  def post_init
  end
end

All the forked workers are connected to master via
UNIX sockets, and hence messages passed to workers from master
will be available in receive_data method. Also,
when you are passing messages to workers, or worker is passing
message to master ( in a nutshell, all the internal
communication between workers and master ) directly takes
place using ruby objects. All the passed ruby objects are
dumped and marshalled across unix sockets in a non blocking
manner. BinParser class parses dumped binary objects and
makes sure, packets received at other end are complete.
Usually, you wouldn’t need to worry about this little detail.

Packet provides various ways of interacting with
workers. Usually, when a worker is instantiated, a proxy for
that worker will also be instantiated at master
process. Packet automatically provides a worker proxy(See
meta_pimp.rb)
for you, but if you need to multiplex/demultiplex
requests based on certain criteria, you may as well define your
own worker proxies. Code, would like something like this:

class FooWorker < Packet::Worker
  set_worker_proxy :foo_handler
end

When you define, :foo_handler as a proxy for
this worker, packet is gonna search for FooHandler class and
instantiate it when the worker gets started. All
the worker proxies must inherit from Packet::Pimp.
Have a look at, Packet::MetaPimp,
which acts as a meta pimp for all the workers,
which don’t have a explicit worker proxy defined.

=== A complete Case :

Just for kicks, lets write a sample server,
which evals whatever clients send to it. But, assuming this 'eval' 

of
client data can be potentially time/cpu
consuming ( not to mention dangerous too ), we are gonna ask our
eval_worker, to
perform eval and return the result to master process, which in
turn returns the result to happy client.

# APP_ROOT/bin/eval_server.rb
EVAL_APP_ROOT = File.expand_path(File.join(File.dirname(__FILE__) + 

“/…”))
[“bin”,“worker”,“lib”].each { |x| $LOAD_PATH.unshift(EVAL_APP_ROOT

  • “/#{x}”)}
    WORKER_ROOT = EVAL_APP_ROOT + “/worker”

    require “packet”
    class EvalServer
    def receive_data p_data
    ask_worker(:eval_worker,:data => p_data, :type => :request)
    end

    # will be called, when any worker sends data back to master 
    

process
# it should be noted that, you may have several instances of
eval_server in
# your master, for each connected client, but worker_receive
will be always
# be invoked for the instance, which originally made the request.
# If you need fine control, over this behaviour, you can
implement a worker proxy
# on the lines of meta_pimp class. This API will change in
future perhaps, as i
# expect, better ideas to come.
def worker_receive p_data
send_data “#{p_data[:data]}\n”
end

  def show_result p_data
    send_data("#{p_data[:response]}\n")
  end

  def connection_completed
  end

  def post_init
  end

  def wow
    puts "Wow"
  end
end

Packet::Reactor.run do |t_reactor|
  t_reactor.start_server("localhost", 11006,EvalServer) do 

|instance|
instance.wow
end
end

APP_ROOT/worker/eval_worker.rb

 class EvalWorker < Packet::Worker
 set_worker_name :eval_worker
 def worker_init
   p "Starting no proxy worker"
 end

 def receive_data data_obj
   eval_data = eval(data_obj[:data])
   data_obj[:data] = eval_data
   data_obj[:type] = :response
   send_data(data_obj)
 end

end

=== Passing file descriptors to workers :
Packet, allows this style of load
distribution as well, you can use method send_fd to
pass file descriptors to workers.
WARNING: This feature hasn’t been tested well.

=== Disable auto loading of certain workers:
Sometimes, you would need to start a
worker at runtime and don’t want this pre-forking mechanism.
Packet, allows this. You just need to define
“set_no_auto_load true” in your worker class and worker
will not be automatically forked. Although name is a bit misleading
perhaps.

Now, at runtime, you can call start_worker(:foo_worker, options)
to start a worker as usual. It should
be noted that, forking a worker, which is already
forked can be disastrous, since worker names are being
used as unique keys that represent a worker.

== Performance:
Although written in pure ruby, packet performs
reasonably well. Mongrel, running on top of Packet is a tad
slower than Mongrel running on top of EventMachine. More benchmarks
coming soon.

== Code:
SVN Code for packet is on google code, svn repo is:
http://packet.googlecode.com/svn/trunk/

Rubyforge URL:
http://rubyforge.org/projects/packet/

== Credits
Francis for awesome EventMachine lib, which has constantly acted as
an inspiration.
Ezra, for being a early user and porting mongrel to run on top of
packet.


Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

On Wed, 2007-11-21 at 04:14 +0900, hemant wrote:

It also provides real easy to user UNIX workers for concurrent programming.

Hemant,

In the eval server example, is a single worker process created,
corresponding to eval_worker.rb, or is it possible to create a pool of
pre-forked worker processes of the same class?

Andre

On Nov 21, 2007 1:25 AM, Andre N. [email protected] wrote:

In the eval server example, is a single worker process created,
corresponding to eval_worker.rb, or is it possible to create a pool of
pre-forked worker processes of the same class?

Andre,

Currently, for each defined worker, only one process is created. You
can’t have a pool of workers instantiated from class.
I am quite inclined to implement this feature and hence will be coming
soon. If you can provide a patch, it would be awesome.


Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

On Nov 20, 2007, at 1:14 PM, hemant wrote:

Hi Folks,

I am pleased to announce “Packet”, A pure ruby library for Event
Driven Network Programming.

Packet is a pure ruby library for writing network applications in
Ruby.
It follows Evented Model of network programming and implements
almost all the
features provided by EventMachine.

Did you ever consider working on the EventMachine project so it could
handle your requirements? While I’m happy to see more event-driven
programming, I’m puzzled why you re-invented so much of the wheel for
one or two features that EM didn’t have.

On Nov 21, 2007 5:56 AM, Chris C. [email protected] wrote:

Packet is a pure ruby library for writing network applications in

I think mainly that this is pure ruby, and event machine is a C++ backend.

I would have liked nothing more than having it in EventMachine. But as
you see the biggest differentiator is
ability to have process based workers, so I needed low level access of
sockets. Once started as proof of concept,
it took off on its own, and I didn’t see any harm in having a pure
ruby version.


Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

On Nov 20, 2007 6:09 PM, Chuck R. [email protected] wrote:

It follows Evented Model of network programming and implements
almost all the
features provided by EventMachine.

Did you ever consider working on the EventMachine project so it could
handle your requirements? While I’m happy to see more event-driven
programming, I’m puzzled why you re-invented so much of the wheel for
one or two features that EM didn’t have.

I think mainly that this is pure ruby, and event machine is a C++
backend.

Hi.
I was trying out “packet” tonight, but failed at the starting block with

c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:133:in
start_reactor': uninitialized constant Packet::Core::CommonMethods::UNIXSocket (NameError) from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:132:ineach’
from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:132:in
start_reactor' from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:127:inloop’
from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/core.rb:127:in
start_reactor' from c:/Ruby/lib/ruby/gems/1.8/gems/packet-0.1.0/lib/packet_master.rb:16:inrun’
from wikipacket.rb:23

I groped around and it seems there is no UNIXSocket support in my Ruby
installation (ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32])
on Windows XP.
Any idea what I’m doing wrong ?

Tad

Ezra Z. wrote:

Windows does not support UnixSockets. Therefor packet does not run on
windows.

It wouldn’t be hard to use Named Pipes instead, which are the Windows
equivalent.

Clifford H…

On Dec 5, 2007 8:15 AM, Clifford H. [email protected] wrote:

Ezra Z. wrote:

  Windows does not support UnixSockets. Therefor packet does not run on

windows.

It wouldn’t be hard to use Named Pipes instead, which are the Windows equivalent.

Never used them, but I will have a look at see, what can be done about
Windows support.


Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

On Dec 4, 2007, at 2:22 PM, Tadeusz B. wrote:

`start_reactor’
installation (ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32])
on Windows XP.
Any idea what I’m doing wrong ?

Tad


Posted via http://www.ruby-forum.com/.

Windows does not support UnixSockets. Therefor packet does not run on
windows.

Cheers-

On Dec 5, 2007 3:06 AM, hemant [email protected] wrote:

Never used them, but I will have a look at see, what can be done about

You do know that there is a maintained pure ruby version of EventMachine
sitting right next to the C++ extension, right?

Jason

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs