Forum: Ruby NetworkFacade 0.4

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Florent S. (Guest)
on 2007-05-09 12:28
NetworkFacade is an alternative to DRb, XML/RPC and other REST api,
all in one gem !

  http://network-facade.rubyforge.org/

The easiest way to describe it, is through examples.

A simple tcp client/server :

  # -- Server --
  class Foo
    def bar; 42; end
  end
  server = NetworkFacade::Server.new(:port => 5042)
  server << Foo.new
  server.start

  # -- Client --
  class Foo < NetworkFacade::Client 'nf://localhost:5042'
  end
  f = Foo.new
  puts f.bar

A REST client, for Digg :

  class Digg < NetworkFacade::REST::Client 'http://services.digg.com'
    mapping :apple_stories, '/stories/topic/apple'
    mapping :me, '/user/pyros'
  end

  d = Digg.new
  puts d.apple_stories(:appkey => 'http://network-facade.rubyforge.org',
                       :count => 3, :type => :json)
  puts d.me(:appkey => 'http://network-facade.rubyforge.org')

Many other examples (Unix socket, SSL, Compression, Flickr,
Remember the milk, Netvibes) are explained here :

  http://network-facade.rubyforge.org/doc

--
Florent S.
removed_email_address@domain.invalid
Felipe C. (Guest)
on 2007-05-09 13:03
(Received via mailing list)
On 5/9/07, Florent S. <removed_email_address@domain.invalid> wrote:
>   class Foo
>   puts f.bar
>                        :count => 3, :type => :json)
>   puts d.me(:appkey => 'http://network-facade.rubyforge.org')
>
> Many other examples (Unix socket, SSL, Compression, Flickr,
> Remember the milk, Netvibes) are explained here :
>
>   http://network-facade.rubyforge.org/doc

Sweet!

I did something similar:
http://snipplr.com/view/2476/ruby-rest/

But I definitely like your :mapping thing much better.

I'm still not sure if I would like the whole Network Facade just for
REST, but maybe it makes sense for XML/RPC or DRb.

Good job.
Florent S. (Guest)
on 2007-05-09 13:06
Felipe C. wrote:
>
> I'm still not sure if I would like the whole Network Facade just for
> REST, but maybe it makes sense for XML/RPC or DRb.

You know, network-facade is just a 15k lib :)

>
> Good job.

Thanks
Brian C. (Guest)
on 2007-05-09 15:57
(Received via mailing list)
On Wed, May 09, 2007 at 05:28:56PM +0900, Florent S. wrote:
> Many other examples (Unix socket, SSL, Compression, Flickr,
> Remember the milk, Netvibes) are explained here :
>
>   http://network-facade.rubyforge.org/doc

I can't find any documentation of the TCP protocol itself.

Looking at the code, the call appears to be a four-byte length followed
by
a Marshal dump of
    [@uri.path[1..-1], name, args]
and the response is a 4-byte length followed by a Marshal dump of a
single
object. This means that the protocol is similar to DRb, but different,
i.e.
not compatible :-(

I can see a problem with this protocol: it's impossible to distinguish
between a method returning an Exception object and a method raising an
exception.

There are certainly problems which DRb has which could do with solving.
Chief in my mind these are:

(1) Doesn't run easily over FastCGI, IO.popen and stdin/stdout, or Unix
domain sockets.

(2) Doesn't allow bi-directional method calls through a single
connection,
e.g. for use through a firewall and/or NAT and port forwarding.

(3) Doesn't interoperate with non-Ruby applications. I would like to be
able
to carry strings and integers as plain strings. This would make it easy
to
interoperate with Perl, say.

(4) Not easy to secure.

(5) Large and complex implementation, not easy to understand, especially
with regards to connection caching and what happens if connections are
dropped.

Your library addresses (4), (5) and part of (1). But I think (2) and (3)
need careful thinking to produce a suitable protocol. As for (2), I'm
not
sure your protocol supports proxy objects at all, a.k.a. DRbUndumped,
but
please correct me if I'm wrong.

I think a replacement protocol for DRb should have the following
characteristics:

* 'Requests' and 'Responses' explicitly marked as such, so that calls
can be
made in both directions down the same socket.

* Each Request and Response to be tagged with an ID, so multiple
overlapping
calls can be multiplexed onto the same socket, meaning that only a
single
TCP connection needs to be kept open between a particular pair of hosts.

* Ability to pass Undumped objects in such a way that the callback is
either
made back down the same socket, or to a global URI (the latter is what
DRb
does). The latter also allows the callback to be made independently, and
avoids forming long chains of tunnels. (+)

* Basic protocol uses only string encoding, allowing tiny implementation
suitable for embedded devices etc, and cross-language interoperability.

* Optionally enable other native marshalling protocols (e.g. Marshall,
YAML,
Perl Storable etc) if understood by both sides. Ideally negotiated.

* Optional SASL authentication as well as, or instead of, SSL. This
allows
basic username/password authentication (AUTH PLAIN) which is simpler to
configure than client certificates.

* Optional synchronous mode (one request -> one response) for tunneling
over
HTTP or FastCGI and for simplistic implementations in embedded devices

* Automatic establishment or re-establishment of dropped connections

I also think that implementations should consider using an opaque key to
identify each object, rather than it's object_id. For one thing,
object_id's
can be recycled; for another, it prevents people probing for random
objects
which have not been explicitly shared. Keeping a mapping table of
opaque_key
=> object will prevent DRbUndumped objects from being garbage collected
(although this may or may not be desirable)

Just a few thoughts.

Regards,

Brian.

(+) It would be useful to pass a different global URI depending on where
you
are connecting to. For example, if you are connecting outbound through
the
Internet, the global URI you expose may be your firewalls' outside IP or
hostname and a port on the firewall which forwards inbound connections.

To make this more transparent, it could be useful to auto-detect NAT.
This
could be done by passing your *actual* source IP and port inside the
connection message; the far end would compare these to the *seen* source
IP
and port. IPSEC NAT traversal uses a similar mechanism.
Florent S. (Guest)
on 2007-05-09 17:11
Before anything else, thanks for your long and interesting reply.

> I can't find any documentation of the TCP protocol itself.

The documentation is still incomplete, I will wait some
versions to improve it.

>
> Looking at the code, the call appears to be a four-byte length followed
> by
> a Marshal dump of
>     [@uri.path[1..-1], name, args]
> and the response is a 4-byte length followed by a Marshal dump of a
> single
> object. This means that the protocol is similar to DRb, but different,
> i.e.
> not compatible :-(

It's true. I don't try to do a compatible protocol, because I think it's
difficult to add some new ideas with the same protocol.

>
> I can see a problem with this protocol: it's impossible to distinguish
> between a method returning an Exception object and a method raising an
> exception.

Also true. I do not realize that. I will try to fix it.

> There are certainly problems which DRb has which could do with solving.
> Chief in my mind these are:
>
> (1) Doesn't run easily over FastCGI, IO.popen and stdin/stdout, or Unix
> domain sockets.
>
> (2) Doesn't allow bi-directional method calls through a single
> connection,e.g. for use through a firewall and/or NAT and port forwarding.
>
> (3) Doesn't interoperate with non-Ruby applications. I would like to be
> able to carry strings and integers as plain strings. This would make it
> easy to interoperate with Perl, say.
>
> (4) Not easy to secure.
>
> (5) Large and complex implementation, not easy to understand, especially
> with regards to connection caching and what happens if connections are
> dropped.
>
> Your library addresses (4), (5) and part of (1). But I think (2) and (3)
> need careful thinking to produce a suitable protocol. As for (2), I'm
> not sure your protocol supports proxy objects at all, a.k.a.
> DRbUndumped, but please correct me if I'm wrong.
>
> I think a replacement protocol for DRb should have the following
> characteristics:
>
> * 'Requests' and 'Responses' explicitly marked as such, so that calls
> can be made in both directions down the same socket.

I will do this, you are right.

> * Each Request and Response to be tagged with an ID, so multiple
> overlapping calls can be multiplexed onto the same socket, meaning
> that only a single TCP connection needs to be kept open
> between a particular pair of hosts.

Very interesting too, I will implement this too.

> * Ability to pass Undumped objects in such a way that the callback
> is either made back down the same socket, or to a global URI (the
> latter is what DRb does). The latter also allows the callback to
> be made independently, and avoids forming long chains of tunnels. (+)

I don't realy understand this point, my english is not perfect :)
Could you explain it, thanks.

> * Basic protocol uses only string encoding, allowing tiny implementation
> suitable for embedded devices etc, and cross-language interoperability.
>
> * Optionally enable other native marshalling protocols (e.g. Marshall,
> YAML, Perl Storable etc) if understood by both sides. Ideally negotiated.

I already thought about that, and I would like to release this in the
next
version (0.5). I would like to have these serialization methods :
  - YAML
  - JSON
  - XML
  - Marshall-Ruby

And in a second time :
  - PHP
  - Perl
  - String encoding

I already don't know some problem, for example, how to serialize a ruby
exception in php format :)

> * Optional SASL authentication as well as, or instead of, SSL. This
> allows basic username/password authentication (AUTH PLAIN) which is
> simpler to configure than client certificates.

It could be a great idea but I'm not familiar with SASL, I will take a
look
at some docs.

> * Optional synchronous mode (one request -> one response) for tunneling
> over HTTP or FastCGI and for simplistic implementations in embedded devices

I will think about this too, I could be cool, but I don't know if I will
be
able to merge this with the current code.

> * Automatic establishment or re-establishment of dropped connections

Already planned :)

> I also think that implementations should consider using an opaque key to
> identify each object, rather than it's object_id. For one thing,
> object_id's can be recycled; for another, it prevents people probing
> for random objects which have not been explicitly shared. Keeping a
> mapping table of opaque_key => object will prevent DRbUndumped objects
> from being garbage collected (although this may or may not be desirable)

object_id is just used with the logger (through client_id method) to
print
something revelant to the user, and for example, it's overloaded by
TCP::Server. So, if I'm not wrong, it's not necessary.

> Just a few thoughts.

Thanks again for your thoughts.

>
> Regards,
>
> Brian.
>
> (+) It would be useful to pass a different global URI depending on where
> you are connecting to. For example, if you are connecting outbound through
> the Internet, the global URI you expose may be your firewalls' outside IP or
> hostname and a port on the firewall which forwards inbound connections.
>
> To make this more transparent, it could be useful to auto-detect NAT.
> This could be done by passing your *actual* source IP and port inside the
> connection message; the far end would compare these to the *seen* source
> IP and port. IPSEC NAT traversal uses a similar mechanism.

This could be great, but not for the moment. I will wait a stable and
clean
API before doing this kind of feature.

Best regards.
Brian C. (Guest)
on 2007-05-09 18:56
(Received via mailing list)
On Wed, May 09, 2007 at 10:11:44PM +0900, Florent S. wrote:
> > * Ability to pass Undumped objects in such a way that the callback
> > is either made back down the same socket, or to a global URI (the
> > latter is what DRb does). The latter also allows the callback to
> > be made independently, and avoids forming long chains of tunnels. (+)
>
> I don't realy understand this point, my english is not perfect :)
> Could you explain it, thanks.

Firstly, do you understand DRbUndumped? It took me a while to work this
out,
and I ended up writing it up on the web. See
http://wiki.rubygarden.org/Ruby/page/show/DRbTutorial
and see if that makes sense.

In particular, skip down to section headed "Why does the client run
'DRb.start_service'?"

>   - JSON
>   - XML
>   - Marshall-Ruby
>
> And in a second time :
>   - PHP
>   - Perl
>   - String encoding
>
> I already don't know some problem, for example, how to serialize a ruby
> exception in php format :)

As the lowest common denominator, I would encode an exception as a
string.
It's pretty much impossible to map arbitary PHP exceptions to Ruby
exceptions and vice versa, so on receiving one of these exceptions you'd
create a generic exception, say NetworkFacade::RemoteError, with the
message
containing the string.

Or maybe you could return an array of [class, message, backtrace].

If the two ends agree on Ruby Marshal, then the exception can be sent in
this format. You might still have to turn this into a generic
RemoteError,
e.g. if the exception class contained in the marshalled object is not
known
at the client side.

> > * Optional SASL authentication as well as, or instead of, SSL. This
> > allows basic username/password authentication (AUTH PLAIN) which is
> > simpler to configure than client certificates.
>
> It could be a great idea but I'm not familiar with SASL, I will take a
> look
> at some docs.

RFC 2222 is the base SASL document, but it's not very helpful. A good
starting point is to see how CRAM-MD5 is done in IMAP (RFC 2195) and
then
look at the PLAIN method (RFC 2595). This brings SASL back down to the
level
of a simple username and password login :-)

It doesn't have to be SASL of course, but this is the official
"extensible"
authentication mechanism for protocols. In the end, having a simple
username/password or shared secret authentication on top of TLS is what
I'd
like to see.

> TCP::Server. So, if I'm not wrong, it's not necessary.
Some sort of object_id is needed when making callbacks - but we are back
to
the issue of DRbUndumped.

BTW I think if you're going to use your existing library as a starting
point
and start modifying it, then you'll need at least a protocol version
field
on connection :-)

I think many or all the options - SSL, Zlib, Marshal, Pipeline
(overlapping
requests down the same socket) etc - could be offered as bits in a
bitmap.
Some of them are conflicting, e.g. the server will not choose both SSL
and
Zlib simultaneously.

Another thought: when using SSL, the client should negotiate SSL so that
the
same port can be used for SSL and non-SSL connections. Also it should
send
the hostname it connected to in the initial connection message, before
SSL
is turned on. This enables 'virtual hosting' where the server can choose
the
correct one out of many certificates. You might not think this is
important,
but it's something which can easily be gotten right if you design it in
at
the start.

RFC 2817 describes a HTTP extension for this. Unfortunately, because it
came
along late, no-one implements it so we're stuck with having separate
ports
for HTTP and HTTPS :-(

Regards,

Brian.
Tomas P.'s Mailing L. (Guest)
on 2007-05-09 23:59
(Received via mailing list)
On Wed, 9 May 2007, Brian C. wrote:

> I think many or all the options - SSL, Zlib, Marshal, Pipeline (overlapping
> requests down the same socket) etc - could be offered as bits in a bitmap.
> Some of them are conflicting, e.g. the server will not choose both SSL and
> Zlib simultaneously.

If you, Florent, choose to implement this, and implement this as a
bitmap, then take care not to make the bitmap too short or fixed length
since the "negotiable features" will be limited then.
*t

--
Brian C. (Guest)
on 2007-05-10 01:03
(Received via mailing list)
On Thu, May 10, 2007 at 04:59:22AM +0900, Tomas P.'s Mailing L.
wrote:
> On Wed, 9 May 2007, Brian C. wrote:
>
> >I think many or all the options - SSL, Zlib, Marshal, Pipeline (overlapping
> >requests down the same socket) etc - could be offered as bits in a bitmap.
> >Some of them are conflicting, e.g. the server will not choose both SSL and
> >Zlib simultaneously.
>
> If you, Florent, choose to implement this, and implement this as a
> bitmap, then take care not to make the bitmap too short or fixed length
> since the "negotiable features" will be limited then.

Nah. You just make bit 31 be the "extended options" option :-)

I'm keen to see the protocol be lightweight and simple to parse.
Florent S. (Guest)
on 2007-05-10 12:21
Tomas P.'s Mailing L. wrote:
> On Wed, 9 May 2007, Brian C. wrote:
>
> If you, Florent, choose to implement this, and implement this as a
> bitmap, then take care not to make the bitmap too short or fixed length
> since the "negotiable features" will be limited then.
> *t
>

For the next release (0.5) I will focus on these features :
  - Exception confusion
  - Auto reconnection
  - Request/Response container with uniq ID
  - Pipe support (not something over a socket, but popen)
  - Serialization abstraction
      - JSON support (via json gem)
      - YAML support (via stdlib)
      - XML support (via a new gem I will wrote)

After that, I will come back and talk here about SASL, NAT traversal,
pipeline and tunnel over HTTP features.

Thanks again for your comments. If you want to contribute or patch the
lib, you are welcome.
Florent S. (Guest)
on 2007-05-10 12:22
Brian C. wrote:
> On Thu, May 10, 2007 at 04:59:22AM +0900, Tomas P.'s Mailing L.
> wrote:
>
> Nah. You just make bit 31 be the "extended options" option :-)
>
> I'm keen to see the protocol be lightweight and simple to parse.

I would like to keep the protocol K.I.S.S as much as possible.
List R. (Guest)
on 2007-05-14 05:39
(Received via mailing list)
Awesome, Thanks Florent.  A++ effort! :-)
This topic is locked and can not be replied to.