Forum: Ruby drb over pipes

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Reto S. (Guest)
on 2006-03-23 11:43
(Received via mailing list)
Hi

I'd like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes?  If not, is it easy to implement?

I've seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I'd like to do something like that:
  # ruby client.rb | ssh host 'ruby agent.rb'

Unfortunately using drb's tcp protocol directly isn't really an option
in
our environment.

Does anybody have a good idea?

Thanks!

Cheers,
Reto
Schüttel
Peter E. (Guest)
on 2006-03-23 11:47
(Received via mailing list)
> Unfortunately using drb's tcp protocol directly isn't really an option in
> our environment.

I would suggest using ssh-tunneling.

That way you have the security of secure shell
and the comfort of tcp sockets.
Peter E. (Guest)
on 2006-03-23 11:56
(Received via mailing list)
Yoann G. (Guest)
on 2006-03-23 12:01
(Received via mailing list)
On Thu, Mar 23, 2006 at 06:42:49PM +0900, Reto S. wrote:
>   # ruby client.rb | ssh host 'ruby agent.rb'
>

With drb, the client needs to talk to the agent and vice-versa, so this
command line will never work (the client will talk to the agent, but not
the
other way around).

> Unfortunately using drb's tcp protocol directly isn't really an option in
> our environment.
>
> Does anybody have a good idea?

As suggested, tunelling is probably the best/simpler choice.
Reto S. (Guest)
on 2006-03-23 12:24
(Received via mailing list)
Hi

On Thu, Mar 23, 2006 at 07:01:35PM +0900, Yoann G. wrote:
> On Thu, Mar 23, 2006 at 06:42:49PM +0900, Reto S. wrote:
> > At the end of the day I'd like to do something like that:
> >   # ruby client.rb | ssh host 'ruby agent.rb'
> >
>
> With drb, the client needs to talk to the agent and vice-versa, so this
> command line will never work (the client will talk to the agent, but not the
> other way around).

A pipe provides two streams (in & out), and tcp does the same. The
question is now if drb really needs two connections (-> 4 streams)?

> > Unfortunately using drb's tcp protocol directly isn't really an option in
> > our environment.
> >
> > Does anybody have a good idea?
>
> As suggested, tunelling is probably the best/simpler choice.

Tunneling may be a solution, although I'd prefer the pipe. Otherwise
there would be a daemon, somewhere listening on localhost:port.

Regards,
Reto
Schüttel
Peter E. (Guest)
on 2006-03-23 12:50
(Received via mailing list)
the 'daemon' is just ssh which you already have.
there will be no additional executables when you use tunneling,
just secure shell on port 22 (or whatever). tunneling is configured
in the client and can be allowed/forbidden on the server side.
Jonathan Paisley (Guest)
on 2006-03-23 14:35
(Received via mailing list)
On Thu, 23 Mar 2006 18:42:49 +0900, Reto S. wrote:


> I'd like to establish a drb connection over ssh, preferably over the
> stdin/stdout channels which are provided by ssh itself. Is there a
> drb protocol which uses pipes?  If not, is it easy to implement?
>
> I've seen that the tcpprotocol uses two connections, this could be
> problem, as pipes/stdin/stdout only provide one stream.
>
> At the end of the day I'd like to do something like that:
>   # ruby client.rb | ssh host 'ruby agent.rb'

I implemented a quick hack to do this. Hopefully the code below will
help.
It's a bit of a mess, but allows you to use a DRb scheme like
'drbfd://0,1' which means uses file descriptors 0 and 1 (stdin/stdout)
for
communication.

module DRb
  class DRbFDSocket
    def self.parse_uri(uri)
      if /^drbfd:\/\/(\d+),(\d+)(?:\?(.*))?$/ =~ uri
        [$1.to_i,$2.to_i, $3]
      else
        raise DRbBadScheme,uri unless uri =~ /^drbfd:/
        raise DRbBadURI, "can't parse uri: " + uri
      end
    end

    def self.uri_option(uri,config)
      infd,outfd, option = parse_uri(uri)
      return ["drbfd://#{infd},#{outfd}", option]
    end

    def self.open(uri,config)
      opts = parse_uri(uri)
      self.new(uri,opts,config)
    end

    def self.open_server(uri,config)
      opts = parse_uri(uri)
      self.new(uri,opts,config)
    end

    def initialize(uri,fds,config)
      @uri = uri
      @in_fp = IO.new(fds[0],"r")
      @out_fp = IO.new(fds[1],"w")
      @out_fp.sync = true

      @msg = DRbMessage.new(config)
    end

    attr_reader :uri

    def send_request(ref, msg_id, arg, b)
      begin
        result = @msg.send_request(@out_fp, ref, msg_id, arg, b)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
      result
    end

    def recv_request
      begin
        result = @msg.recv_request(@in_fp)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
      result
    end

    def send_reply(succ, result)
      begin
        result = @msg.send_reply(@out_fp, succ, result)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
      result
    end

    def recv_reply
      begin
        @msg.recv_reply(@in_fp)
      rescue DRb::DRbConnError
        Kernel.exit! 0
      end
    end

    def alive?
      true
    end

    def close
      @in_fp.close
      @out_fp.close
    end

    def accept
      if @accepted then
        sleep 60 while true
      else
        @accepted = true
        self
      end
    end

  end
  DRbProtocol.add_protocol(DRbFDSocket)
end
Robert K. (Guest)
on 2006-03-23 14:59
(Received via mailing list)
Jonathan Paisley wrote:
>> At the end of the day I'd like to do something like that:
>>   # ruby client.rb | ssh host 'ruby agent.rb'
>
> I implemented a quick hack to do this. Hopefully the code below will help

The code didn't make it into the newsgroup.  I can't detect something
strange about your original message - it's all there on my email
account.  This is strange...

Cheers

	robert
Jonathan Paisley (Guest)
on 2006-03-23 15:05
(Received via mailing list)
On Thu, 23 Mar 2006 21:58:52 +0900, Robert K. wrote:

> Jonathan Paisley wrote:
>> I implemented a quick hack to do this. Hopefully the code below will help
>
> The code didn't make it into the newsgroup.  I can't detect something
> strange about your original message - it's all there on my email
> account.  This is strange...

Very strange. I sent the message to the mailing list, where it has shown
up correctly in the archives [1] and been forwarded correctly as your
email account attests. I guess it must be something broken in the
gateway.

Jonathan

[1] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/...
Reto S. (Guest)
on 2006-03-23 20:09
(Received via mailing list)
Hi Peter

On Thu, Mar 23, 2006 at 07:49:56PM +0900, Peter E. wrote:
> the 'daemon' is just ssh which you already have.
> there will be no additional executables when you use tunneling,
> just secure shell on port 22 (or whatever). tunneling is configured
> in the client and can be allowed/forbidden on the server side.

Well.. yeah, there's no need for an additional daemon on the outside,
but a daemon at the beginning of the session would be needed (although
it could be shut down shortly later). Eg

$ ssh -L 4444:localhost:4444 'drb.rb drb://localhost:4444'

This would work quite well, but the drb port would be accessible by
local processes.

Reto S.
Reto S. (Guest)
on 2006-03-23 20:30
(Received via mailing list)
Hi Jonathan

On Thu, Mar 23, 2006 at 12:34:31PM +0000, Jonathan Paisley wrote:
> I implemented a quick hack to do this. Hopefully the code below will help.
> It's a bit of a mess, but allows you to use a DRb scheme like
> 'drbfd://0,1' which means uses file descriptors 0 and 1 (stdin/stdout) for
> communication.
>
> module DRb

That's great! exactly what I needed.

For the record: My statement in the first mail wasn't completely
correct, the pipe I used in my example is only a 'one-way' thing. But
ssh actually transports three streams/pipes: stdin, stdout and stderr,
so this
isn't really a problem.

There's one small problem with your fdsocket, it seems like the drb
server/thread doesn't always terminate itself when the streams get
closed. But I will try to investigate this by myself tomorrow :).

I've appended an example which uses your library. It creates two pipes,
forks, connects the pipes to the stdin/stdout of the child and then
execs
the ssh client. The same technique could also be used to communicate
with a child (instead of a unix/tcp socket or a simple plain pipe). IMHO
that's really nice, great work Jonathan!

require 'drb'
require 'fdsocket'

ctprd, ctpwr = IO.pipe
ptcrd, ptcwr = IO.pipe

pid = fork

unless pid
  # child
  ctprd.close
  ptcwr.close

  $stdin.reopen( ptcrd )
  $stdout.reopen( ctpwr )

  exec("ssh", "hostname", "./agent.rb")

  exit # child shounld't reach this point
end

# parent
ctpwr.close
ptcrd.close

fd_read =  ctprd.fileno
fd_write = ptcwr.fileno

# The URI to connect to
SERVER_URI="drbfd://#{fd_read},#{fd_write}"
DRb.start_service

obj = DRbObject.new_with_uri(SERVER_URI)

[...]


Regards,
Reto S.
Jonathan Paisley (Guest)
on 2006-03-25 15:03
(Received via mailing list)
On Fri, 24 Mar 2006 03:30:00 +0900, Reto S. wrote:

> There's one small problem with your fdsocket, it seems like the drb
> server/thread doesn't always terminate itself when the streams get
> closed. But I will try to investigate this by myself tomorrow :).

I encountered that too; hence all those exception handlers and calls to
exit. I worked around it by ensuring that my code made periodic calls
across the DRb connection --- this would trigger an exception.
Otherwise,
I ended up with loads of server processes doing nothing.

I'd be interested to hear if you've figured out how to get notification
of
the socket closing.

> I've appended an example which uses your library. It creates two pipes,
> forks, connects the pipes to the stdin/stdout of the child and then execs
> the ssh client. The same technique could also be used to communicate
> with a child (instead of a unix/tcp socket or a simple plain pipe). IMHO
> that's really nice, great work Jonathan!

I should have included an example with my code --- although it turns out
now that mine is practically the same as yours (I too am using ssh
transport). Sorry for omitting the extra code...

Jonathan
unknown (Guest)
on 2006-03-25 17:25
(Received via mailing list)
On Fri, 24 Mar 2006, Reto S. wrote:


> ptcrd, ptcwr = IO.pipe
>
> fd_write = ptcwr.fileno
>
> # The URI to connect to
> SERVER_URI="drbfd://#{fd_read},#{fd_write}"
> DRb.start_service
>
> obj = DRbObject.new_with_uri(SERVER_URI)
>
> [...]

i think this will fail with calls that use 'yield' or that return
'DrbUndumped' objects because they will attempt to open a connection
__from__
'hostname' to 'localhost'.

this doccument details this in the ssh section

   http://www.rubygarden.org/ruby?DrbTutorial

fyi.


-a
This topic is locked and can not be replied to.