Drb over pipes


#1

Hi

I’d like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes? If not, is it easy to implement?

I’ve seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I’d like to do something like that:

ruby client.rb | ssh host ‘ruby agent.rb’

Unfortunately using drb’s tcp protocol directly isn’t really an option
in
our environment.

Does anybody have a good idea?

Thanks!

Cheers,
Reto
Schüttel


#2

some
links:
http://www.ssh.com/support/documentation/online/ssh/winhelp/32/Tunneling_Explained.html
http://www.rzg.mpg.de/networking/tunnelling.html


#3

Unfortunately using drb’s tcp protocol directly isn’t really an option in
our environment.

I would suggest using ssh-tunneling.

That way you have the security of secure shell
and the comfort of tcp sockets.


#4

On Thu, Mar 23, 2006 at 06:42:49PM +0900, Reto S. wrote:

ruby client.rb | ssh host ‘ruby agent.rb’

With drb, the client needs to talk to the agent and vice-versa, so this
command line will never work (the client will talk to the agent, but not
the
other way around).

Unfortunately using drb’s tcp protocol directly isn’t really an option in
our environment.

Does anybody have a good idea?

As suggested, tunelling is probably the best/simpler choice.


#5

Hi

On Thu, Mar 23, 2006 at 07:01:35PM +0900, Yoann G. wrote:

On Thu, Mar 23, 2006 at 06:42:49PM +0900, Reto S. wrote:

At the end of the day I’d like to do something like that:

ruby client.rb | ssh host ‘ruby agent.rb’

With drb, the client needs to talk to the agent and vice-versa, so this
command line will never work (the client will talk to the agent, but not the
other way around).

A pipe provides two streams (in & out), and tcp does the same. The
question is now if drb really needs two connections (-> 4 streams)?

Unfortunately using drb’s tcp protocol directly isn’t really an option in
our environment.

Does anybody have a good idea?

As suggested, tunelling is probably the best/simpler choice.

Tunneling may be a solution, although I’d prefer the pipe. Otherwise
there would be a daemon, somewhere listening on localhost:port.

Regards,
Reto
Schüttel


#6

the ‘daemon’ is just ssh which you already have.
there will be no additional executables when you use tunneling,
just secure shell on port 22 (or whatever). tunneling is configured
in the client and can be allowed/forbidden on the server side.


#7

On Thu, 23 Mar 2006 21:58:52 +0900, Robert K. wrote:

Jonathan Paisley wrote:

I implemented a quick hack to do this. Hopefully the code below will help

The code didn’t make it into the newsgroup. I can’t detect something
strange about your original message - it’s all there on my email
account. This is strange…

Very strange. I sent the message to the mailing list, where it has shown
up correctly in the archives [1] and been forwarded correctly as your
email account attests. I guess it must be something broken in the
gateway.

Jonathan

[1] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/185525


#8

On Thu, 23 Mar 2006 18:42:49 +0900, Reto S. wrote:

I’d like to establish a drb connection over ssh, preferably over the
stdin/stdout channels which are provided by ssh itself. Is there a
drb protocol which uses pipes? If not, is it easy to implement?

I’ve seen that the tcpprotocol uses two connections, this could be
problem, as pipes/stdin/stdout only provide one stream.

At the end of the day I’d like to do something like that:

ruby client.rb | ssh host ‘ruby agent.rb’

I implemented a quick hack to do this. Hopefully the code below will
help.
It’s a bit of a mess, but allows you to use a DRb scheme like
‘drbfd://0,1’ which means uses file descriptors 0 and 1 (stdin/stdout)
for
communication.

module DRb
class DRbFDSocket
def self.parse_uri(uri)
if /^drbfd://(\d+),(\d+)(?:?(.*))?$/ =~ uri
[$1.to_i,$2.to_i, $3]
else
raise DRbBadScheme,uri unless uri =~ /^drbfd:/
raise DRbBadURI, "can’t parse uri: " + uri
end
end

def self.uri_option(uri,config)
  infd,outfd, option = parse_uri(uri)
  return ["drbfd://#{infd},#{outfd}", option]
end

def self.open(uri,config)
  opts = parse_uri(uri)
  self.new(uri,opts,config)
end

def self.open_server(uri,config)
  opts = parse_uri(uri)
  self.new(uri,opts,config)
end

def initialize(uri,fds,config)
  @uri = uri
  @in_fp = IO.new(fds[0],"r")
  @out_fp = IO.new(fds[1],"w")
  @out_fp.sync = true

  @msg = DRbMessage.new(config)
end

attr_reader :uri

def send_request(ref, msg_id, arg, b)
  begin
    result = @msg.send_request(@out_fp, ref, msg_id, arg, b)
  rescue DRb::DRbConnError
    Kernel.exit! 0
  end
  result
end

def recv_request
  begin
    result = @msg.recv_request(@in_fp)
  rescue DRb::DRbConnError
    Kernel.exit! 0
  end
  result
end

def send_reply(succ, result)
  begin
    result = @msg.send_reply(@out_fp, succ, result)
  rescue DRb::DRbConnError
    Kernel.exit! 0
  end
  result
end

def recv_reply
  begin
    @msg.recv_reply(@in_fp)
  rescue DRb::DRbConnError
    Kernel.exit! 0
  end
end

def alive?
  true
end

def close
  @in_fp.close
  @out_fp.close
end

def accept
  if @accepted then
    sleep 60 while true
  else
    @accepted = true
    self
  end
end

end
DRbProtocol.add_protocol(DRbFDSocket)
end


#9

Jonathan Paisley wrote:

At the end of the day I’d like to do something like that:

ruby client.rb | ssh host ‘ruby agent.rb’

I implemented a quick hack to do this. Hopefully the code below will help

The code didn’t make it into the newsgroup. I can’t detect something
strange about your original message - it’s all there on my email
account. This is strange…

Cheers

robert

#10

Hi Peter

On Thu, Mar 23, 2006 at 07:49:56PM +0900, Peter E. wrote:

the ‘daemon’ is just ssh which you already have.
there will be no additional executables when you use tunneling,
just secure shell on port 22 (or whatever). tunneling is configured
in the client and can be allowed/forbidden on the server side.

Well… yeah, there’s no need for an additional daemon on the outside,
but a daemon at the beginning of the session would be needed (although
it could be shut down shortly later). Eg

$ ssh -L 4444:localhost:4444 ‘drb.rb drb://localhost:4444’

This would work quite well, but the drb port would be accessible by
local processes.

Reto S.


#11

Hi Jonathan

On Thu, Mar 23, 2006 at 12:34:31PM +0000, Jonathan Paisley wrote:

I implemented a quick hack to do this. Hopefully the code below will help.
It’s a bit of a mess, but allows you to use a DRb scheme like
‘drbfd://0,1’ which means uses file descriptors 0 and 1 (stdin/stdout) for
communication.

module DRb

That’s great! exactly what I needed.

For the record: My statement in the first mail wasn’t completely
correct, the pipe I used in my example is only a ‘one-way’ thing. But
ssh actually transports three streams/pipes: stdin, stdout and stderr,
so this
isn’t really a problem.

There’s one small problem with your fdsocket, it seems like the drb
server/thread doesn’t always terminate itself when the streams get
closed. But I will try to investigate this by myself tomorrow :).

I’ve appended an example which uses your library. It creates two pipes,
forks, connects the pipes to the stdin/stdout of the child and then
execs
the ssh client. The same technique could also be used to communicate
with a child (instead of a unix/tcp socket or a simple plain pipe). IMHO
that’s really nice, great work Jonathan!

require ‘drb’
require ‘fdsocket’

ctprd, ctpwr = IO.pipe
ptcrd, ptcwr = IO.pipe

pid = fork

unless pid

child

ctprd.close
ptcwr.close

$stdin.reopen( ptcrd )
$stdout.reopen( ctpwr )

exec(“ssh”, “hostname”, “./agent.rb”)

exit # child shounld’t reach this point
end

parent

ctpwr.close
ptcrd.close

fd_read = ctprd.fileno
fd_write = ptcwr.fileno

The URI to connect to

SERVER_URI=“drbfd://#{fd_read},#{fd_write}”
DRb.start_service

obj = DRbObject.new_with_uri(SERVER_URI)

[…]

Regards,
Reto S.


#12

On Fri, 24 Mar 2006, Reto S. wrote:

ptcrd, ptcwr = IO.pipe

fd_write = ptcwr.fileno

The URI to connect to

SERVER_URI=“drbfd://#{fd_read},#{fd_write}”
DRb.start_service

obj = DRbObject.new_with_uri(SERVER_URI)

[…]

i think this will fail with calls that use ‘yield’ or that return
‘DrbUndumped’ objects because they will attempt to open a connection
from
‘hostname’ to ‘localhost’.

this doccument details this in the ssh section

http://www.rubygarden.org/ruby?DrbTutorial

fyi.

-a


#13

On Fri, 24 Mar 2006 03:30:00 +0900, Reto S. wrote:

There’s one small problem with your fdsocket, it seems like the drb
server/thread doesn’t always terminate itself when the streams get
closed. But I will try to investigate this by myself tomorrow :).

I encountered that too; hence all those exception handlers and calls to
exit. I worked around it by ensuring that my code made periodic calls
across the DRb connection — this would trigger an exception.
Otherwise,
I ended up with loads of server processes doing nothing.

I’d be interested to hear if you’ve figured out how to get notification
of
the socket closing.

I’ve appended an example which uses your library. It creates two pipes,
forks, connects the pipes to the stdin/stdout of the child and then execs
the ssh client. The same technique could also be used to communicate
with a child (instead of a unix/tcp socket or a simple plain pipe). IMHO
that’s really nice, great work Jonathan!

I should have included an example with my code — although it turns out
now that mine is practically the same as yours (I too am using ssh
transport). Sorry for omitting the extra code…

Jonathan