Ruby-dipus 0.1.2

DIPUS - Distributed IPC by Proxying UNIX Sockets

Sockets with user authentication, file permissions, encryption,
service discovery and network transparent addressing.

Using services:

host1$ dcat lame-svc foo.wav | dcat madplay.host3
host1$ dcat sep://example.com/image/jpeg/to/image/png foo.jpeg | dcat
ssh://[email protected]/image/png/to/text/plain | less

Creating services:

host3$ dipus-local-pipe ‘madplay -’ madplay
host2$ dipus-export-pipe ‘lame - -’ lame-svc

Some performance numbers:

Local throughput around 50MBps (or 100MBps, depending whether you use
socat as a client or not), unencrypted network throughput 9.8MBps in a
100Mbps LAN, CPU-limited when using SSH (my other computer is slow.)

In what languages can I use and create services?

Anything that supports a) UNIX domain sockets, and b) listing
filesystem directories. So, just about every language available. If
you install the helper scripts in bin, you don’t even need those, just
the ability to read from stdin and write to stdout.

Download: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2.tar.gz
RDocs: http://www.cs.helsinki.fi/u/irkheikk/dipus-rdoc/
Darcs repo: http://dark.fhtr.org/repos/ruby-dipus

Readme: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/readme.txt
Design:
http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-design.txt
Protocol spec:
http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-specification.txt

^^^^^^^^^^^^^^^^^^^^^^^^^^

Hi, designed and implemented a distributed IPC system that takes
lessons from UNIX (keep it simple) and HTTP (keep it simple.) Opinions
on the design or anything else? I’m dying to get some feedback,
writing stuff in a bubble can take you only so far.

And, now, with the first version of this December project out of the
way, I’m off to librend land~

Ilmari H.

vvvvvvvvvvvvvvvvvvvvvvvvvv

What is this for?

You have a program. You want to access it remotely with the minimum
hassle. It’d be nice to access it efficiently locally too. Without
rewriting your connection code.

And it’d be nice to put some connections over SSH. And be notified
when a service dies or a new one is created. And not expose any
of the individual services to the network, or even to other users
on the machine running them. And it’d be cool to be able to have a
different name and protocol for each service without having to shoehorn
FTP over XML-RPC. Oh, also, display a real-time list of services
running on the local network and on explicitly defined machines that
aren’t on the local network. And it’d be especially cool if all the
software could just use existing libraries like sockets and files.

So, take UNIX sockets and sprinkle some sugar to make them usable for
distributed IPC.

Supported systems

Should work on Linux and *BSD, likely on OS X too.
No Windows support, although it’s theoretically possible.

Required

Ruby version 1.8.2+ (one released in 2005)
cat

Recommended

OpenSSH version > 3.9 (for connection sharing support (ControlPath))
ssh-agent setup (for maximum convenience)
socat
More than one computer.

How does it work?

To create a service, make a listening socket in a subdir of
/tmp/dipus, so that the path is like:
/tmp/dipus/exported/audio/x-wav/to/audio/mpeg/lame.mybox…0

To access a service, open its socket.

To make a service only accessible to members of group ‘audio’:
chgrp audio socket_name
chmod 770 socket_name

Listing services that you can see:
tree /tmp/dipus

DIPUS provides helper methods and daemons to manipulate the
sockets in /tmp/dipus, e.g. create proxies to remote sockets,
delete dead sockets, provide network access to exported
sockets, find sockets by their service name, hostname, or
protocol.

Quick usage example

(optional) start dipusd to automatically discover and import services,

export services, and delete dead sockets.

[kig@desktop] dipusd
[kig@encode] dipusd

Create local madplay service for playing MP3s on

the computer connected to speakers.

The first argument is the command to run for each incoming connection.

The second argument is service_name/protocol, e.g. my_www/http.

The following command creates a socket with the path

/tmp/dipus/local/audio/mpeg/to/audio/out/madplay.stereo…0

and runs ‘madplay -’ for each incoming connection, setting the

connection to madplay’s stdin and stdout.

[kig@stereo] dipus-local-pipe ‘madplay -’
madplay/audio/mpeg/to/audio/out

Create an exported lame service for encoding wavs to MP3s.

[kig@encode] dipus-export-pipe ‘lame - -’ lame/audio/x-wav/to/audio/mpeg

Manually import the services from stereo.

Import over SSH to gain access to all services and not just exported

ones.

[kig@desktop] dipus-import ssh://stereo

Dipusd should’ve discovered and imported the lame service from encode,

so we don’t need to manually import that. But if that didn’t happen,

we

can import it by doing dipus-import sep://encode, sep being

Socket Export Protocol (see doc/dipus-specification.txt)

Now, let’s pipe some wavs to the stereo.

First encode the wav to MP3, using any service with a protocol that

says

that the service converts audio/x-wav to audio/mpeg.

Play using a service named madplay running on stereo.

[kig@desktop] dcat audio/x-wav/to/audio/mpeg *.wav | dcat madplay.stereo

We could also access the services without importing them by giving the

service URI as the service name.

Here’s the above piping example using service URIs.

Dcat lists the services matching the URI and uses the first match.

[kig@desktop] dcat sep://encode/audio/x-wav/to/audio/mpeg *.wav |
dcat
ssh://stereo/audio/mpeg/to/audio/out/madplay.stereo…0

List of DIPUS commands

Pass --help as an argument to get the full USAGE.

  • dls – list services
    dls sep://foobar/audio
    dls ssh://barfoo

  • dcat – pipe stdin and/or given files to a DIPUS socket
    dcat lame foo.wav bar.wav > foobar.mp3
    dcat ssh://foobar/image/png/to/image/jpeg cat.png > cat.jpg

  • dipus-import – imported all found service from an URI
    dipus-import ssh://foobar/audio
    dipus-import sep://barfoo

  • dipusd – start exporter, importer, advertiser, discoverer,
    dead socket monitor, and default services

  • dipus-exporter – start exporter

  • dipus-discoverer – discover exporters on the LAN and print
    their addresses

  • dipus-service-monitor – monitor for service changes on localhost,
    used by SSH importers

  • dipus-ssh-importer – import services over SSH to user’s DIPUS
    subdirectory (/tmp/dipus/username), can use
    discovery to find exporter-running hosts on
    localnet

  • dipus-local-pipe – create a local DIPUS service that forks the
    given command for the connection
    dipus-local-pipe ‘head’ head/text

  • dipus-export-pipe – create an exported DIPUS service like above
    dipus-export-pipe ‘madplay -’ madplay/audio/mpeg/to/audio/out

Using socat to create and connect to DIPUS services

proxy a web server

mkdir -p /tmp/dipus/exported/http
socat UNIX-LISTEN:/tmp/dipus/exported/http/foo.foobar…0,fork
TCP4-CONNECT:foo.example.com:80
rm /tmp/dipus/exported/http/foo.foobar…0

using a lame service to encode a wav

socat foo.wav
UNIX-CONNECT:/tmp/dipus/local/audio/x-wav/to/audio/mpeg/lame.foobar…0

> foo.mp3

or by using dls to find the filename

socat foo.wav UNIX-CONNECT:dls -1 -f lame > foo.mp3

proxy a DIPUS service to a TCP socket

socat TCP4-LISTEN:1234,fork EXEC:‘dcat lame’

Possibilities

Carry around a mobile phone with an Internet connection. Use it to
run a media player. The media player sees files on your home computer
and the files on the local network. Then it pipes the files in its
playlist to the closest encode server, and from there to the wanted
output (headphones, speakers, file, streaming server…)

Send your 3D animation file over the Internet to a thousand idle
machines to render.

Walk into an auditorium and display slides on the overhead projector,
streamed from a network mount, controlled from your mobile.

Run a search engine, giving URIs to remote services, making automatable
the task of finding and utilizing a network service.

Quoting Ilmari H. [email protected]:

Wow, that’s exactly what I was looking for! Thank you!

Just one “but”: I think using a known path under /tmp/dipus may be a
security
risk.

On 12/27/05, Pau Garcia i Quiles [email protected] wrote:

Quoting Ilmari H. [email protected]:

Wow, that’s exactly what I was looking for! Thank you!

Thank you for trying it out, let me know if / when you find bugs :wink:

Just one “but”: I think using a known path under /tmp/dipus may be a security
risk.

How do you mean?