DIPUS - Distributed IPC by Proxying UNIX Sockets
Sockets with user authentication, file permissions, encryption,
service discovery and network transparent addressing.
Using services:
host1$ dcat lame-svc foo.wav | dcat madplay.host3
host1$ dcat sep://example.com/image/jpeg/to/image/png foo.jpeg | dcat
ssh://[email protected]/image/png/to/text/plain | less
Creating services:
host3$ dipus-local-pipe ‘madplay -’ madplay
host2$ dipus-export-pipe ‘lame - -’ lame-svc
Some performance numbers:
Local throughput around 50MBps (or 100MBps, depending whether you use
socat as a client or not), unencrypted network throughput 9.8MBps in a
100Mbps LAN, CPU-limited when using SSH (my other computer is slow.)
In what languages can I use and create services?
Anything that supports a) UNIX domain sockets, and b) listing
filesystem directories. So, just about every language available. If
you install the helper scripts in bin, you don’t even need those, just
the ability to read from stdin and write to stdout.
Download: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2.tar.gz
RDocs: http://www.cs.helsinki.fi/u/irkheikk/dipus-rdoc/
Darcs repo: http://dark.fhtr.org/repos/ruby-dipus
Readme: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/readme.txt
Design:
http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-design.txt
Protocol spec:
http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-specification.txt
^^^^^^^^^^^^^^^^^^^^^^^^^^
Hi, designed and implemented a distributed IPC system that takes
lessons from UNIX (keep it simple) and HTTP (keep it simple.) Opinions
on the design or anything else? I’m dying to get some feedback,
writing stuff in a bubble can take you only so far.
And, now, with the first version of this December project out of the
way, I’m off to librend land~
Ilmari H.
vvvvvvvvvvvvvvvvvvvvvvvvvv
What is this for?
You have a program. You want to access it remotely with the minimum
hassle. It’d be nice to access it efficiently locally too. Without
rewriting your connection code.
And it’d be nice to put some connections over SSH. And be notified
when a service dies or a new one is created. And not expose any
of the individual services to the network, or even to other users
on the machine running them. And it’d be cool to be able to have a
different name and protocol for each service without having to shoehorn
FTP over XML-RPC. Oh, also, display a real-time list of services
running on the local network and on explicitly defined machines that
aren’t on the local network. And it’d be especially cool if all the
software could just use existing libraries like sockets and files.
So, take UNIX sockets and sprinkle some sugar to make them usable for
distributed IPC.
Supported systems
Should work on Linux and *BSD, likely on OS X too.
No Windows support, although it’s theoretically possible.
Required
Ruby version 1.8.2+ (one released in 2005)
cat
Recommended
OpenSSH version > 3.9 (for connection sharing support (ControlPath))
ssh-agent setup (for maximum convenience)
socat
More than one computer.
How does it work?
To create a service, make a listening socket in a subdir of
/tmp/dipus, so that the path is like:
/tmp/dipus/exported/audio/x-wav/to/audio/mpeg/lame.mybox…0
To access a service, open its socket.
To make a service only accessible to members of group ‘audio’:
chgrp audio socket_name
chmod 770 socket_name
Listing services that you can see:
tree /tmp/dipus
DIPUS provides helper methods and daemons to manipulate the
sockets in /tmp/dipus, e.g. create proxies to remote sockets,
delete dead sockets, provide network access to exported
sockets, find sockets by their service name, hostname, or
protocol.
Quick usage example
(optional) start dipusd to automatically discover and import services,
export services, and delete dead sockets.
[kig@desktop] dipusd
[kig@encode] dipusd
Create local madplay service for playing MP3s on
the computer connected to speakers.
The first argument is the command to run for each incoming connection.
The second argument is service_name/protocol, e.g. my_www/http.
The following command creates a socket with the path
/tmp/dipus/local/audio/mpeg/to/audio/out/madplay.stereo…0
and runs ‘madplay -’ for each incoming connection, setting the
connection to madplay’s stdin and stdout.
[kig@stereo] dipus-local-pipe ‘madplay -’
madplay/audio/mpeg/to/audio/out
Create an exported lame service for encoding wavs to MP3s.
[kig@encode] dipus-export-pipe ‘lame - -’ lame/audio/x-wav/to/audio/mpeg
Manually import the services from stereo.
Import over SSH to gain access to all services and not just exported
ones.
[kig@desktop] dipus-import ssh://stereo
Dipusd should’ve discovered and imported the lame service from encode,
so we don’t need to manually import that. But if that didn’t happen,
we
can import it by doing dipus-import sep://encode, sep being
Socket Export Protocol (see doc/dipus-specification.txt)
Now, let’s pipe some wavs to the stereo.
First encode the wav to MP3, using any service with a protocol that
says
that the service converts audio/x-wav to audio/mpeg.
Play using a service named madplay running on stereo.
[kig@desktop] dcat audio/x-wav/to/audio/mpeg *.wav | dcat madplay.stereo
We could also access the services without importing them by giving the
service URI as the service name.
Here’s the above piping example using service URIs.
Dcat lists the services matching the URI and uses the first match.
[kig@desktop] dcat sep://encode/audio/x-wav/to/audio/mpeg *.wav |
dcat
ssh://stereo/audio/mpeg/to/audio/out/madplay.stereo…0
List of DIPUS commands
Pass --help as an argument to get the full USAGE.
-
dls – list services
dls sep://foobar/audio
dls ssh://barfoo -
dcat – pipe stdin and/or given files to a DIPUS socket
dcat lame foo.wav bar.wav > foobar.mp3
dcat ssh://foobar/image/png/to/image/jpeg cat.png > cat.jpg -
dipus-import – imported all found service from an URI
dipus-import ssh://foobar/audio
dipus-import sep://barfoo -
dipusd – start exporter, importer, advertiser, discoverer,
dead socket monitor, and default services -
dipus-exporter – start exporter
-
dipus-discoverer – discover exporters on the LAN and print
their addresses -
dipus-service-monitor – monitor for service changes on localhost,
used by SSH importers -
dipus-ssh-importer – import services over SSH to user’s DIPUS
subdirectory (/tmp/dipus/username), can use
discovery to find exporter-running hosts on
localnet -
dipus-local-pipe – create a local DIPUS service that forks the
given command for the connection
dipus-local-pipe ‘head’ head/text -
dipus-export-pipe – create an exported DIPUS service like above
dipus-export-pipe ‘madplay -’ madplay/audio/mpeg/to/audio/out
Using socat to create and connect to DIPUS services
proxy a web server
mkdir -p /tmp/dipus/exported/http
socat UNIX-LISTEN:/tmp/dipus/exported/http/foo.foobar…0,fork
TCP4-CONNECT:foo.example.com:80
rm /tmp/dipus/exported/http/foo.foobar…0
using a lame service to encode a wav
socat foo.wav
UNIX-CONNECT:/tmp/dipus/local/audio/x-wav/to/audio/mpeg/lame.foobar…0
> foo.mp3
or by using dls to find the filename
socat foo.wav UNIX-CONNECT:dls -1 -f lame > foo.mp3
proxy a DIPUS service to a TCP socket
socat TCP4-LISTEN:1234,fork EXEC:‘dcat lame’
Possibilities
Carry around a mobile phone with an Internet connection. Use it to
run a media player. The media player sees files on your home computer
and the files on the local network. Then it pipes the files in its
playlist to the closest encode server, and from there to the wanted
output (headphones, speakers, file, streaming server…)
Send your 3D animation file over the Internet to a thousand idle
machines to render.
Walk into an auditorium and display slides on the overhead projector,
streamed from a network mount, controlled from your mobile.
Run a search engine, giving URIs to remote services, making automatable
the task of finding and utilizing a network service.
