DIPUS - Distributed IPC by Proxying UNIX Sockets
Sockets with user authentication, file permissions, encryption,
service discovery and network transparent addressing.
host1$ dcat lame-svc foo.wav | dcat madplay.host3
host1$ dcat sep://example.com/image/jpeg/to/image/png foo.jpeg | dcat
ssh://firstname.lastname@example.org/image/png/to/text/plain | less
host3$ dipus-local-pipe ‘madplay -’ madplay
host2$ dipus-export-pipe ‘lame - -’ lame-svc
Some performance numbers:
Local throughput around 50MBps (or 100MBps, depending whether you use
socat as a client or not), unencrypted network throughput 9.8MBps in a
100Mbps LAN, CPU-limited when using SSH (my other computer is slow.)
In what languages can I use and create services?
Anything that supports a) UNIX domain sockets, and b) listing
filesystem directories. So, just about every language available. If
you install the helper scripts in bin, you don’t even need those, just
the ability to read from stdin and write to stdout.
Hi, designed and implemented a distributed IPC system that takes
lessons from UNIX (keep it simple) and HTTP (keep it simple.) Opinions
on the design or anything else? I’m dying to get some feedback,
writing stuff in a bubble can take you only so far.
And, now, with the first version of this December project out of the
way, I’m off to librend land~
What is this for?
You have a program. You want to access it remotely with the minimum
hassle. It’d be nice to access it efficiently locally too. Without
rewriting your connection code.
And it’d be nice to put some connections over SSH. And be notified
when a service dies or a new one is created. And not expose any
of the individual services to the network, or even to other users
on the machine running them. And it’d be cool to be able to have a
different name and protocol for each service without having to shoehorn
FTP over XML-RPC. Oh, also, display a real-time list of services
running on the local network and on explicitly defined machines that
aren’t on the local network. And it’d be especially cool if all the
software could just use existing libraries like sockets and files.
So, take UNIX sockets and sprinkle some sugar to make them usable for
Should work on Linux and *BSD, likely on OS X too.
No Windows support, although it’s theoretically possible.
Ruby version 1.8.2+ (one released in 2005)
OpenSSH version > 3.9 (for connection sharing support (ControlPath))
ssh-agent setup (for maximum convenience)
More than one computer.
How does it work?
To create a service, make a listening socket in a subdir of
/tmp/dipus, so that the path is like:
To access a service, open its socket.
To make a service only accessible to members of group ‘audio’:
chgrp audio socket_name
chmod 770 socket_name
Listing services that you can see:
DIPUS provides helper methods and daemons to manipulate the
sockets in /tmp/dipus, e.g. create proxies to remote sockets,
delete dead sockets, provide network access to exported
sockets, find sockets by their service name, hostname, or
Quick usage example
(optional) start dipusd to automatically discover and import services,
export services, and delete dead sockets.
Create local madplay service for playing MP3s on
the computer connected to speakers.
The first argument is the command to run for each incoming connection.
The second argument is service_name/protocol, e.g. my_www/http.
The following command creates a socket with the path
and runs ‘madplay -’ for each incoming connection, setting the
connection to madplay’s stdin and stdout.
[kig@stereo] dipus-local-pipe ‘madplay -’
Create an exported lame service for encoding wavs to MP3s.
[kig@encode] dipus-export-pipe ‘lame - -’ lame/audio/x-wav/to/audio/mpeg
Manually import the services from stereo.
Import over SSH to gain access to all services and not just exported
[kig@desktop] dipus-import ssh://stereo
Dipusd should’ve discovered and imported the lame service from encode,
so we don’t need to manually import that. But if that didn’t happen,
can import it by doing
dipus-import sep://encode, sep being
Socket Export Protocol (see doc/dipus-specification.txt)
Now, let’s pipe some wavs to the stereo.
First encode the wav to MP3, using any service with a protocol that
that the service converts audio/x-wav to audio/mpeg.
Play using a service named madplay running on stereo.
[kig@desktop] dcat audio/x-wav/to/audio/mpeg *.wav | dcat madplay.stereo
We could also access the services without importing them by giving the
service URI as the service name.
Here’s the above piping example using service URIs.
Dcat lists the services matching the URI and uses the first match.
[kig@desktop] dcat sep://encode/audio/x-wav/to/audio/mpeg *.wav |
List of DIPUS commands
Pass --help as an argument to get the full USAGE.
dls – list services
dcat – pipe stdin and/or given files to a DIPUS socket
dcat lame foo.wav bar.wav > foobar.mp3
dcat ssh://foobar/image/png/to/image/jpeg cat.png > cat.jpg
dipus-import – imported all found service from an URI
dipusd – start exporter, importer, advertiser, discoverer,
dead socket monitor, and default services
dipus-exporter – start exporter
dipus-discoverer – discover exporters on the LAN and print
dipus-service-monitor – monitor for service changes on localhost,
used by SSH importers
dipus-ssh-importer – import services over SSH to user’s DIPUS
subdirectory (/tmp/dipus/username), can use
discovery to find exporter-running hosts on
dipus-local-pipe – create a local DIPUS service that forks the
given command for the connection
dipus-local-pipe ‘head’ head/text
dipus-export-pipe – create an exported DIPUS service like above
dipus-export-pipe ‘madplay -’ madplay/audio/mpeg/to/audio/out
Using socat to create and connect to DIPUS services
proxy a web server
mkdir -p /tmp/dipus/exported/http
using a lame service to encode a wav
or by using dls to find the filename
socat foo.wav UNIX-CONNECT:
dls -1 -f lame > foo.mp3
proxy a DIPUS service to a TCP socket
socat TCP4-LISTEN:1234,fork EXEC:‘dcat lame’
Carry around a mobile phone with an Internet connection. Use it to
run a media player. The media player sees files on your home computer
and the files on the local network. Then it pipes the files in its
playlist to the closest encode server, and from there to the wanted
output (headphones, speakers, file, streaming server…)
Send your 3D animation file over the Internet to a thousand idle
machines to render.
Walk into an auditorium and display slides on the overhead projector,
streamed from a network mount, controlled from your mobile.
Run a search engine, giving URIs to remote services, making automatable
the task of finding and utilizing a network service.