Custom Protocol

Ruby
1

I understand that I’ve asked a similar question that of custom packets.
But this is different. This is protocol.

Has anyone constructed a custom protocol, such as UDT, in Ruby?
What would one use to construct a custom protocol?

Right now, I’m using code from metasploit 3.0 (written in Ruby :-), and
the CStruct2 library included. Some sample stuff I have…

UDT_CTRL_PKT = Rex::Struct2::CStructTemplate.new(
[ ‘uint1v’, ‘PktType’, 0 ],
[ ‘uint8v’, ‘Type’, 0 ],
[ ‘uint16v’, ‘ExtType’, 0 ],
[ ‘uint1’, ‘X’, 0 ],
[ ‘uint30v’, ‘SubSeqNum’, 0 ],
[ ‘uint32v’, ‘TimeStamp’, 0 ]
)

UDT_DATA_PKT = Rex::Struct2::CStructTemplate.new(
[ ‘uint1v’, ‘PktType’, 0 ],
[ ‘uint30v’, ‘SeqNum’, 0 ],
[ ‘uint2v’, ‘FF’, 0 ],
[ ‘uint1v’, ‘O’, 0 ],
[ ‘uint28v’, ‘MsgNum’, 0 ],
[ ‘uint32v’, ‘TimeStamp’, 0 ]
)

What I’m intending to do - what you’re supposed to do - with the
CStruct2 lib is to create a binary string out of it and write that
to a raw socket.

What are some techniques someone would use to accomplish the goal of a
new protocol?

Thanks,
Ari B.

2

On 2007-11-29 07:38 +0900 (Thu), thefed wrote:

What are some techniques someone would use to accomplish the goal of a
new protocol?

Typically, protocols have state. So I’d start with as clear an
expression of the protocol state machine as you can generate, and then
see about twisting it into a DSL that can be implemented in Ruby.

cjs

3

On Nov 28, 2007, at 6:55 PM, Curt S. wrote:

On 2007-11-29 07:38 +0900 (Thu), thefed wrote:

What are some techniques someone would use to accomplish the goal
of a
new protocol?

Typically, protocols have state. So I’d start with as clear an
expression of the protocol state machine as you can generate, and then
see about twisting it into a DSL that can be implemented in Ruby.

What do you mean by state machine?

I’m looking to implement UDT, which requires custom packet headers
and all.
Is there some suggested way to do that?

BTW, creating a DSL should be pretty much standard :slight_smile:

Thanks,
-Ari

4

On 2007-11-29 12:18 +0900 (Thu), thefed wrote:

On Nov 28, 2007, at 6:55 PM, Curt S. wrote:

Typically, protocols have state. So I’d start with as clear an
expression of the protocol state machine as you can generate, and then
see about twisting it into a DSL that can be implemented in Ruby.

What do you mean by state machine?

For a typical protocol, you might have states such as:

- Idle
- Sent start message, waiting for response
- Connection running but idle
- Message sent, waiting for response,

And you’d move between various states based on the packets you send.

I will send you (off-list) an example of a state machine described in a
DSL embedded in Ruby.

cjs

5

On Nov 29, 2007 4:45 AM, Curt S. [email protected] wrote:

Finite-state machine - Wikipedia
I will send you (off-list) an example of a state machine described in a
DSL embedded in Ruby.

I would like to see that example on-list, or at least be part of the
off-list
conversation, if possible :-).

Thanks,

Jesus.

6

On Nov 28, 2007 5:38 PM, thefed [email protected] wrote:

I understand that I’ve asked a similar question that of custom packets.
But this is different. This is protocol.

Has anyone constructed a custom protocol, such as UDT, in Ruby?
What would one use to construct a custom protocol?

I took a very quick look at UDT and I think it’s quite interesting. The
basic approach has appeared before (people were discussing the
“long-fat-pipe” problem in TCP even before the WWW came along), but I
like
the optimizations in this one.

If I read their material correctly, UDT is based completely on UDP so
there’s no need for you to create another protocol at the IP level
(analogous to TCP, UDP, ICMP, etc). That means you don’t need raw
sockets,
which makes your task considerably easier. You don’t need to construct
the
IP header either.

Instead, implementing UDT is analogous to rebuilding TCP inside of sets
of
related UDP packets. Is there any particular reason you want to do this
in
Ruby rather than just wrapping a C or Java implementation?

7

Thanks! Thats a nice protocol you have!

On Nov 28, 2007, at 10:45 PM, Curt S. wrote:

For a typical protocol, you might have states such as:

- Idle
- Sent start message, waiting for response
- Connection running but idle
- Message sent, waiting for response,

And you’d move between various states based on the packets you send.

Ah, you mean a high level state. In UDT, since it is used to make
large data transfer faster, I still need to have lower-level control.

My current bottleneck is that of the packets. Is there a recommended
way to go about constructing them?

BTW, the website: http://udt.sourceforge.net/

Thanks,
-Ari B.

8

On Nov 29, 2007 4:18 PM, thefed [email protected] wrote:

If I read their material correctly, UDT is based completely on UDP so
this in
information be held in the payload?

I’m planning to look the implementation over and give it a try. Maybe
even
wrap it as a Ruby extension. Based on my limited reading of the spec,
all of
the data needed to relate packets to streams goes into the UDP packets.
I’m
still seeing no need for you to use raw sockets.

9

On Nov 29, 2007, at 8:37 PM, Francis C. wrote:

I’m planning to look the implementation over and give it a try.
Maybe even
wrap it as a Ruby extension. Based on my limited reading of the
spec, all of
the data needed to relate packets to streams goes into the UDP
packets. I’m
still seeing no need for you to use raw sockets.

Where would you recommend, then, including the excess information?
The Payload?

Also, is there a way to get C++ libraries loaded into C?

Thanks again,
ari

10

On Nov 29, 2007, at 8:37 AM, Francis C. wrote:

sockets,
which makes your task considerably easier. You don’t need to
construct the
IP header either.

Instead, implementing UDT is analogous to rebuilding TCP inside of
sets of
related UDP packets. Is there any particular reason you want to do
this in
Ruby rather than just wrapping a C or Java implementation?

The reason I’d like to do it in Ruby is first for my comprehension of
the code. Second, the only implementation so far is in C++, and the
bindings there are pretty messy (I think). UDT + Ruby DSL = a lot
nicer, IMHO.

As you might’ve seen, there is a lot of information, such as the sub-
sequence number and a variety of other things that are important and
pertain to the packets. If I didn’t use raw sockets, wouldn’t that
information be held in the payload?

Thanks,
Ari

11

On Nov 29, 2007, at 8:37 PM, Francis C. wrote:

I’m planning to look the implementation over and give it a try.
Maybe even
wrap it as a Ruby extension. Based on my limited reading of the
spec, all of
the data needed to relate packets to streams goes into the UDP
packets. I’m
still seeing no need for you to use raw sockets.

Where would you recommend, then, including the excess information?
The Payload?

Also, is there a way to get C++ libraries loaded into C?

Thanks again,
ari

12

On Nov 30, 2007, at 12:09 AM, Francis C. wrote:

been a pain.)
That’s good news. Does this mean their alternate information is
contained in the UDP Payload?

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

However, I’m considerably less impressed with this protocol after
reading
the spec, and I decided not to spend any time on it. There is no
straightforward way to do end-to-end encryption. They’ve redone
from scratch
all of the painstaking work of congestion avoidance, etc. that has
been
fine-tuned in TCP for decades. And who knows what kind of security
vulnerabilities they’ve introduced. For one thing, packet numbers are
sequential, a problem that TCP got rid of years ago.

I saw the sequential number issue and the cryptography problem, but I
figured two things.
a) These people aren’t that stupid, so they probably know someway
to protect against injection.
b) I was going to try to use some fast one-way encryption algorithm,
and use public keys and a central server, if I bundled this with some
other software.
b 1) Or i was going to wrap something secure and simple around it,
and figure out of to get the symmetric key across the net.

Thanks for your help anyways,
Ari

13

On Nov 29, 2007 9:52 PM, thefed [email protected] wrote:

Where would you recommend, then, including the excess information?
The Payload?

According to the spec, the header of a UDT data packet is 16 bytes
long.
That’s going to the first 16 bytes of the UDP packet’s data. You don’t
need
to do anything with the “UDP header” that IP looks at. This
implementation
is completely in userland and not in IP-land, which is a positive thing.
(They could have defined another IP-level protocol instead, which would
have
been a pain.)

However, I’m considerably less impressed with this protocol after
reading
the spec, and I decided not to spend any time on it. There is no
straightforward way to do end-to-end encryption. They’ve redone from
scratch
all of the painstaking work of congestion avoidance, etc. that has been
fine-tuned in TCP for decades. And who knows what kind of security
vulnerabilities they’ve introduced. For one thing, packet numbers are
sequential, a problem that TCP got rid of years ago.

Also, is there a way to get C++ libraries loaded into C?

Look at the C wrapper in Ruby/EventMachine, which includes an extension
written in C++.

14

On Nov 30, 2007, at 10:35 AM, Francis C. wrote:

Strictly out of curiosity, what’s your need for this protocol? Are you
running up against the long-fat-pipe limitation in TCP?

I don’t actually need this for anything. Just something that I wanted
to do. I’m not running up against anything, I just wanted to
implement something that would make data transfer faster, since
connectivity and hardware are at a low for me.

Is there some alternate to fast data transfer that you could suggest?

Thanks again,
Ari

15

On Nov 30, 2007 7:43 AM, thefed [email protected] wrote:

thing.
reading
figured two things.
a) These people aren’t that stupid, so they probably know someway
to protect against injection.
b) I was going to try to use some fast one-way encryption algorithm,
and use public keys and a central server, if I bundled this with some
other software.
b 1) Or i was going to wrap something secure and simple around it,
and figure out of to get the symmetric key across the net.

Strictly out of curiosity, what’s your need for this protocol? Are you
running up against the long-fat-pipe limitation in TCP?

16

On Nov 30, 2007 3:49 PM, thefed [email protected] wrote:

Is there some alternate to fast data transfer that you could suggest?

If you’re bandwidth-constrained, then I would doubt that replacing TCP
will
help you. Especially with a pure-Ruby implementation. Still, if it
weren’t
for the unknown-unknown issues, I’d think this was a pretty interesting
protocol. (By that, I mean that we have no idea what bizarre
edge-conditions
this protocol will create in the network, that have already been
thoroughly
shaken out of TCP.) Thanks for bringing it to our attention.