How to do UDP calls

Dobai-Pataky_BSSSSl · December 15, 2010, 11:17am

Hey guys…

I have a question here and maybe someone can help me.

I wrote a script to get information from some websites. Basically I call
an url and I get a file as response. I’m using Mechanize to do that,
like this:

@agent = Mechanize.new do |a|
a.history.max_size = 1
a.read_timeout = 60
end

res = @agent.get(url).save_as(path)

After I get the file, I just need to open and extract the information I
need. Everything is working good.

My problem now is that I have some UDP urls and Mechanize does not
support UDP urls…

So, my question is, how can I get the files from UDP urls?
I looked online for gems to do that, but I didn’t find anything…
Any of you can help me?

Thanks,

Luis

luisfgnr · December 15, 2010, 1:11pm

On Wed, Dec 15, 2010 at 11:17 AM, Luis G. [email protected] wrote:

a.read_timeout = 60
So, my question is, how can I get the files from UDP urls?
I looked online for gems to do that, but I didn’t find anything…
Any of you can help me?

The standard library has basic UDP functionality already.

http://www.ruby-doc.org/stdlib/libdoc/socket/rdoc/classes/UDPSocket.html

Kind regards

robert

luisfgnr · December 15, 2010, 4:05pm

Luis G. wrote in post #968518:

So, my question is, how can I get the files from UDP urls?
I looked online for gems to do that, but I didn’t find anything…
Any of you can help me?

Can you give an example of what you mean by a “UDP url”? I don’t know of
any standard for doing HTTP over UDP. Is this something like a SIP url
perhaps? TFTP? In that case you could look for a SIP or TFTP client
library.

Otherwise, as has already been said, the code for doing UDP sockets is
there. Have a look at resolv.rb in the standard library for some example
code (this is a pure-ruby DNS resolver)

luisfgnr · December 16, 2010, 3:49pm

Maybe I have not explained well…

I’m basically sending requests to torrent trackers to get the number of
seeds and peers of a torrent in a specific tracker.
As you might know, you can have this:

http://9.rarbg.com:2710/announce
udp://tracker.openbittorrent.com:80/announce

so, you can have http and udp trackers.

For the first one (HTTP) I just use this:

url = tracker_url +
‘?info_hash=%19%89B%0AL%87k%9A%2B%28%3A%B36%9D%93%9A%8D%FA%97I’

agent = Mechanize.new do |a|
a.history.max_size = 1
a.read_timeout = 60
end

res = agent.get(url).save_as(path)

with this I get a file with the number of seeds and peers on it. I just
need to open the file in ‘path’ and extract the information.

But for the second one (UDP) I can’t use Mechanize, so I need to work
with the standard library as you told before.

I never used socket library before, so I’m kinda lost.
I was trying something like:

host = ‘tracker.openbittorrent.com’
port = 80
data = ‘%19%89B%0AL%87k%9A%2B%28%3A%B36%9D%93%9A%8D%FA%97I’

sock = UDPSocket.open
sock.connect(host, port)
sock.send(“?info_hash=” + data, 0)

But there are some things here that I don’t know:

1- The main idea is to create a socket, connect it to the remote host,
send the request and get the response, right?

2- do I need to include the ‘udp://’ in the host?

3- is ok to include the name of the field I want to send (?info_hash=)
in the data variable? I need to include that as you can see here:
http://wiki.theory.org/BitTorrent_Tracker_Protocol

4- how can I get the response from the server? something like:
sock.recvfrom(1024)?

I tried several options and I had no success.

Any clues?

Thanks,

Luis

luisfgnr · December 16, 2010, 4:16pm

OK, there’s a spec linked from here:

It’s not HTTP. It’s bittorrent-specific (so using “udp” as the URL label
was silly). You’ll find Array#pack will be your friend.

luisfgnr · December 16, 2010, 4:13pm

Luis G. wrote in post #968857:

As you might know, you can have this:

http://9.rarbg.com:2710/announce
udp://tracker.openbittorrent.com:80/announce

No, I haven’t seen the udp:// URL format at all. Is there an RFC which
describes it?

For the first one (HTTP) I just use this:
…
But for the second one (UDP) I can’t use Mechanize, so I need to work
with the standard library as you told before.

And not only that, you need to know what format data to put inside the
UDP packet. That’s what I was getting at. You need some documentation of
the application-layer protocol, which sits inside the transport layer
protocol.

I guess it might be HTTP, complete with a full set of HTTP headers.
Worth a try, if you have a valid URL to point at.

You can use netcat (nc) to test:

echo "GET /announce" | nc -u tracker.openbittorrent.com 80

and run

sudo tcpdump -i eth0 -nn -s0 udp port 80

in another window to check for any replies. When I try that particular
example, I get nothing back.

1- The main idea is to create a socket, connect it to the remote host,
send the request and get the response, right?

I guess so…

2- do I need to include the ‘udp://’ in the host?

Nope. The socket functions take hostnames/IP addresses, not URLs. There
is a ‘URI’ class you can use to parse it out of the URL though.

3- is ok to include the name of the field I want to send (?info_hash=)
in the data variable?

That I have no idea, because I’ve not seen any documentation for this
protocol.

4- how can I get the response from the server? something like:
sock.recvfrom(1024)?

Yes, recvfrom returns a tuple of [data, address_info]. Or I believe you
could ‘connect’ the socket first, send, recv.

I tried several options and I had no success.

Any clues?

Look at a C implementation of this protocol, and port it to ruby?
tcpdump -X an existing working client, and see what it sends/receives?
Ask on a bittorrent development mailing list?

luisfgnr · December 16, 2010, 6:15pm

Hey man, thanks for your reply.

I read the documentation but this is hard for me… But actually I don’t
have enough skills to do this without help…

So, as far as I understood in the structures section of the UDP tracker
protocol documentation, the first thing I need to do is:

Choose a (random) transaction ID.
Fill the connect input structure.
Send the packet.

connect input
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

You told me that Array#pack will help me. This means that I need to put
the the content of the connect input in an array and pack it so I can
send it, right?
Actually I thought we need to have something like:
connection_id =>0x41727101980
action => 0
…

Anyway, see this example:

data = Array.new
data << 0x41727101980 #connection_id
data << 0 #action
data << 16 #transaction_id

host = ‘tracker.openbittorrent.com’
port = 80

s = data.pack(‘q’) #q?!

sock = UDPSocket.open
sock.connect(host, port)
sock.send(s, 0)
response = sock.recvfrom(1024)

After this I should:

Receive the packet (response).
Check whether the packet is at least 16 bytes.
Check whether the transaction ID is equal to the one you chose.
Check whether the action is connect.
Store the connection ID for future use.

You can see something wrong with this code?

BTW, I tried

echo “GET /announce” | nc -u tracker.openbittorrent.com 80
sudo tcpdump -i eth0 -nn -s0 udp port 80

and I got results:
18:09:14.719638 IP 192.168.182.123.57232 > 95.215.62.26.80: UDP, length
14

Thanks a lot,

Luis

luisfgnr · December 16, 2010, 9:58pm

Luis G. wrote in post #968897:

connect input
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

You told me that Array#pack will help me. This means that I need to put
the the content of the connect input in an array and pack it so I can
send it, right?
Actually I thought we need to have something like:
connection_id =>0x41727101980
action => 0
…

Anyway, see this example:

data = Array.new
data << 0x41727101980 #connection_id
data << 0 #action
data << 16 #transaction_id

host = ‘tracker.openbittorrent.com’
port = 80

s = data.pack(‘q’) #q?!

There are three elements in the array ‘data’, so:
s = data.pack(‘qNN’)

However ‘q’ may be wrong here, if you’re on a little-endian machine.
Unfortunately, Array#pack doesn’t have a 64-bit network-order flag.

I think you should use something like this:

conn_id = 0x41727101780
s = [conn_id >> 32, conn_id & 0xffffffff, 0, 16].pack(‘NNNN’)

Then for debugging, to check you’re sending what you think:

STDERR.puts s.unpack(“H*”).first

BTW, I tried

echo “GET /announce” | nc -u tracker.openbittorrent.com 80
sudo tcpdump -i eth0 -nn -s0 udp port 80

and I got results:
18:09:14.719638 IP 192.168.182.123.57232 > 95.215.62.26.80: UDP, length
14

That’s the outbound packet (from 192.168.182.123 local port 57232, to
95.215.62.26 port 80), and you got no response.

luisfgnr · December 17, 2010, 1:08pm

Hello again.

Man, thanks a lot for you time. You’ve been a really great help.

first question, shouldn’t the array just contain 3 element?
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

I tried to run the script with the information you gave me. So far I
have this:

host = ‘tracker.openbittorrent.com’
port = 80
connection_id = 0x41727101980

s = [connection_id >> 32, connection_id & 0xffffffff, 0,
16].pack(‘NNNN’)

sock = UDPSocket.open
sock.connect(host, port)
sock.send(s, 0)
response = sock.recvfrom(1024)

And actually now I have something in the ‘response’ variable. If I print
the content I got something like

[“\000\000\000\000\000\000\000\020#B\005\027�AP\377”,
[“AF_INET”, 80, “tracker.openbittorrent.com”, “95.215.62.5”]]

And STDERR.puts response.unpack(“H*”).first will return

000000000000001023420517de4150ff

In the documentation says that ‘Check whether the transaction ID is
equal to the one you chose.’. So basically I should get the same value I
sent, right?

Thanks,

Luis

luisfgnr · December 17, 2010, 4:34pm

Luis G. wrote in post #969040:

first question, shouldn’t the array just contain 3 element?
64-bit integer connection_id 0x41727101980
32-bit integer action 0
32-bit integer transaction_id 16

A 64-bit integer is 8 bytes, and two 32-bit integers are 8 bytes.

The problem is you want to pack 0x123456789abcdef as 0123456789abcdef,
but the ‘q’ option to pack gives something else (on Intel-machines
anyway):

n = 0x123456789abcdef
=> 81985529216486895
[n].pack(“q”)
=> “\357\315\253\211gE#\001”
[n].pack(“q”).unpack(“H*”).first
=> “efcdab8967452301”

You want to send it in network-byte order, MSB first. There’s no
function in pack to do this, but “N” packs 32-bits in network-order. So:

[n>>32,n&((1<<32)-1)].pack(“NN”)
=> “\001#Eg\211\253\315\357”
[n>>32,n&((1<<32)-1)].pack(“NN”).unpack(“H*”).first
=> “0123456789abcdef”

Mind you, if the only thing to need to do is check the conn_id is the
same, maybe you will get away with “q” if you use it the same for
packing and unpacking.

I tried to run the script with the information you gave me.
…
And actually now I have something in the ‘response’ variable. If I print
the content I got something like

[“\000\000\000\000\000\000\000\020#B\005\027�AP\377”,
[“AF_INET”, 80, “tracker.openbittorrent.com”, “95.215.62.5”]]

And STDERR.puts response.unpack(“H*”).first will return

000000000000001023420517de4150ff

In the documentation says that ‘Check whether the transaction ID is
equal to the one you chose.’. So basically I should get the same value I
sent, right?

The result is an array of two elements, the data and the address info.

Take the first element, then look at data.unpack(“NNq”) or
data.unpack(“NNNN”). To handle the byte-ordering problem again:

action, trans_id, c0, c1 = data.unpack(“NNNN”)
conn_id = (c0 << 32) | c1

(note that they come in a different order in the response than in the
request)

luisfgnr · December 20, 2010, 12:08pm

Luis G. wrote in post #969537:

In the documentation says:

Check whether the transaction ID is equal to the one you choose - I
got the same value ‘16’, so I think is ok;

Check whether the action is connect - I got a ‘0’… Shouldn’t I get a
‘1’?

Not from my reading of the description at
http://xbtt.sourceforge.net/udp_tracker_protocol.html

It says the ‘connect input’ packet is

Offset Size Name Value
0 64-bit integer connection_id 0x41727101980
8 32-bit integer action 0
12 32-bit integer transaction_id
16

and the ‘connect output’ packet is
Offset Size Name Value
0 32-bit integer action 0
4 32-bit integer transaction_id
8 64-bit integer connection_id
16

This is admittedly a pretty rubbish protocol document, but if you
understand the semantics of each of these messages from some other
reference, it might make sense to you.

The next step is to fill the announce input and send it, so I can get
the seeds and peers (my goal :)).
I have one question here: All the fields are required, right? Because I
don’t really know some fields, like: peer_id, downloaded, left,
uploaded, event, key. the field key, for example don’t even appear in
BitTorrent Tracker Protocol
(BitTorrent Tracker Protocol - TheoryOrg)

I’d say this is no longer anything to do with Ruby, but all about how to
write a BitTorrent protocol implementation, and so this would be best
answered on a BitTorrent mailing list. Sorry.

Just one more question… The info_hash is a 20-byte string. To pack
this variable, I can use the ‘m’ flag, right?

I doubt it, that would MIME-encode it. You probably want ‘a20’

[-1,“foo”,-1].pack(“Na20N”)
=>
“\377\377\377\377foo\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\377\377\377\377”
[-1,“foobarfoobarfoobarfoobar”,-1].pack(“Na20N”)
=> “\377\377\377\377foobarfoobarfoobarfo\377\377\377\377”

luisfgnr · December 20, 2010, 11:36am

Hello again.

Thanks for your explanation.
I understood that we need to split a 64bit integer in two 32 bit integer
and pack them in network-byte order.
So, as you said:

connection_id = 0x41727101980
pp connection_id

4497486125440

s = [connection_id >> 32, connection_id & 0xffffffff, 0,
16].pack(‘NNNN’)
c0, c1, action, trans_id = s.unpack(“NNNN”)
conn = (c0 << 32) | c1

pp conn

4497486125440

pp action

0

pp trans_id

16

I did this small test like you explained and I get the same values after
pack and unpack the information(connection_id = conn).

So, after I pack the data, I send it and I get a response:

sock = UDPSocket.open
sock.connect(host, port)
sock.send(s, 0)
response = sock.recvfrom(1024)

so, I extract the content of the response like this (using a different
order of the request):

action, trans_id, c0, c1 = response[0].unpack(“NNNN”)
conn_id = (c0 << 32) | c1

pp action

0

pp trans_id

16

pp conn_id

2540598739861590271

In the documentation says:

Check whether the transaction ID is equal to the one you choose - I
got the same value ‘16’, so I think is ok;
Check whether the action is connect - I got a ‘0’… Shouldn’t I get a
‘1’?

The next step is to fill the announce input and send it, so I can get
the seeds and peers (my goal :)).
I have one question here: All the fields are required, right? Because I
don’t really know some fields, like: peer_id, downloaded, left,
uploaded, event, key. the field key, for example don’t even appear in
BitTorrent Tracker Protocol
(BitTorrent Tracker Protocol - TheoryOrg)

Just one more question… The info_hash is a 20-byte string. To pack
this variable, I can use the ‘m’ flag, right?

Thanks,

Luis Goncalves