TCPserver how to signal EOT

Ruby
1

Hi I am writing some networking code and am having a heck of a time
figuring out how to break on EOT. I was sending an ASCII control
character, but this doesn’t work when I send binary data because
sometimes it is seen as the EOT character so breaks early. Here is one
thing I have tried:

server = TCPServer.open(address, port)

loop do
Thread.start(server.accept) do |client|
message = “”
loop do
message << client.recvfrom( 4 )[0]
break if client.eof? == true
end
yield(message, client)
end
end

if I replace the break if client.eof? == true with simply puts
client.eof? it will return false if it is not the end of what is being
sent, otherwise it just hangs there indefinitely. This is only one of
many ways I have tried, client.recvfrom does not seem to return 0 or -1
when no more data is to be sent, it also does not seem to return an
empty string. Pretty much the only way I have gotten this to work is by
looping client.recvfrom and then breaking if the character at -1 in what
it gets is the ASCII control character for EOT. Unfortunately I can not
use that because when I send random binary data there is a chance it
will end on EOT without it being a control character. Everything else I
try ends up just hanging indefinitely and never making it to the yield
statement, sometimes it is just waiting for more data to recvfrom other
times I have no idea what happens such as when it hangs at client.eof?
when it is actually the end of what is being sent but returns false
otherwise.

Surely there is some easy way to do this, but none of the Ruby
documentation mentions anything about how to and most of the examples do
not even have recv in a loop for some reason.

2

On Sun, Aug 19, 2012 at 1:03 PM, tammy roberts [email protected]
wrote:

Hi I am writing some networking code and am having a heck of a time
figuring out how to break on EOT.

This is only one of
many ways I have tried, client.recvfrom does not seem to return 0 or -1
when no more data is to be sent, it also does not seem to return an
empty string.

Then it probably blocks.

Pretty much the only way I have gotten this to work is by
looping client.recvfrom and then breaking if the character at -1 in what
it gets is the ASCII control character for EOT. Unfortunately I can not
use that because when I send random binary data there is a chance it
will end on EOT without it being a control character.

I have seen recvfrom mainly used with UDP sockets and I am not sure
whether it blocks. Apparently you want blocking behavior since you
create a thread per connection.

Surely there is some easy way to do this, but none of the Ruby
documentation mentions anything about how to and most of the examples do
not even have recv in a loop for some reason.

Yes, just use Ruby’s ways to use custom line delimiters. Here’s one
working way:

DELIMITER = “\x04”.freeze

loop do
Thread.new(server.accept) do |client|
printf “Client %p START\n”, client

client.each_line DELIMITER do |msg|
  printf "msg len = %4d msg: %p bytes: %p\n", msg.length, msg,

msg.unpack(‘C*’).map {|i| “%02x” % i}
msg.chomp! DELIMITER
printf “msg len = %4d msg: %p bytes: %p\n”, msg.length, msg,
msg.unpack(‘C*’).map {|i| “%02x” % i}
end

printf "Client %p STOP\n", client

end
end

Kind regards

robert

3

Hi Robert, it is me again, my computer crashed and I lost my login
information. Here is what I am using right now, it seems to be working
fine, but maybe you see problems with it or a way to improve. Also maybe
it will help someone else out who is struggling with the same problem I
was. I have not yet gone through to polish it up but it seems pretty
decent to me already, let me know what you think please :).

class Networking

def listen(server)
loop do
Thread.start(server.accept) do |client|
data_received = []
total_received = 0
incoming_data_size = client.recvfrom( 4 )[0].unpack(‘N*’)[0]

    loop do
      section_size = client.recvfrom( 4 )[0].unpack('N*')[0]
      message_section = ""

      loop do
        message_section << client.recvfrom( section_size )[0]
        break if message_section.bytesize == section_size
      end

      data_received << message_section
      total_received += message_section.bytesize
      break if total_received + 4 == incoming_data_size
    end

yield(data_received, client)
  end
end

end

def send_data(socket, message)
forward_message = [message.bytesize].pack(‘N*’) + message
socket.send(forward_message, 0)
yield(socket)
end

def get_reply(socket)
incoming_data_size = socket.recvfrom( 4 )[0].unpack(‘N*’)[0]
section_size = socket.recvfrom( 4 )[0].unpack(‘N*’)[0]
reply = “”

loop do
  reply << socket.recvfrom( section_size )[0]
  break if reply.bytesize == incoming_data_size - 4
end

yield(reply, socket)

end
end

4

On Mon, Aug 20, 2012 at 11:57 AM, tammy roberts2 [email protected]
wrote:

Hi Robert, it is me again, my computer crashed and I lost my login
information. Here is what I am using right now, it seems to be working
fine, but maybe you see problems with it or a way to improve. Also maybe
it will help someone else out who is struggling with the same problem I
was.

This is a completely different approach: you do not use a termination
byte any more but transmit the message length beforehand.

    data_received = []
      end

def send_data(socket, message)
forward_message = [message.bytesize].pack(‘N*’) + message
socket.send(forward_message, 0)
yield(socket)
end

Why do you yield the socket?

yield(reply, socket)

end
end

If you use #send then I’d also use #read for reading instead of
#recvfrom. I’d only use that method if I needed the extra options.
Method #read will also block until as many bytes have arrived so you
do not take care of fragment messages yourself.

I’d probably choose a tad different approach by wrapping the
connection with something that does the message handling and creation
of a message type (for specific parsing etc.):

could be more sophisticated

Message = Struct.new :bytes

Connection = Struct.new :socket do
def send(msg)
b = msg.bytes
socket.write [b.bytesize].pack ‘N’
socket.write b
self
end

def close
socket.close
end

def each_msg
until socket.eof?
size = socket.read(4).unpack(‘N’).first
msg = socket.read(size)
yield Message.new(msg)
end
end

self

end
end

loop do
Thread.new(server.accept) do |client|
conn = Connection.new client

conn.each_msg do |msg|
  yield msg, conn
end

end
end

Kind regards

robert

5

Ah I see that I actually do not need to yield the socket as it is
already available at the calling method, good point

6

Robert K. wrote in post #1072878:

On Mon, Aug 20, 2012 at 11:57 AM, tammy roberts2 [email protected]
wrote:

Hi Robert, it is me again, my computer crashed and I lost my login
information. Here is what I am using right now, it seems to be working
fine, but maybe you see problems with it or a way to improve. Also maybe
it will help someone else out who is struggling with the same problem I
was.

This is a completely different approach: you do not use a termination
byte any more but transmit the message length beforehand.

    data_received = []
      end

def send_data(socket, message)
forward_message = [message.bytesize].pack(‘N*’) + message
socket.send(forward_message, 0)
yield(socket)
end

Why do you yield the socket?

So that it can be passed to the get_reply method if the client expects a
reply from the server, or closed by the calling method if it does not.

yield(reply, socket)

end
end

If you use #send then I’d also use #read for reading instead of
#recvfrom. I’d only use that method if I needed the extra options.
Method #read will also block until as many bytes have arrived so you
do not take care of fragment messages yourself.

So am I correct in thinking that if I use read I do not need to put it
in a loop because it will always wait until that many bytes have
arrived?

I’d probably choose a tad different approach by wrapping the
connection with something that does the message handling and creation
of a message type (for specific parsing etc.):

The listen method currently yields to the caller an array with the parts
of the data that has been transmitted to it, and the calling method then
knows how to process this array depending on what it is (the message
type).

could be more sophisticated

Message = Struct.new :bytes

Connection = Struct.new :socket do
def send(msg)
b = msg.bytes
socket.write [b.bytesize].pack ‘N’
socket.write b
self
end

def close
socket.close
end

def each_msg
until socket.eof?
size = socket.read(4).unpack(‘N’).first
msg = socket.read(size)
yield Message.new(msg)
end
end

self

end
end

loop do
Thread.new(server.accept) do |client|
conn = Connection.new client

conn.each_msg do |msg|
  yield msg, conn
end

end
end

Kind regards

robert

Thanks for your advice I will be seriously considering the way you
presented this :).

7

On Tue, Aug 21, 2012 at 4:17 AM, tammy roberts2 [email protected]
wrote:

Robert K. wrote in post #1072878:

On Mon, Aug 20, 2012 at 11:57 AM, tammy roberts2 [email protected]
wrote:

Why do you yield the socket?

So that it can be passed to the get_reply method if the client expects a
reply from the server, or closed by the calling method if it does not.

Aha. The issue with that is that you can read only with one thread at
a time from the socket. Since you created a thread after socket
accept and that thread loops reading this should be the only thread
fetching data from the socket. Sending is a different story (but btw.
you should also properly synchronize to make sure only one thread
writes at a time).

in a loop because it will always wait until that many bytes have
arrived?

Yes, #read will block. But I confused method pairs: it should be
#read and #write and not #read and #send.

I’d probably choose a tad different approach by wrapping the
connection with something that does the message handling and creation
of a message type (for specific parsing etc.):

The listen method currently yields to the caller an array with the parts
of the data that has been transmitted to it, and the calling method then
knows how to process this array depending on what it is (the message
type).

No. Method #listen does not yield. It’s the reader thread which you
start in #listen. And note that this can be a problem if the block
you pass in is not prepared to be executed concurrently.

Thanks for your advice I will be seriously considering the way you
presented this :).

You’re welcome!

Kind regards

robert

8

On Tue, Aug 21, 2012 at 3:17 PM, tammy roberts2 [email protected]
wrote:

Aha. The issue with that is that you can read only with one thread at
a time from the socket. Since you created a thread after socket
accept and that thread loops reading this should be the only thread
fetching data from the socket. Sending is a different story (but btw.
you should also properly synchronize to make sure only one thread
writes at a time).

Please correct me if I am wrong, but shouldn’t there be no problem as
the yield is inside the thread itself? It seems like the listen method
can not pass the socket to the caller until it is done reading from it.

Right you are. But this is a quite inefficient approach IMHO: you
must read all announced messages before you can process them. Also,
what happens if there are messages after the initially announced n
messages? Processing will usually be easier and more efficient if you
yield each message as soon as it has arrived. But then again, I do
know nothing about your use case and your requirements.

No. Method #listen does not yield. It’s the reader thread which you
start in #listen. And note that this can be a problem if the block
you pass in is not prepared to be executed concurrently.

Ah good observation. That makes it seem like the first issue you point
out will not come into play though, because the reader thread in listen
is what yields the socket so it seems it will not yield it until it is
done reading from it. Am I maybe misunderstanding something here?

No, apparently not. But see above.

I am going to switch my code to read and write as per your suggestion. I
notice that these methods do not seem to be documented in the Ruby
socket documentation, could you maybe point me towards some
documentation regarding them please? I will post my networking code once
more when it is finished up, again thanks for the tips and advice.

Which methods are not documented? If you refer to #read and #write,
please observe this:

$ irb19 -r socket
Ruby version 1.9.3
irb(main):001:0> TCPSocket.instance_method(:read)
=> #<UnboundMethod: TCPSocket(IO)#read>
irb(main):002:0> TCPSocket.instance_method(:write)
=> #<UnboundMethod: TCPSocket(IO)#write>

In other words: these methods are inherited from class IO.

Kind regards

robert

9

Aha. The issue with that is that you can read only with one thread at
a time from the socket. Since you created a thread after socket
accept and that thread loops reading this should be the only thread
fetching data from the socket. Sending is a different story (but btw.
you should also properly synchronize to make sure only one thread
writes at a time).

Please correct me if I am wrong, but shouldn’t there be no problem as
the yield is inside the thread itself? It seems like the listen method
can not pass the socket to the caller until it is done reading from it.

No. Method #listen does not yield. It’s the reader thread which you
start in #listen. And note that this can be a problem if the block
you pass in is not prepared to be executed concurrently.

Ah good observation. That makes it seem like the first issue you point
out will not come into play though, because the reader thread in listen
is what yields the socket so it seems it will not yield it until it is
done reading from it. Am I maybe misunderstanding something here?

I am going to switch my code to read and write as per your suggestion. I
notice that these methods do not seem to be documented in the Ruby
socket documentation, could you maybe point me towards some
documentation regarding them please? I will post my networking code once
more when it is finished up, again thanks for the tips and advice.