Maximum number of sockets

Ruby
1

Hi,

I am trying to test how many concurrent connections a certain web
server (nginx) can support in our machines, so I tried to make a
little ruby script to just open sockets connecting to that machine. I
first increased the fd limit of both the server and the client to
100000 and 50000 respectively, and then did this:

require ‘socket’
include Socket::Constants

sockets = []

1086.times do |i|
print “#{i},”
socket = Socket.new(AF_INET, SOCK_STREAM, 0)
sockaddr = Socket.pack_sockaddr_in(80, ‘localhost’)
socket.connect(sockaddr)
sockets << socket
end

print “All sockets connected (#{sockets.size})”
gets

I’m testing first in my own machine both the client and the server
(hence the ‘localhost’, which shall be replaced by the server IP when
this is ready). I started doing 50000 sockets and I faced a problem:

nginx_client.rb:10: [BUG] Segmentation fault
ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

(surrounded by all the printed #{i}s)

Testing different numbers, it seems that 1085 works fine, while 1086
crashes. Anybody has any idea why this could be? Am I doing something
wrong in the socket code? Also, I couldn’t find a good description of
the sockaddr stuff or the pack_sockaddr_in method: can the result of
pack_sockaddr_in be reused with different sockets?

I’m using the Ubuntu ruby 1.8.7:

Linux jesus-laptop 2.6.27-14-generic #1 SMP Tue Aug 18 16:25:45 UTC
2009 i686 GNU/Linux

ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

Thanks,

Jesus.

2

I’d have thought that you’d hit a limit of 1024 because of fd_set (man
select). When I run your code, it stops after 1021 with:

in `initialize’: Too many open files - socket(2) (Errno::EMFILE)

This is ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux]
from Karmic.

I suspect you have managed to go 8 bytes beyond the limit (1024+64=1088)
before overwriting something important and killing the ruby interpreter

  • presumably the newer 1.8.7 I have fixes this.

To go beyond this limit you may need to use something like eventmachine
which uses poll/epoll instead of select.

3

Brian C. [email protected] wrote:

I’d have thought that you’d hit a limit of 1024 because of fd_set (man
select). When I run your code, it stops after 1021 with:

in `initialize’: Too many open files - socket(2) (Errno::EMFILE)

0, 1, 2 = stdin, stdout, stderr

This is ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux]
from Karmic.

I suspect you have managed to go 8 bytes beyond the limit (1024+64=1088)
before overwriting something important and killing the ruby interpreter

  • presumably the newer 1.8.7 I have fixes this.

Not from what I can tell, it looks like 1.9.2dev has this problem as
well.
Neither checks for fd < 1024 before calling FD_SET().

To go beyond this limit you may need to use something like eventmachine
which uses poll/epoll instead of select.

Yes, there’s also Rev which is based on libev.

4

On Tue, Apr 6, 2010 at 10:33 PM, Brian C. [email protected]
wrote:

I’d have thought that you’d hit a limit of 1024 because of fd_set (man
select). When I run your code, it stops after 1021 with:

in `initialize’: Too many open files - socket(2) (Errno::EMFILE)

This is ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux]
from Karmic.

Did you increase the ulimit for file descriptors for your process (I
did ulimit -n 50000)?

I suspect you have managed to go 8 bytes beyond the limit (1024+64=1088)
before overwriting something important and killing the ruby interpreter

  • presumably the newer 1.8.7 I have fixes this.

To go beyond this limit you may need to use something like eventmachine
which uses poll/epoll instead of select.

But who is calling select in my code? I’ll take a look at the
implementation of the socket class, although my C is a bit rusty. I
just create the sockets and connect to a server, I don’t want to read
from them or anything (yet).

Jesus.

5

Logged at http://redmine.ruby-lang.org/issues/show/3106

6

On Wed, Apr 7, 2010 at 10:47 AM, Brian C. [email protected]
wrote:

fcntl(1088, F_SETFL, O_RDWR) = 0
write(1, “All sockets connected (1086)”, 28All sockets connected (1086))
= 28

It looks like ruby makes a non-blocking connect, then does a select to
wait for the response (in case there’s another green thread which can
run)

Oh, ok. So this means that a ruby program cannot (or shouldn’t be able
to) open more than 1024 sockets?

}
Running this gives me the same values as yours: 1024/128

For now I will limit the script to 1000 and spawn different ruby
processes to get more sockets…

Thanks,

Jesus.

7

Jesús Gabriel y Galán wrote:

Did you increase the ulimit for file descriptors for your process (I
did ulimit -n 50000)?

Oh you’re right, I had -n 1024. I changed it to -n 4096 and it stopped
successfully at “All sockets connected (1086)”

If I increase the loop count to 2048 then it stops at

1213,ert.rb:10:in `connect’: Socket operation on non-socket - connect(2)
(Errno::ENOTSOCK)

[although I didn’t get a SEGV]

Apache error.log showed I was hitting Apache MaxClients too, so I
tweaked some Apache mpm_worker values to allow 1500, but the ruby script
still failed at 1213.

I suspect you have managed to go 8 bytes beyond the limit (1024+64=1088)
before overwriting something important and killing the ruby interpreter

  • presumably the newer 1.8.7 I have fixes this.

To go beyond this limit you may need to use something like eventmachine
which uses poll/epoll instead of select.

But who is calling select in my code?

Ruby itself. Use strace to see:

connect(1088, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(“127.0.0.1”)}, 16) = -1 EINPROGRESS (Operation now in
progress)
select(1089, NULL, [64 1028 1030 1031 1032 1034 1037 1038 1039 1043 1044
1045 1050 1051 1054 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065
1066 1067 1068 1069 1070 1088], [1088], NULL) = 31 (out [1028 1030 1031
1032 1034 1037 1038 1039 1043 1044 1045 1050 1051 1054 1056 1057 1058
1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1088]])
connect(1088, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(“127.0.0.1”)}, 16) = 0
fcntl(1088, F_SETFL, O_RDWR) = 0
write(1, “All sockets connected (1086)”, 28All sockets connected (1086))
= 28

It looks like ruby makes a non-blocking connect, then does a select to
wait for the response (in case there’s another green thread which can
run)

But I don’t understand how select is going above 1024, when this little
C program shows that the size of fd_set is 128 bytes / 1024 fds:

#include <stdio.h>
#include <sys/select.h>
int main(void)
{
printf("%ld\n", (long)FD_SETSIZE);
printf("%ld\n", (long)sizeof(fd_set)); // bytes
return 0;
}

I can only imagine that it is stomping on data beyond its bounds, which
if true is a really serious bug.

Regards,

Brian.

8

Jesús Gabriel y Galán wrote:

Oh, ok. So this means that a ruby program cannot (or shouldn’t be able
to) open more than 1024 sockets?

Yes, unless they are handled outside of the ruby core, e.g. by
eventmachine or Rev. But if you’re going to that trouble you might as
well use a language better suited to the job, e.g. erlang, so your idea
of forking multiple processes makes sense.

9

On Wed, Apr 7, 2010 at 1:34 PM, Brian C. [email protected]
wrote:

Jesús Gabriel y Galán wrote:

Oh, ok. So this means that a ruby program cannot (or shouldn’t be able
to) open more than 1024 sockets?

Yes, unless they are handled outside of the ruby core, e.g. by
eventmachine or Rev. But if you’re going to that trouble you might as
well use a language better suited to the job, e.g. erlang, so your idea
of forking multiple processes makes sense.

I did this:

#!/bin/bash
for i in {1…25}
do
ruby ./nginx_client.rb $i &
done

read
pkill -f “ruby ./nginx_client.rb”

#nginx_client.rb
require ‘socket’
include Socket::Constants

id = ARGV[0] || “N/A”

sockets = []

1000.times do |i|
socket = Socket.new(AF_INET, SOCK_STREAM, 0)
sockaddr = Socket.pack_sockaddr_in(80, ‘127.0.0.1’)
socket.connect(sockaddr)
sockets << socket
end

puts “[#{id}] All sockets connected (#{sockets.size})”
sleep

and I sucessfully managed to open 25000 sockets to the server, which
handled this really nicely taking nearly 0 cpu and only 23 MB of
memory.

Thanks,

Jesus.