Too many open files and unix

Hello,

today I had too many open files and I need some advice to investigate in
this issue.

I googled around and found a lot information how to solve this issue
(e.g. http://forum.nginx.org/read.php?2,234191,234191)
I finally was able to get rid of the error.

But I wondered about nginx behaviour with unix sockets and open files on
my machine, because every worker process on my machine opened 15892
sockets which seems to be very much compared to other machines I took a
look at. On other machines I only have as much open sockets as worker
processes.

Perhaps someone can point me to the right direction to debug and solve
this.

The system has 16 cores and 16 GB Ram (10 GB RAM used, Load average:
0.3) Ulimits are set to

su - www-data -s /bin/bash
$ ulimit -Hn
32768
$ ulimit -Sn
16384

In nginx.conf I set

user www-data;
worker_processes 12;
worker_rlimit_nofile 65536; # raised from 16384
worker_connections 65536; # raised from 16384

Checking the open files of the master process returns

[email protected]:~ # lsof -p 31022 | wc -l
16066

[email protected]:~ # lsof -p 31022 | grep socket | wc -l
15904

Checking the open sockets returns for every single worker process 15892
I restarted nginx several times, but this does not change anything.

This is what I get when I raise errorlog level to debug:

2013/10/25 12:31:40 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:31:40 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:31:40 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
00000000029B2B80:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113558 accept: 78.46.79.55
fd:21
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
000000000276D520:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113558 event timer add: 21:
60000:1382697160518
2013/10/25 12:31:40 [debug] 45534#0: *257113558 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113558 epoll add event: fd:21
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113558 post event
00007F7BB9FCEB70
2013/10/25 12:31:40 [debug] 45534#0: *257113558 delete posted event
00007F7BB9FCEB70
2013/10/25 12:31:40 [debug] 45534#0: *257113558 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113558 malloc:
0000000002BD1B50:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113558 recv: fd:21 39 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113558 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113558 posix_memalign:
0000000002B2BE80:4096 @16
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
0000000002A78700:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113587 accept: 78.46.79.55
fd:19
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
0000000002716BF0:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113587 event timer add: 19:
60000:1382697160776
2013/10/25 12:31:40 [debug] 45534#0: *257113587 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113587 epoll add event: fd:19
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113587 post event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 delete posted event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113587 malloc:
0000000002B89FC0:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113587 recv: fd:19 39 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113587 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 posix_memalign:
0000000002571760:4096 @16
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
00000000025F13C0:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113589 accept: 91.118.111.100
fd:19
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
0000000002658200:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113589 event timer add: 19:
60000:1382697160795
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113589 epoll add event: fd:19
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113589 post event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 delete posted event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113589 malloc:
0000000002927660:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 recv: fd:19 27 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 posix_memalign:
0000000002571760:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002BCE320:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 accept: 83.169.27.46
fd:47
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002B75DB0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 event timer add: 47:
60000:1382697161017
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113643 epoll add event: fd:47
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113589 post event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 delete posted event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113589 malloc:
0000000002927660:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 recv: fd:19 27 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 posix_memalign:
0000000002571760:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002BCE320:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 accept: 83.169.27.46
fd:47
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002B75DB0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 event timer add: 47:
60000:1382697161017
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113643 epoll add event: fd:47
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113643 post event
00007F7BB9FD0028
2013/10/25 12:31:41 [debug] 45534#0: *257113643 delete posted event
00007F7BB9FD0028
2013/10/25 12:31:41 [debug] 45534#0: *257113643 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113643 malloc:
0000000002B39900:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113643 recv: fd:47 27 of 1024
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113643 posix_memalign:
0000000003359E40:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002A40C80:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113751 accept: 77.75.254.73
fd:16014
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000029F4AE0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113751 event timer add: 16014:
60000:1382697161443
2013/10/25 12:31:41 [debug] 45534#0: *257113751 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113751 epoll add event:
fd:16014 op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113751 post event
00007F7BB9FD0C58
2013/10/25 12:31:41 [debug] 45534#0: *257113751 delete posted event
00007F7BB9FD0C58
2013/10/25 12:31:41 [debug] 45534#0: *257113751 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113751 malloc:
0000000002BB46F0:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113751 recv: fd:16014 27 of
1024
2013/10/25 12:31:41 [debug] 45534#0: *257113751 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113751 posix_memalign:
0000000002ADDDD0:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000028658D0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113758 accept: 78.46.79.55
fd:15779
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000026528B0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113758 event timer add: 15779:
60000:1382697161489
2013/10/25 12:31:41 [debug] 45534#0: *257113758 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113758 epoll add event:
fd:15779 op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113758 post event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113758 delete posted event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113758 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113758 malloc:
0000000002BB46F0:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113758 recv: fd:15779 39 of
1024
2013/10/25 12:31:41 [debug] 45534#0: *257113758 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113758 posix_memalign:
0000000002ADDDD0:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
000000000276C050:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113763 accept: 134.0.76.216
fd:51
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000029EE590:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113763 event timer add: 51:
60000:1382697161522
2013/10/25 12:31:41 [debug] 45534#0: *257113763 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113763 epoll add event: fd:51
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:31:41 [debug] 45534#0: *257113763 post event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113763 delete posted event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113763 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113763 malloc:
00000000034A7F00:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113763 recv: fd:51 27 of 1024
2013/10/25 12:31:41 [debug] 45534#0: *257113763 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113763 posix_memalign:
000000000322E2C0:4096 @16



2013/10/25 12:32:16 [debug] 45531#0: *257119454 epoll add event:
fd:14481 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45531#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45531#0: *257119454 post event
00007F7BB9FCFC18
2013/10/25 12:32:16 [debug] 45531#0: *257119454 delete posted event
00007F7BB9FCFC18
2013/10/25 12:32:16 [debug] 45531#0: *257119454 http wait request
handler
2013/10/25 12:32:16 [debug] 45531#0: *257119454 malloc:
0000000002BB42E0:1024
2013/10/25 12:32:16 [debug] 45531#0: *257119454 recv: fd:14481 27 of
1024
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45531#0: *257119454 reusable connection: 0
2013/10/25 12:32:16 [debug] 45531#0: *257119454 posix_memalign:
000000000286B6D0:4096 @16
2013/10/25 12:32:16 [debug] 45531#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45535#0: post event 00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45535#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45535#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:32:16 [debug] 45535#0: posix_memalign:
0000000002AC72E0:256 @16
2013/10/25 12:32:16 [debug] 45535#0: *257119455 accept: 77.75.254.133
fd:15745
2013/10/25 12:32:16 [debug] 45535#0: posix_memalign:
000000000276A4D0:256 @16
2013/10/25 12:32:16 [debug] 45535#0: *257119455 event timer add: 15745:
60000:1382697196597
2013/10/25 12:32:16 [debug] 45535#0: *257119455 reusable connection: 1
2013/10/25 12:32:16 [debug] 45535#0: *257119455 epoll add event:
fd:15745 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45535#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45535#0: *257119455 post event
00007F7BB9FD1000
2013/10/25 12:32:16 [debug] 45535#0: *257119455 delete posted event
00007F7BB9FD1000
2013/10/25 12:32:16 [debug] 45535#0: *257119455 http wait request
handler
2013/10/25 12:32:16 [debug] 45535#0: *257119455 malloc:
00000000033F7A20:1024
2013/10/25 12:32:16 [debug] 45535#0: *257119455 recv: fd:15745 27 of
1024
2013/10/25 12:32:16 [debug] 45535#0: *257119455 reusable connection: 0
2013/10/25 12:32:16 [debug] 45535#0: *257119455 posix_memalign:
00000000026AF8C0:4096 @16
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [info] 45879#0: Using 32768KiB of shared memory for
push module in /etc/nginx/nginx.conf:100
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: post event 00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45536#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45536#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:32:16 [debug] 45536#0: posix_memalign:
000000000298FF90:256 @16
2013/10/25 12:32:16 [debug] 45536#0: *257119502 accept: 46.252.18.244
fd:16232
2013/10/25 12:32:16 [debug] 45536#0: posix_memalign:
0000000003CC5040:256 @16
2013/10/25 12:32:16 [debug] 45536#0: *257119502 event timer add: 16232:
60000:1382697196773
2013/10/25 12:32:16 [debug] 45536#0: *257119502 reusable connection: 1
2013/10/25 12:32:16 [debug] 45536#0: *257119502 epoll add event:
fd:16232 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45536#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45536#0: *257119502 post event
00007F7BB9FD3358
2013/10/25 12:32:16 [debug] 45536#0: *257119502 delete posted event
00007F7BB9FD3358
2013/10/25 12:32:16 [debug] 45536#0: *257119502 http wait request
handler
2013/10/25 12:32:16 [debug] 45536#0: *257119502 malloc:
000000000263BD90:1024
2013/10/25 12:32:16 [debug] 45536#0: *257119502 recv: fd:16232 27 of
1024
2013/10/25 12:32:16 [debug] 45536#0: *257119502 reusable connection: 0
2013/10/25 12:32:16 [debug] 45536#0: *257119502 posix_memalign:
000000000323C850:4096 @16
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45532#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45532#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000

Thanks in advance,
Axel

Hello Aron,

Am 25.10.2013 18:35, schrieb Aron:

Hello,
Can you tell the output from this command ps -eLF |grep www-data |wc
-l ?

Yes, it’s 13

And What does /proc/sys/fs/nr_open output ?

1048576

Thanks, Axel

Hi,

tonight I force-restarted nginx and everything seems to be fixed.
Checking the sockets shortly after restart returned 48 sockets for the
master process and 36 sockets for worker processes.

Now, 14 hours later sockets raise again - there are now 1128 sockets for
the master and 1116 for workers.

I noted the difference of 12 sockets between master and workers. Theese
seem to be those for communication between master and workers, but I
don’t know how to classify this information.

Is this the intended behaviour of nginx or is it a bug in Ubuntus
ppa-package of nginx?

Regards, Axel

Am 26.10.2013 02:06, schrieb Axel:

Hello,
Can you tell the output from this command ps -eLF |grep www-data |wc -l
?
And What does /proc/sys/fs/nr_open output ?

Regards
Aron


From: Axel [email protected]
To: [email protected]
Sent: Friday, October 25, 2013 5:37 PM
Subject: Too many open files and unix

Hello,

today I had too many open files and I need some advice to investigate in
this issue.

I googled around and found a lot information how to solve this issue
(e.g. http://forum.nginx.org/read.php?2,234191,234191)
I finally was able to get rid of the error.

But I wondered about nginx behaviour with unix sockets and open files on
my machine, because every worker process on my machine opened 15892
sockets which seems to be very much compared to other machines I took a
look at. On other machines I only have as much open sockets as worker
processes.

Perhaps someone can point me to the right direction to debug and solve
this.

The system has 16 cores and 16 GB Ram (10 GB RAM used, Load average:
0.3) Ulimits are set to

su - www-data -s /bin/bash
$ ulimit -Hn
32768
$ ulimit -Sn
16384

In nginx.conf I set

user www-data;
worker_processes 12;
worker_rlimit_nofile 65536; # raised from 16384
worker_connections 65536; # raised from 16384

Checking the open files of the master process returns

[email protected]:~ # lsof -p 31022 | wc -l
16066

[email protected]:~ # lsof -p 31022 | grep socket | wc -l
15904

Checking the open sockets returns for every single worker process 15892
I restarted nginx several times, but this does not change anything.

This is what I get when I raise errorlog level to debug:

2013/10/25 12:31:40 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:31:40 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:31:40 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
00000000029B2B80:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113558 accept: 78.46.79.55
fd:21
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
000000000276D520:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113558 event timer add: 21:
60000:1382697160518
2013/10/25 12:31:40 [debug] 45534#0: *257113558 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113558 epoll add event: fd:21
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113558 post event
00007F7BB9FCEB70
2013/10/25 12:31:40 [debug] 45534#0: *257113558 delete posted event
00007F7BB9FCEB70
2013/10/25 12:31:40 [debug] 45534#0: *257113558 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113558 malloc:
0000000002BD1B50:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113558 recv: fd:21 39 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113558 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113558 posix_memalign:
0000000002B2BE80:4096 @16
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
0000000002A78700:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113587 accept: 78.46.79.55
fd:19
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
0000000002716BF0:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113587 event timer add: 19:
60000:1382697160776
2013/10/25 12:31:40 [debug] 45534#0: *257113587 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113587 epoll add event: fd:19
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113587 post event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 delete posted event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113587 malloc:
0000000002B89FC0:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113587 recv: fd:19 39 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113587 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113587 posix_memalign:
0000000002571760:4096 @16
2013/10/25 12:31:40 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:40 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
00000000025F13C0:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113589 accept: 91.118.111.100
fd:19
2013/10/25 12:31:40 [debug] 45534#0: posix_memalign:
0000000002658200:256 @16
2013/10/25 12:31:40 [debug] 45534#0: *257113589 event timer add: 19:
60000:1382697160795
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 1
2013/10/25 12:31:40 [debug] 45534#0: *257113589 epoll add event: fd:19
op:1 ev:80000001
2013/10/25 12:31:40 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113589 post event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 delete posted event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113589 malloc:
0000000002927660:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 recv: fd:19 27 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 posix_memalign:
0000000002571760:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002BCE320:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 accept: 83.169.27.46
fd:47
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002B75DB0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 event timer add: 47:
60000:1382697161017
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113643 epoll add event: fd:47
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:40 [debug] 45534#0: *257113589 post event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 delete posted event
00007F7BB9FCEAA0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 http wait request
handler
2013/10/25 12:31:40 [debug] 45534#0: *257113589 malloc:
0000000002927660:1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 recv: fd:19 27 of 1024
2013/10/25 12:31:40 [debug] 45534#0: *257113589 reusable connection: 0
2013/10/25 12:31:40 [debug] 45534#0: *257113589 posix_memalign:
0000000002571760:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002BCE320:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 accept: 83.169.27.46
fd:47
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002B75DB0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113643 event timer add: 47:
60000:1382697161017
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113643 epoll add event: fd:47
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113643 post event
00007F7BB9FD0028
2013/10/25 12:31:41 [debug] 45534#0: *257113643 delete posted event
00007F7BB9FD0028
2013/10/25 12:31:41 [debug] 45534#0: *257113643 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113643 malloc:
0000000002B39900:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113643 recv: fd:47 27 of 1024
2013/10/25 12:31:41 [debug] 45534#0: *257113643 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113643 posix_memalign:
0000000003359E40:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
0000000002A40C80:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113751 accept: 77.75.254.73
fd:16014
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000029F4AE0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113751 event timer add: 16014:
60000:1382697161443
2013/10/25 12:31:41 [debug] 45534#0: *257113751 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113751 epoll add event:
fd:16014 op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113751 post event
00007F7BB9FD0C58
2013/10/25 12:31:41 [debug] 45534#0: *257113751 delete posted event
00007F7BB9FD0C58
2013/10/25 12:31:41 [debug] 45534#0: *257113751 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113751 malloc:
0000000002BB46F0:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113751 recv: fd:16014 27 of
1024
2013/10/25 12:31:41 [debug] 45534#0: *257113751 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113751 posix_memalign:
0000000002ADDDD0:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000028658D0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113758 accept: 78.46.79.55
fd:15779
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000026528B0:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113758 event timer add: 15779:
60000:1382697161489
2013/10/25 12:31:41 [debug] 45534#0: *257113758 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113758 epoll add event:
fd:15779 op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45534#0: *257113758 post event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113758 delete posted event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113758 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113758 malloc:
0000000002BB46F0:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113758 recv: fd:15779 39 of
1024
2013/10/25 12:31:41 [debug] 45534#0: *257113758 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113758 posix_memalign:
0000000002ADDDD0:4096 @16
2013/10/25 12:31:41 [debug] 45534#0: post event 00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:31:41 [debug] 45534#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
000000000276C050:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113763 accept: 134.0.76.216
fd:51
2013/10/25 12:31:41 [debug] 45534#0: posix_memalign:
00000000029EE590:256 @16
2013/10/25 12:31:41 [debug] 45534#0: *257113763 event timer add: 51:
60000:1382697161522
2013/10/25 12:31:41 [debug] 45534#0: *257113763 reusable connection: 1
2013/10/25 12:31:41 [debug] 45534#0: *257113763 epoll add event: fd:51
op:1 ev:80000001
2013/10/25 12:31:41 [debug] 45534#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:31:41 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:31:41 [debug] 45534#0: *257113763 post event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113763 delete posted event
00007F7BB9FD0918
2013/10/25 12:31:41 [debug] 45534#0: *257113763 http wait request
handler
2013/10/25 12:31:41 [debug] 45534#0: *257113763 malloc:
00000000034A7F00:1024
2013/10/25 12:31:41 [debug] 45534#0: *257113763 recv: fd:51 27 of 1024
2013/10/25 12:31:41 [debug] 45534#0: *257113763 reusable connection: 0
2013/10/25 12:31:41 [debug] 45534#0: *257113763 posix_memalign:
000000000322E2C0:4096 @16



2013/10/25 12:32:16 [debug] 45531#0: *257119454 epoll add event:
fd:14481 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45531#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45531#0: *257119454 post event
00007F7BB9FCFC18
2013/10/25 12:32:16 [debug] 45531#0: *257119454 delete posted event
00007F7BB9FCFC18
2013/10/25 12:32:16 [debug] 45531#0: *257119454 http wait request
handler
2013/10/25 12:32:16 [debug] 45531#0: *257119454 malloc:
0000000002BB42E0:1024
2013/10/25 12:32:16 [debug] 45531#0: *257119454 recv: fd:14481 27 of
1024
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45531#0: *257119454 reusable connection: 0
2013/10/25 12:32:16 [debug] 45531#0: *257119454 posix_memalign:
000000000286B6D0:4096 @16
2013/10/25 12:32:16 [debug] 45531#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45535#0: post event 00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45535#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45535#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:32:16 [debug] 45535#0: posix_memalign:
0000000002AC72E0:256 @16
2013/10/25 12:32:16 [debug] 45535#0: *257119455 accept: 77.75.254.133
fd:15745
2013/10/25 12:32:16 [debug] 45535#0: posix_memalign:
000000000276A4D0:256 @16
2013/10/25 12:32:16 [debug] 45535#0: *257119455 event timer add: 15745:
60000:1382697196597
2013/10/25 12:32:16 [debug] 45535#0: *257119455 reusable connection: 1
2013/10/25 12:32:16 [debug] 45535#0: *257119455 epoll add event:
fd:15745 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45535#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45535#0: *257119455 post event
00007F7BB9FD1000
2013/10/25 12:32:16 [debug] 45535#0: *257119455 delete posted event
00007F7BB9FD1000
2013/10/25 12:32:16 [debug] 45535#0: *257119455 http wait request
handler
2013/10/25 12:32:16 [debug] 45535#0: *257119455 malloc:
00000000033F7A20:1024
2013/10/25 12:32:16 [debug] 45535#0: *257119455 recv: fd:15745 27 of
1024
2013/10/25 12:32:16 [debug] 45535#0: *257119455 reusable connection: 0
2013/10/25 12:32:16 [debug] 45535#0: *257119455 posix_memalign:
00000000026AF8C0:4096 @16
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [info] 45879#0: Using 32768KiB of shared memory for
push module in /etc/nginx/nginx.conf:100
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: post event 00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45536#0: delete posted event
00007F7BB9FCE010
2013/10/25 12:32:16 [debug] 45536#0: accept on 0.0.0.0:80, ready: 1
2013/10/25 12:32:16 [debug] 45536#0: posix_memalign:
000000000298FF90:256 @16
2013/10/25 12:32:16 [debug] 45536#0: *257119502 accept: 46.252.18.244
fd:16232
2013/10/25 12:32:16 [debug] 45536#0: posix_memalign:
0000000003CC5040:256 @16
2013/10/25 12:32:16 [debug] 45536#0: *257119502 event timer add: 16232:
60000:1382697196773
2013/10/25 12:32:16 [debug] 45536#0: *257119502 reusable connection: 1
2013/10/25 12:32:16 [debug] 45536#0: *257119502 epoll add event:
fd:16232 op:1 ev:80000001
2013/10/25 12:32:16 [debug] 45536#0: accept() not ready (11: Resource
temporarily unavailable)
2013/10/25 12:32:16 [debug] 45536#0: *257119502 post event
00007F7BB9FD3358
2013/10/25 12:32:16 [debug] 45536#0: *257119502 delete posted event
00007F7BB9FD3358
2013/10/25 12:32:16 [debug] 45536#0: *257119502 http wait request
handler
2013/10/25 12:32:16 [debug] 45536#0: *257119502 malloc:
000000000263BD90:1024
2013/10/25 12:32:16 [debug] 45536#0: *257119502 recv: fd:16232 27 of
1024
2013/10/25 12:32:16 [debug] 45536#0: *257119502 reusable connection: 0
2013/10/25 12:32:16 [debug] 45536#0: *257119502 posix_memalign:
000000000323C850:4096 @16
2013/10/25 12:32:16 [debug] 45530#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45532#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45530#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45532#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45535#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45536#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45535#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll add event: fd:97 op:1
ev:00000001
2013/10/25 12:32:16 [debug] 45536#0: epoll del event: fd:97 op:2
ev:00000000
2013/10/25 12:32:16 [debug] 45534#0: epoll del event: fd:97 op:2
ev:00000000

Thanks in advance,
Axel

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs