Connection timeout on download server

I have an nginx setup on multiple download servers. I’ve setup a monitor
server to check my servers at interval and when the servers seem to be
their peak I get connection timeout from my servers. I’ve set limit_conn
for each remote_ip
to make sure one user can’t hog the server.
today when I was receiving the timeout warnings, I checked the stub
of my server and it seems the server connection is nowhere near it’s
This is the stub status page:
Server 1:
Active connections: 764
server accepts handled requests
326451 326451 346395
Reading: 0 Writing: 604 Waiting: 160

Server 2:
Active connections: 426
server accepts handled requests
538407 538407 576918
Reading: 0 Writing: 418 Waiting: 8

However my config files are way too high:

worker_processes 1;
pid /run/;
worker_rlimit_nofile 90000;

events {
worker_connections 64000;
multi_accept on;
#aio on; turned off for linux
upload_progress uploads 5m;
sendfile off;
output_buffers 1 1m;
directio 512;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
server_tokens off;
server_names_hash_bucket_size 128;
variables_hash_max_size 1024;


limit perip

limit_rate_after 1m;
limit_rate 96k;
limit_conn perip 64;

nginx version: nginx/1.6.2
Linux s17 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24)
x86_64 GNU/Linux

Am I missing something? Is there another connection limitation I should

Posted at Nginx Forum: