Hi,
We’ve recently shifted to FreeBSD-10 due to its robust asynchronous
performance for big storage based on .mp4 files. Here is the server
specs :
2 x Intel Xeon X5690
96GB DDR3 Memory
12 x 3TB SATA Raid-10 (HBA LSI-9211)
ZFS FileSystem with 18TB usable space
2 x 1Gbps LACP (2Gbps Throughput)
Things are working quite well, no high I/O due to Big Ram cache and AIO
performance but once network port started to go over 1Gbps, performance
begins to lag, download speed started to stuck around 60-100Kbps on a
4Mbps
connection (using wget) which works quite efficient under 800Mbps port
(450kbps on 4Mbps). We first thought it could be network issue or LACP
issue but doesn’t looks like it is. We also checked that if requests are
in
queue using following command but it was ‘0’:
[root@cw005 ~/scripts]# netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen Local Address
tcp4 0/0/6000 *.80
tcp4 0/0/6000 *.443
tcp4 0/0/10 127.0.0.1.25
tcp4 0/0/128 *.1880
tcp6 0/0/128 *.1880
tcp4 0/0/5 *.5666
tcp6 0/0/5 *.5666
tcp4 0/0/128 *.199
unix 0/0/6000 /var/run/www.socket
unix 0/0/4 /var/run/devd.pipe
unix 0/0/4 /var/run/devd.seqpacket.pipe
Here is the output of mbcluster :
119747/550133/669880/6127378 mbuf clusters in use
(current/cache/total/max)
661065/1410183/2071248/6063689 4k (page size) jumbo clusters in use
(current/cache/total/max)
We also checked with Disk Busy rate using gstat which was quite stable
as
well.
So it looks like either the sysctl values need to tweak or Nginx
configurations are not optimized. Here is the sysctl.conf :
kern.ipc.somaxconn=6000
set to at least 16MB for 10GE hosts
kern.ipc.maxsockbuf=16777216
socket buffers
net.inet.tcp.recvspace=4194304
net.inet.tcp.sendspace=4197152
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
security
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
drop UDP packets destined for closed sockets
net.inet.udp.blackhole=1
drop TCP packets destined for closed sockets
net.inet.tcp.blackhole=2
ipfw
net.inet.ip.fw.verbose_limit=3
maximum incoming and outgoing IPv4 network queue sizes
net.inet.ip.intr_queue_maxlen=2048
net.route.netisr_maxqlen=2048
net.inet.icmp.icmplim: 2048
net.inet.tcp.fast_finwait2_recycle=1
kern.random.sys.harvest.ethernet=0
net.inet.ip.portrange.randomized=0
net.link.lagg.0.use_flowid=0
Here is the bootloader.conf :
zpool_cache_load=“YES”
zpool_cache_type=“/boot/zfs/zpool.cache”
zpool_cache_name=“/boot/zfs/zpool.cache”
aio_load=“YES”
zfs_load=“YES”
ipmi_load=“YES”
Here is the nginx.conf :
user www www;
worker_processes 48;
worker_rlimit_nofile 900000; #2 filehandlers for each connection
error_log /var/log/nginx-error.log error;
#pid logs/nginx.pid;
events {
worker_connections 10240;
multi_accept on;
}
http {
include mime.types;
default_type application/octet-stream;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection “1; mode=block”;
client_max_body_size 4096M;
client_body_buffer_size 800M;
output_buffers 1 512k;
sendfile_max_chunk 128k;
fastcgi_connect_timeout 30;
fastcgi_send_timeout 30;
fastcgi_read_timeout 30;
proxy_read_timeout 30;
fastcgi_buffer_size 64k;
fastcgi_buffers 16 64k;
fastcgi_temp_file_write_size 256k;
server_tokens off; #Conceals nginx version
access_log off;
sendfile off;
tcp_nodelay on;
aio on;
client_header_timeout 30s;
client_body_timeout 30s;
send_timeout 30s;
keepalive_timeout 15s;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
gzip off;
gzip_vary on;
gzip_disable "MSIE [1-6]\.";
gzip_proxied any;
gzip_http_version 1.0;
gzip_min_length 1280;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_types text/plain text/xml text/css application/x-javascript
image/png image/x-icon image/gif image/jpeg image/jpg application/xml
application/xml+rss text/javascr ipt application/atom+xml;
include /usr/local/etc/nginx/vhosts/*.conf;
}
Here is the vhost :
server {
listen 80 sndbuf=16k;
server_name cw005.files.com cw005.domain.com
www.cw005.files.com www.cw005.domain.com cw005.domain.net
www.cw005.domain.net;
location / {
root /files;
index index.html index.htm index.php;
autoindex off;
}
location ~ .(jpg)$ {
* sendfile on;*
tcp_nopush on;
* aio off;*
root /files;
try_files $uri /thumbs.php;
expires 1y;
}
location ~* \.(js|css|png|gif|ico)$ {
root /files;
expires 1y;
log_not_found off;
}
location ~ \.(flv)$ {
flv;
root /files;
expires 7d;
include hotlink.inc;
}
location ~ \.(mp4)$ {
mp4;
mp4_buffer_size 4M;
mp4_max_buffer_size 20M;
expires 1y;
add_header Cache-Control "public";
root /files;
include hotlink.inc;
}
pass the PHP scripts to FastCGI server listening on
unix:/var/run/www.socket
location ~ .php$ {
root /files;
fastcgi_pass unix:/var/run/www.socket;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME
$document_root$fastcgi_script_name;
include fastcgi_params;
fastcgi_read_timeout 10000;
}
location ~ /\.ht {
deny all;
}
}
====================================================
Please i need guidance to handle with this problem, i am sure that some
value needs to tweak.
Thanks in advance !!