Randomly stopping large file downloads?

Hi,
1.) What would cause nginx to randomly stop serving large file
downloads? When I try to download a 800MB file it stops serving the file
after between 10 and 30 MB. I’ve had to revert to lighttpd because of
this.

I checked the usual things: There wasn’t anything in the error log, the
system had plenty of memory (several hundred MB free) and the worker
processes were all still there after the download dies. It doesn’t
matter what browser I use.

Any idea what would cause this?

2.) BTW there’s a bug in the forum.nginx.org forum where when you first
get an account it prepopulates the login form with your email address
rather than your user name. If you try to log in like that it won’t let
you in. You have to clear out what it prepopulated and enter your
username. This is confusing.

Posted at Nginx Forum:

On 5/19/11 4:34 PM, nickn wrote:

Any idea what would cause this?

2.) BTW there’s a bug in the forum.nginx.org forum where when you first
get an account it prepopulates the login form with your email address
rather than your user name. If you try to log in like that it won’t let
you in. You have to clear out what it prepopulated and enter your
username. This is confusing.

First, this is an issue with how your browser is “pre-populating” that
field, not the forum software. See attached for what I see when I
attempt to log in. This behavior is consistent in Firefox, Chrome, and
Safari.

Second, a “Username” is a user name and an email is an email. You should
be able to distinguish between the two. Again, please see the attached
login form. This hardly seems “confusing” to me. Again please see the
attached.

An ideas why nginx would be cutting the downloads short?

Posted at Nginx Forum:

Hello!

On Thu, May 19, 2011 at 04:34:34PM -0400, nickn wrote:

Hi,
1.) What would cause nginx to randomly stop serving large file
downloads? When I try to download a 800MB file it stops serving the file
after between 10 and 30 MB. I’ve had to revert to lighttpd because of
this.

Are you serving file as static one with nginx, or proxy_pass to
some backend server? In later case most likely reason is out of
space (or no access to) in proxy_temp_path.

I checked the usual things: There wasn’t anything in the error log, the
system had plenty of memory (several hundred MB free) and the worker
processes were all still there after the download dies. It doesn’t
matter what browser I use.

Any idea what would cause this?

Please make sure you have error_log properly configured and you
are looking at right one. Most simple aproach is to configure one
at global level and remove others if any (i.e. at http, server,
etc. levels), e.g.

error_log /path/to/error.log info;

Note that setting logging level to ‘info’ may be important in some
cases (e.g. if nginx thinks client timed out or so on).

If the above wouldn’t be enough to solve you problem, please
follow generic instructions as outlined at
Debugging | NGINX, i.e. please provide

  1. nginx -V output
  2. full nginx config
  3. debug log

Maxim D.

Hello!

On Fri, May 20, 2011 at 03:43:53PM -0400, nickn wrote:

Here’s the info below, any ideas what it could be?
Timeout to tune is send_timeout, not keepalive one. Anyway,
please provide debug log and tcpdump of the connection in
question.

[…]

root@217948:/var/log/nginx# nginx -V
nginx version: nginx/0.7.65

Note that 0.7.65 is rather old. You may want to upgrade to 1.0.2.

[…]

sendfile        on;

When using sendfile with big files it’s good idea to use something
like

  sendfile_max_chunk 1m;

as well.

[…]

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

If there are any files here except the one you already showed -
you may want to show them as well (or at least parts which may
affect server{} in question).

Maxim D.

Hi Maxim, thanks for your help.
I’m not using a proxy, it’s just a straight connection to nginx.

I ran it again with error log on info and found that it thinks the
client is timing out, although watching firefox it seemed to be
downloading ok the entire time and I wasn’t having connectivity issues
on other sites.

I tried keepalive_timeout 65; rather than 5 but had the same problem.

Here’s the info below, any ideas what it could be?


/var/log/nginx# tail cosmic-access.log
76.250.143.130 - - [20/May/2011:19:32:50 +0000] “GET /bigfile.sql.gz
HTTP/1.1” 200 114499284 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17”

/var/log/nginx# tail cosmic-error.log
2011/05/20 19:32:50 [info] 12222#0: *10 client timed out (110:
Connection timed out) while sending response to client, client:
76.250.143.130, server: cosmicchannelings.com, request: “GET
/bigfile.sql.gz HTTP/1.1”, host: “cosmicchannelings.com:81

root@217948:/var/log/nginx# nginx -V
nginx version: nginx/0.7.65
TLS SNI support enabled
configure arguments: --conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid
–lock-path=/var/lock/nginx.lock
–http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/lib/nginx/body
–http-proxy-temp-path=/var/lib/nginx/proxy
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug
–with-http_stub_status_module --with-http_flv_module
–with-http_ssl_module --with-http_dav_module
–with-http_gzip_static_module --with-http_realip_module --with-mail
–with-mail_ssl_module --with-ipv6
–add-module=/build/buildd/nginx-0.7.65/modules/nginx-upstream-fair


/etc/nginx/nginx.conf:

user www-data;
#worker_processes 1;
worker_processes 2;

pid /var/run/nginx.pid;

events {
worker_connections 1024;
# multi_accept on;
}

http {
include /etc/nginx/mime.types;

error_log  /var/log/nginx/error.log info;
access_log  /var/log/nginx/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
#keepalive_timeout  65;
keepalive_timeout  5;
tcp_nodelay        on;

gzip on;

gzip_disable “MSIE [1-6].(?!.*SV1)”;

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

}

mail {

# See sample authentication script at:

# http://wiki.nginx.org/NginxImapAuthenticateWithApachePhpScript

# auth_http localhost/auth.php;

# pop3_capabilities “TOP” “USER”;

# imap_capabilities “IMAP4rev1” “UIDPLUS”;

server {

listen localhost:110;

protocol pop3;

proxy on;

}

server {

listen localhost:143;

protocol imap;

proxy on;

}

}


/etc/nginx/sites-enabled/sites-enabled/cosmic:

You may add here your

server {

}

statements for each of your virtual hosts

server {
listen 81;
server_name cosmicchannelings.com www.cosmicchannelings.com;

access_log /var/log/apache2/cosmic-access.log;

error_log /var/log/apache2/cosmic-error.log;

access_log /var/log/nginx/cosmic-access.log;
error_log /var/log/nginx/cosmic-error.log info;

root /var/www/cosmic;
index index.php index.html index.htm;

    # wordpress url writing
    error_page  404              /blog/index.php;

if ($http_user_agent ~ “magpie-crawler” ) { return 403; }

    # prevents nginx from returning 404s all the time along with

good content
try_files $uri $uri/ /blog/index.php;

    # also works, may be slower?
    # if (!-e $request_filename) {
    #        rewrite ^/blog/(.+)$ /blog/index.php?q=$1 last;
    # }

cave yahoo group

“^/i/([a-z0-9_-]+)/([a-z0-9_-]+)/([a-z0-9_-]+)?$” =>

“/cave/link.php?m=$1&g=$2&s=$3”,
rewrite ^/i/([a-z0-9_-]+)/([a-z0-9_-]+)/([a-z0-9_-]+)?$
/cave/link.php?m=$1&g=$2&s=$3 permanent;

convert old blog url format to new. old had post id, new only has

slug

“^(/blog/[a-z0-9_-]+)/[0-9]+$” => “$1”,

rewrite ^(/blog/[a-z0-9_-]+)/[0-9]+$ $1 permanent;

fix broken link errors. /blog/k_karma_29/family.html =>

/blog/k_family

“^/blog/k_[a-z0-9_-]+/([a-z0-9_-]+).html+$” => “/blog/k_$1”,

rewrite ^/blog/k_[a-z0-9_-]+/([a-z0-9_-]+).html+$ /blog/k_$1
permanent;

/blog/tag/new-world-order/topic.html

“^/blog/tag/[a-z0-9_-]+/([a-z0-9_-]+).html+$” => “/blog/k_$1”,

rewrite ^/blog/tag/[a-z0-9_-]+/([a-z0-9_-]+).html+$ /blog/k_$1
permanent;

redirect old archive html pages to blog pages

“^/archive/topic/([a-z0-9_-]+).html+$” => “/blog/k_$1”,

rewrite ^/archive/topic/([a-z0-9_-]+).html+$ /blog/k_$1 permanent;

fix bug where topic lines in keyword pages got linked as

http://www.cosmicchannelings.com/blog/heaven.html

“^/blog/([a-z0-9_-]+).html+$” => “/blog/k_$1”

rewrite ^/blog/([a-z0-9_-]+).html+$ /blog/k_$1 permanent;

redirect server error pages to the static page /50x.html

#error_page 500 502 503 504 /50x.html;
#location = /50x.html {

root /var/www/nginx-default;

#}

proxy the PHP scripts to Apache listening on 127.0.0.1:80

#location ~ .php$ {
#proxy_pass http://127.0.0.1;
#}

pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000

location ~ .php$ {
    fastcgi_split_path_info ^(.+\.php)(.*)$;
    fastcgi_pass   backend;
    fastcgi_index  index.php;
    #fastcgi_param  SCRIPT_FILENAME

/var/www/cosmic$fastcgi_script_name;
fastcgi_param SCRIPT_FILENAME
$document_root$fastcgi_script_name;

    include fastcgi_params;
    fastcgi_param  QUERY_STRING     $query_string;
    fastcgi_param  REQUEST_METHOD   $request_method;
    fastcgi_param  CONTENT_TYPE     $content_type;
    fastcgi_param  CONTENT_LENGTH   $content_length;
    fastcgi_intercept_errors        on;
    fastcgi_ignore_client_abort     off;
    fastcgi_connect_timeout 60;
    fastcgi_send_timeout 180;
    fastcgi_read_timeout 180;

    #fastcgi_buffer_size 128k;
    #fastcgi_buffers 4 256k;

from Nginx + Php-fpm = Intermittent 502 Bad Gateway errors

fastcgi_buffers 256 16k;
fastcgi_buffer_size 32k;

    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
}

deny access to .htaccess files, if Apache’s document root

concurs with nginx’s one

location ~ /.ht {
deny all;
}
}

upstream backend {
server 127.0.0.1:9000;
}

Posted at Nginx Forum: