Odd issue with proxy_pass serving wrong cached data

I’m having an odd issue, and I’m not sure where to start.

We’ve been implementing a number of NGINX servers recently, and one of
them is doing a proxy_pass to an older IIS7 server that we’ll be phasing
out soon, which is hosting a couple of minor sites in our datacenter.

When initially browsing the site, everything appears to be working
properly. But in most browsers, if you hit reload a couple of times the
NGINX server either serves the wrong files (i.e. the page calls for
Image1.PNG, but the server is returning the contents of Image2.PNG), or
simply fails to load the image or file at all. It will always load the
base HTML page, but the linked files (css files, images, etc.) seem to
randomly glitch.

Because it only happens when we click reload, I’m presuming that we’re
looking at some sort of configuration problem reading the proxy_pass
cache. The problem doesn’t occur and the sites work normally if we
bypass the NGINX server and connect directly to the content origin
server.

Anyone have any idea where I should start with this one? It’s a bit of a
bizarre problem, and it’s really got me scratching my head. I’ve done a
number of searches and can’t find anything online discussing a similar
problem.

BH

user www-data;
worker_processes 4
pid /var/run/nginx.pid;
error_log /var/log/nginx/error/error.log crit;

events {
worker_connections 20000;
use epoll;
multi_accept on;
accept_mutex off;
}

http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
client_header_timeout 20;
client_body_timeout 20;
reset_timedout_connection on;
types_hash_max_size 2048;
server_tokens off;

            include /etc/nginx/mime.types;
            default_type application/octet-stream;

#gzip
gzip on;
gzip_disable “MSIE [1-6].”;
gzip_http_version 1.0;
gzip_vary on;
gzip_comp_level 6;
gzip_min_length 512;
gzip_proxied any;
gzip_buffers 16 8k;
gzip_types text/plain text/css application/json
application/x-javascript text/xml application/xml application/xml+rss
text/javascript;

            #caching
            open_file_cache max=200000 inactive=20s;
            open_file_cache_valid 30s;
            open_file_cache_min_uses 2;
            open_file_cache_errors on;

proxy_cache_path /tmp/nginxcache levels=1:2:2 keys_zone=mycache:3096m
max_size=3584m inactive=240m;
proxy_temp_path /tmp/nginxtmp;
proxy_redirect off;

            upstream wyebase {
                            server   10.64.1.69:80;

}

server {
listen 80 default_server;
server_name www.domain.edu;
proxy_redirect
off;
proxy_set_header Host
$host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header
X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 75s;
proxy_send_timeout 75s;
proxy_read_timeout 75s;
proxy_pass_header Expires;
proxy_pass_header Cache-Control;
proxy_pass_header Last-Modified;
proxy_pass_header ETag;
proxy_pass_header Content-Length;

                            proxy_cache      mycache;
                            proxy_cache_key 

“$scheme://$host$request_uri”;
proxy_cache_bypass
$http_secret_header;
proxy_ignore_headers “Cache-Control”;
proxy_ignore_headers “Set-Cookie”;
add_header X-Cache-Status
$upstream_cache_status;

                            location / {
                                            proxy_cache_valid 

200 301 302 304 10m; #good requests
proxy_cache_valid
404 403 10m; #access errors
proxy_cache_valid
500 501 502 503 504 1m; # execution or load errors
proxy_cache_use_stale
error timeout invalid_header updating http_500 http_502 http_503
http_504;
expires
60m;
proxy_pass
http://wyebase;
}

                            location ~* 

.(css|js|png|jpe?g|gif|ico)$ {
proxy_cache_valid
200 301 302 304 1h; #good requests
proxy_cache_valid
404 403 10m; #access errors
proxy_cache_valid
500 501 502 503 504 1m; # execution or load errors
proxy_cache_use_stale
error timeout invalid_header updating http_500 http_502 http_503
http_504;
expires 14d;
proxy_pass
http://wyebase;
}
}

}

I’m having an odd issue, and I’m not sure where to start. We’ve been
implementing a number of NGINX servers recently, and one of them is …

http://time4livetv.blogspot.com/2014/02/south-park.html

Posted at Nginx Forum:

Hello!

On Wed, Feb 26, 2014 at 02:32:25AM +0000, Brian Hill wrote:

So now it doesn’t look like it’s a caching issue at all. I’ve
completely gutted my config files and stripped it down, and I’m
still seeing the same issue. I even shot a video of what I’m
seeing and stuck it on YouTube as an example
(NGINX oddity - YouTube). As the video shows, the
connections for the linked images don’t load properly on reloads
if I’m connecting through the NGINX server, but work fine if I
connect directly to the backend origin server. Has anyone ever
seen anything like this? Any ideas?

Looking into logs usually helps a lot.

Symptoms suggest most likely you’ve run out of local ports on
nginx host due to sockets in TIME-WAIT state (and you have no
tw_reuse/tw_recycle switched on), but it’s hard to tell anything
exact from the information you’ve provided.


Maxim D.
http://nginx.org/

disable gzip, sendfile off, use something else then epoll, disable
proxy_cache_path and take it from there to see if it still happens.

Posted at Nginx Forum:

So now it doesn’t look like it’s a caching issue at all. I’ve completely
gutted my config files and stripped it down, and I’m still seeing the
same issue. I even shot a video of what I’m seeing and stuck it on
YouTube as an example (NGINX oddity - YouTube). As the video shows,
the connections for the linked images don’t load properly on reloads if
I’m connecting through the NGINX server, but work fine if I connect
directly to the backend origin server. Has anyone ever seen anything
like this? Any ideas?

My new minimalist config files (I’ve included the SSL version this time
as well):

user www-data;
worker_processes 4
pid /var/run/nginx.pid;
error_log /var/log/nginx/error/error.log crit;
events {
worker_connections 20000;
use epoll;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
proxy_cache_path /tmp/nginxcache levels=1:2:2
keys_zone=mycache:3096m max_size=3584m inactive=240m;
proxy_temp_path /tmp/nginxtmp;

            upstream wyebase {
                            server   10.64.1.69:80;

}

upstream wyessl {
server 10.64.1.69:443;
}
server {
listen 80 default_server;
server_name www.domain.edu;

                            location / {
                                            proxy_pass 

http://wyebase;
}
}

            server {
                            listen 443 ssl spdy;
                            server_name www.domain.edu;
                            include /etc/nginx/ssl.conf;
                            ssl_certificate 

/etc/nginx/ssl/xbs2014.crt;
ssl_certificate_key
/etc/nginx/ssl/xbs2014.key;

                            location / {
                                            proxy_pass 

https://wyessl;
}
}
}

BH

Sent: Tuesday, February 25, 2014 9:52 AM
To: [email protected]
Subject: Odd issue with proxy_pass serving wrong cached data

I’m having an odd issue, and I’m not sure where to start.

We’ve been implementing a number of NGINX servers recently, and one of
them is doing a proxy_pass to an older IIS7 server that we’ll be phasing
out soon, which is hosting a couple of minor sites in our datacenter.

When initially browsing the site, everything appears to be working
properly. But in most browsers, if you hit reload a couple of times the
NGINX server either serves the wrong files (i.e. the page calls for
Image1.PNG, but the server is returning the contents of Image2.PNG), or
simply fails to load the image or file at all. It will always load the
base HTML page, but the linked files (css files, images, etc.) seem to
randomly glitch.

Because it only happens when we click reload, I’m presuming that we’re
looking at some sort of configuration problem reading the proxy_pass
cache. The problem doesn’t occur and the sites work normally if we
bypass the NGINX server and connect directly to the content origin
server.

Anyone have any idea where I should start with this one? It’s a bit of a
bizarre problem, and it’s really got me scratching my head. I’ve done a
number of searches and can’t find anything online discussing a similar
problem.

BH

user www-data;
worker_processes 4
pid /var/run/nginx.pid;
error_log /var/log/nginx/error/error.log crit;

events {
worker_connections 20000;
use epoll;
multi_accept on;
accept_mutex off;
}

http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
client_header_timeout 20;
client_body_timeout 20;
reset_timedout_connection on;
types_hash_max_size 2048;
server_tokens off;

            include /etc/nginx/mime.types;
            default_type application/octet-stream;

#gzip
gzip on;
gzip_disable “MSIE [1-6].”;
gzip_http_version 1.0;
gzip_vary on;
gzip_comp_level 6;
gzip_min_length 512;
gzip_proxied any;
gzip_buffers 16 8k;
gzip_types text/plain text/css application/json
application/x-javascript text/xml application/xml application/xml+rss
text/javascript;

            #caching
            open_file_cache max=200000 inactive=20s;
            open_file_cache_valid 30s;
            open_file_cache_min_uses 2;
            open_file_cache_errors on;

proxy_cache_path /tmp/nginxcache levels=1:2:2 keys_zone=mycache:3096m
max_size=3584m inactive=240m;
proxy_temp_path /tmp/nginxtmp;
proxy_redirect off;

            upstream wyebase {
                            server   10.64.1.69:80;

}

server {
listen 80 default_server;
server_name
www.domain.eduhttp://www.domain.edu;
proxy_redirect
off;
proxy_set_header Host
$host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header
X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 75s;
proxy_send_timeout 75s;
proxy_read_timeout 75s;
proxy_pass_header Expires;
proxy_pass_header Cache-Control;
proxy_pass_header Last-Modified;
proxy_pass_header ETag;
proxy_pass_header Content-Length;

                            proxy_cache      mycache;
                            proxy_cache_key 

“$scheme://$host$request_uri”;
proxy_cache_bypass
$http_secret_header;
proxy_ignore_headers “Cache-Control”;
proxy_ignore_headers “Set-Cookie”;
add_header X-Cache-Status
$upstream_cache_status;

                            location / {
                                            proxy_cache_valid 

200 301 302 304 10m; #good requests
proxy_cache_valid
404 403 10m; #access errors
proxy_cache_valid
500 501 502 503 504 1m; # execution or load errors
proxy_cache_use_stale
error timeout invalid_header updating http_500 http_502 http_503
http_504;
expires
60m;
proxy_pass
http://wyebase;
}

                            location ~* 

.(css|js|png|jpe?g|gif|ico)$ {
proxy_cache_valid
200 301 302 304 1h; #good requests
proxy_cache_valid
404 403 10m; #access errors
proxy_cache_valid
500 501 502 503 504 1m; # execution or load errors
proxy_cache_use_stale
error timeout invalid_header updating http_500 http_502 http_503
http_504;
expires 14d;
proxy_pass
http://wyebase;
}
}

}

Interestingly, this will teach me not to leave out “irrelevant” bits of
my config files when posting them. My configs failed to include a few
items like ModSecurity, because it didn’t occur to me that they could
have any impact on the problem I was describing. As it turns out,
disabling ModSecurity solved the problem. I’m still not sure why
ModSecurity is causing NGINX to serve bad data on reloads, but it’s
clearly a ModSecurity issue or a problem with the communication between
ModSecurity and NGINX.

BH