Problem with Nginx returning correct content, but with 404

Hello,

We have a setup where we have requests coming into a cache server which
is in front of an app server. The requests hit nginx on the cache
server, which makes the request to a process running on the cache
server. That process will return either the requested content (if in the
cache) or a 404. In the case of a 404, nginx on the cache server should
then turn around and make the same request to the app server. The app
server serves the content, and then nginx on the app server should turn
around and return the response to the client.

The problem we are seeing is that this works fine for requests of the
form /admin/fan_pages, but when requesting css or png content (like
/stylesheets/common.css , or /images/admin/fan_pages/refresh.png ), it
is returning the correct content, but with a 404 code. In this case, the
local process on the cache server is returning a 404, the nginx on the
cache server is requesting the page from the app server (which returns
the page with a 200), but then the cache server is turning around and
serving the correct content to the client with a 404.

Anybody have any ideas on how/why this is happening? I’ll include config
files and output below. If there’s a simpler or better setup to
accomplish what we’re trying to do, I’m happy to try that as well.

Thanks,
–Austin

ngrep output (.185 is the cache server, .165 is the app server):

T 24.227.153.188:54091 → 10.102.38.185:80 [AP]
GET /stylesheets/common.css HTTP/1.1.
Host: test5.involver.com.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8.
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8.
Accept-Language: en-us,en;q=0.5.
Accept-Encoding: gzip,deflate.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Keep-Alive: 115.
Connection: keep-alive.
Cookie:
involver-auth=eyJlbWFpbCI6ImF1c3Rpbi5taWxsc0BpbnZvbHZlci5jb20iLCJzaWduYXR1%0AcmUiOiJLTDA5TmZvMG9QR3V2OUl3QTFkekFWOG1rU04wdkU1d2UyMzBvSjA3%0ATGhvUzJSTDNQOU5tdjRwM09FOFRcblFlUzUvWkFScmxXT29UeldIOVlONTJa%0AUEFzVzdDOXlQdUI1UUZDZURZR3dLZUgzT2RTMVpKbm96ZTY0Q1xuYjk2QUhU%0Aa25FVGdpWmRHaFA4eVJkaE13SDB3YmVkS25MejNGeC9Wd2I4WDhYSDNVM1Yw%0AMkl3RUwwRjNDXG5BS3h3ekFTMlhINHU4b0RXUVM1VkorZ2c4eTFUS2V0UCtw%0AbUVZbXVwN01pcUlXa3BDRnBBSVN2WU5TeXZcbmQ0ekhXWWhDSUE5Ulo4UXY.
.

T 10.102.38.185:42920 → 10.102.38.165:80 [AP]
GET /stylesheets/common.css HTTP/1.0.
X-Real-IP: 24.227.153.188.
X-Forwarded-For: 24.227.153.188.
Host: test5.involver.com.
Connection: close.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8.
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8.
Accept-Language: en-us,en;q=0.5.
Accept-Encoding: gzip,deflate.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Cookie:
involver-auth=eyJlbWFpbCI6ImF1c3Rpbi5taWxsc0BpbnZvbHZlci5jb20iLCJzaWduYXR1%0AcmUiOiJLTDA5TmZvMG9QR3V2OUl3QTFkekFWOG1rU04wdkU1d2UyMzBvSjA3%0ATGhvUzJSTDNQOU5tdjRwM09FOFRcblFlUzUvWkFScmxXT29UeldIOVlONTJa%0AUEFzVzdDOXlQdUI1UUZDZURZR3dLZUgzT2RTMVpKbm96ZTY0Q1xuYjk2QUhU%0Aa25FVGdpWmRHaFA4eVJkaE13SDB3YmVkS25MejNGeC9Wd2I4WDhYSDNVM1Yw%0AMkl3RUwwRjNDXG5BS3h3ekFTMlhINHU4b0RXUVM1VkorZ2c4eTFUS2V0UCtw%0AbUVZbXVwN01pcUlXa3BDRnBBSVN2WU5TeXZcbmQ0ekhXWWhDSUE5Ulo4UXY.
.

T 10.102.38.165:80 → 10.102.38.185:42920 [A]
HTTP/1.1 200 OK.
Server: nginx/0.7.65.
Date: Thu, 20 Jan 2011 01:14:02 GMT.
Content-Type: text/css.
Last-Modified: Wed, 19 Jan 2011 22:20:44 GMT.
Connection: close.
Expires: Sun, 17 Jan 2021 01:14:02 GMT.
Cache-Control: max-age=315360000.
Content-Encoding: gzip.
.
…ZYo.H.~…E#~.n[email protected] $.c.........{... ............?............}~.~.....….y…?..|?Z.N.z.>…$.=…I…%…U…U…D.:a5…~.nE…e…V…V.L…rs…YQ…d6…D.Z…Z…D.E…y…
…)e…7…]yD.tV[^…1*J.=?#6{q&.d.G..s...^_...2+x]-2..q.~...t.K.....4!.vbQ.u&.1I.;.%…/…E…0I.m…o…4…]u…!..+…<{…$.Ws6.Av.Q<…UE.&…J)M..?y7.#k.m].@…H.:.P.G.j…X_Q…+.v+…10Z…-.y…xy}./…+…Eo.-.<…Q…P….…&…
…“…$.9<.I.O.H?..?.)…Ed…Ncp…].U.E…47…:.t…r~…x…i…X…h…E.]…d…gh3uLo:…](…(”.c.1…:y.=T…i’…3iW.E.Q(.XB…PC…o.W…b…t…[…6\Q.io…H…D…Q…/…M
.5k.p@S.}.o<…[.h!k.$.Q4.‘…wiR.I…C…2…Z…L…a…Fs…B…hHo8…c.d…HxK9…’{…%}…h…e.z.>…?.
…^.,(.c
…X2…t…P.Q.s)…N]…$.1.?..@…,s…Z…1X…M…pF…d.,…a…y…l.P…R.‘(h.
…Ek…N…G…O$…#…k…L.t…4=…E^1r.&…SK)…[].X…j(.U…V.V1H…gQ…u.Nk…m.7&…L.M…|K.H…,8…I8…J…y.F…0…T.4.
x.,!..b.O…&…q.9.f.u…E…<…8…8.A…B&@.&…/…]…OX.~.+nR.z,n…
…{
…{S…]L….......LoG..[3..3.W|.f...t]......}/..X..5.P...Q.w.b.P..Ix.e(.h.T.0.......y.....l[..J...a2..>.R.{%e .,[email protected]....[..5.../p...;..Uu.....Y....t4.D._.K..'.U....R.z.*..+...Kp.....L"......"...tb...0.uB..p$+.@D5.{..+Uk..*....M.?[4..D....S.^~x..a ..J.F+.i'...=L2._.........X...._.....2..]..k...o(.lp....\....\)...*....T.......L.O..z...Z9.......RP/I.&......(...q.7[.Pb!..7.>...<./..'.P....c"...A..c .sf8..........d.MA.;…H…6.ut2K…@…g7…P…6.3p…Z…"…l…}V60.x.y.#…)y.P…X…C}.t…9…7…b…)9.T…m…^7.%…2!~L…1R…%…:uBt.3…@…B…&.d…r:….A..'..B.y.......|........F...b....n.@.../.y.O.$v.)....u*B..P......f..x8nD.6...T..z.L.{.j...|..#...b......M.u...|)td........G...B.$.wp>[email protected]&j..].l.w..A....+....%[email protected].%i.....:Y].._.l......./,..Q..Lj....|+)...........7.-R.....(..][email protected].......... ..2..k....$e.P.... .v..r....U.sx...]...$$$ .4.&.........G.....-...o1...G<....[...~i.....xp.ar.....n...|....W......B+...z.+c5 ....B.N...>.R...zX.....~.s..J..............0..r.6.5....?L>...........V4&...Z.,uWWWNVl.l7:.0..}Jv....:....=...H..>...2p.z..(..W.:......D..X.I'.a../H....(9.o..6[..G..9..p.kq._. ....&..$C.r...2y:!.d].....…Y.|.D.~d…S.K…>}…V.m…%…<…-wE…’
?.#1…1NZ.p…'=.{…Ic^…4…E5…lWf.x}.n.JT…{…iU-…y…5[.+a…2…J…?I…//…*e4a.i…I/9.^…NR…i.M…i3}:…

T 10.102.38.165:80 → 10.102.38.185:42920 [AP]
.?A…H…Q…?z.‘…6…L…]9…)…a_.:]B.X…K…RO.eQ…7.ko.$R.C…P.8…,.$…@.[.E$L…q.e.%8…cY…c#.
.K…~!’…Y.-.v…}^…t.7.N1.t;.x.vq…(^…=…E|…/…R.;…l<…0/.+t…q_.?}…J…/.56.A…`…Ad…2g…0.9.2…=…&…X…4…1…u…QhqIQ='…@7z…0…+…

T 10.102.38.185:80 → 24.227.153.188:54091 [A]
HTTP/1.1 404 Not Found.
Server: nginx/0.7.65.
Date: Thu, 20 Jan 2011 01:14:02 GMT.
Content-Type: text/css.
Transfer-Encoding: chunked.
Connection: keep-alive.
Last-Modified: Wed, 19 Jan 2011 22:20:44 GMT.
Expires: Sun, 17 Jan 2021 01:14:02 GMT.
Cache-Control: max-age=315360000.
Content-Encoding: gzip.
.
b86.
…ZYo.H.~…E#~.n[email protected] $.c.........{... ............?............}~.~.....….y…?..|?Z.N.z.>…$.=…I…%…U…U…D.:a5…~.nE…e…V…V.L…rs…YQ…d6…D.Z…Z…D.E…y…
…)e…7…]yD.tV[^…1*J.=?#6{q&.d.G..s...^_...2+x]-2..q.~...t.K.....4!.vbQ.u&.1I.;.%…/…E…0I.m…o…4…]u…!..+…<{…$.Ws6.Av.Q<…UE.&…J)M..?y7.#k.m].@…H.:.P.G.j…X_Q…+.v+…10Z…-.y…xy}./…+…Eo.-.<…Q…P….…&…
…“…$.9<.I.O.H?..?.)…Ed…Ncp…].U.E…47…:.t…r~…x…i…X…h…E.]…d…gh3uLo:…](…(”.c.1…:y.=T…i’…3iW.E.Q(.XB…PC…o.W…b…t…[…6\Q.io…H…D…Q…/…M
.5k.p@S.}.o<…[.h!k.$.Q4.‘…wiR.I…C…2…Z…L…a…Fs…B…hHo8…c.d…HxK9…’{…%}…h…e.z.>…?.
…^.,(.c
…X2…t…P.Q.s)…N]…$.1.?..@…,s…Z…1X…M…pF…d.,…a…y…l.P…R.'(h.
…Ek…N…G…O$…#…k…L.t…4=…E^1r.&…SK)…[].X…j(.U…V.V1H…gQ…u.Nk…m.7&…L.M…|K.H…,8…I8…J…y.F…0…T.4.
x.,!..b.O…&…q.9.f.u…E…<…8…8.A…B&@.&…/…]…OX.~.+nR.z,n…
…{
…{S…]L….......LoG..[3..3.W|.f...t]......}/..X..5.P...Q.w.b.P..Ix.e(.h.T.0.......y.....l[..J...a2..>.R.{%e .,[email protected]....[..5.../p...;..Uu.....Y....t4.D._.K..'.U....R.z.*..+...Kp.....L"......"...tb...0.uB..p$+.@D5.{..+Uk..*....M.?[4..D....S.^~x..a ..J.F+.i'...=L2._.........X...._.....2..]..k...o(.lp....\....\)...*....T.......L.O..z...Z9.......RP/I.&......(...q.7[.Pb!..7.>...<./..'.P....c"...A..c .sf8..........d.MA.;…H…6.ut2K…@…g7…P…6.3p…Z…"…l…}V60.x.y.#…)y.P…X…C}.t…9…7…b…)9.T…m…^7.%…2!~L…1R…%…:uBt.3…@…B…&.d…r:….A..'..B.y.......|........F...b....n.@.../.y.O.$v.)....u*B..P......f..x8nD.6...T..z.L.{.j...|..#...b......M.u...|)td........G...B.$.wp>[email protected]&j..].l.w..A....+....%[email protected].%i.....:Y].._.l......./,..Q..Lj....|+)...........7.-R.....(..][email protected].......... ..2..k....$e.P.... .v..r....U.sx...]...$$$ .4.&.........G.....-...o1...G<....[...~i.....xp.ar.....n...|....W......B+...z.+c5 ....B.N...>.R...zX.....~.s..J..............0..r.6.5....?L>...........V4&...Z.,uWWWNVl.l7:.0..}Jv....:....=...H..>...2p.z..(..W.:......D..X.I'.a../H....(9.o..6[..G..9..p.kq._. ....&..$C.r...2y:!.d].....…Y.|.D.~d…S.K…>}…V

T 10.102.38.185:80 → 24.227.153.188:54091 [AP]
.m…%…<…-wE…’
?.#1…1NZ.p…‘=.{…Ic^…4…E5…lWf.x}.n.JT…{…iU-…y…5[.+a…2…J…?I…//…*e4a.i…I/9.^…NR…i.M…i3}:…?A…H…Q…?z.’…6…L…]9…)…a_.:]B.X…K…RO.eQ…7.ko.$R.C…P.8…,.$…@.[.E$L…q.e.%8…cY…c#.
.K…~!‘…Y.-.v…}^…t.7.N1.t;.x.vq…(^…=…E|…/…R.;…l<…0/.+t…q_.?}…J…/.56.A…`…Ad…2g…0.9.2…=…&…X…4…1…u…QhqIQ=’…@7z…0…+…

T 10.102.38.185:80 → 24.227.153.188:54091 [AP]
0.
.

######################################################

nginx.conf:

user involver;
worker_processes 4;
pid /var/run/nginx.pid;
worker_rlimit_nofile 10240;

events {
worker_connections 8192;
use epoll;
}

http {

include /etc/nginx/mime.types;

default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] ’
'“$request” $status $body_bytes_sent “$http_referer”

‘“$http_user_agent” “$http_x_forwarded_for”’;

main access log

access_log /data/nginx/log/nginx_access.log main;

main error log

error_log /data/nginx/log/nginx_error.log debug;

sendfile on;

tcp_nopush on;

include /etc/nginx/apps/*.conf;

}

##########################################

apps/involver-cache-server.conf (test5.involver.com is the app server):

upstream switching_backend {
server localhost:81 weight=100;
server test5.involver.com;
}

upstream rails_backend {
server test5.involver.com;
}

upstream cache_backend {
server localhost:8190;
server localhost:8191;
server localhost:8192;
server localhost:8193;
}

server {
listen 80 default;
server_name _;
root /data/apps/involver-cache-server/current/public;

Fail fast with a 404 for server pages that rails backend will never

serve
location ~* .(php|jsp|asp|aspx|cfm|pl|py|tpl)$ { return 404; }

include /etc/nginx/common/app.conf;
include /etc/nginx/common/proxy.conf;

location = /file_uploads {
client_max_body_size 200m;
proxy_pass http://rails_backend;
break;
}

location ~* /rest/features/(.)/content_items/(.)/images {
proxy_pass http://rails_backend;
break;
}

location /assets/audio_uploads { return 404; }
location /assets {
proxy_pass http://rails_backend;
break;
}

location /cache {
proxy_pass http://rails_backend;
break;
}

location / {
proxy_next_upstream http_404 http_500 http_502 http_503 http_504
error timeout;
proxy_pass http://switching_backend;
}

}

OFFLOADED SSL

server {
listen 88;
server_name _;
root /data/apps/involver-cache-server/current/public;

Fail fast with a 404 for server pages that rails backend will never

serve
location ~* .(php|jsp|asp|aspx|cfm|pl|py|tpl)$ { return 404; }

include /etc/nginx/common/app.conf;
include /etc/nginx/common/proxy.conf;

proxy_set_header X-FORWARDED_PROTO https;

location ~* /rest/features/(.)/content_items/(.)/images {
proxy_pass http://rails_backend;
break;
}

location /assets/audio_uploads { return 404; }
location /assets {
proxy_pass http://rails_backend;
break;
}

location /cache {
proxy_pass http://rails_backend;
break;
}

location / {
proxy_next_upstream http_404 http_500 http_502 http_503 http_504
error timeout;
proxy_pass http://switching_backend;
}
}

CACHING HANDLER

server {
listen 81;
server_name _;
root /data/apps/involver-cache-server/current/public;

include /etc/nginx/common/proxy.conf;

location / {
proxy_pass http://cache_backend;
}
}

#############################################

common/app.conf:

Max size for file uploads

client_max_body_size 100M;

This allows people to use images and css in their maintenance.html

file
if ($request_filename ~* .(css|jpg|gif|png)$) {
break;
}

Rewrite all the requests to the maintenance.html page if it exists.

if (-f $document_root/system/maintenance.html) {
return 503;
}

error_page 404 /404.html;
error_page 500 502 504 /500.html;

error_page 503 @503;
location @503 {
rewrite ^(.*)$ /system/maintenance.html break;
}

##################################################

common/proxy.conf:

index index.html index.htm;

needed to forward user’s IP address to rails

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_redirect off;
proxy_max_temp_file_size 0;

server_name_in_redirect off;

set Expire header on assets: see

location ~ ^/(images|javascripts|stylesheets)/ {
expires 10y;
}

serve any existing file

if (-f $request_filename) {
break;
}

serve any standard Rails page cache file with .html extension

if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}

Posted at Nginx Forum:

Hello!

On Thu, Jan 20, 2011 at 11:23:17AM -0500, austinmills wrote:

form /admin/fan_pages, but when requesting css or png content (like
/stylesheets/common.css , or /images/admin/fan_pages/refresh.png ), it
is returning the correct content, but with a 404 code. In this case, the
local process on the cache server is returning a 404, the nginx on the
cache server is requesting the page from the app server (which returns
the page with a 200), but then the cache server is turning around and
serving the correct content to the client with a 404.

Anybody have any ideas on how/why this is happening? I’ll include config
files and output below. If there’s a simpler or better setup to
accomplish what we’re trying to do, I’m happy to try that as well.

Request to /stylesheets/common.css will generate 404 as long
as you don’t have static file on disk, as it’s matched by this
location in your config:

set Expire header on assets: see

location ~ ^/(images|javascripts|stylesheets)/ {
expires 10y;
}

You have 404 error_page set without overwriting status code:

error_page 404 /404.html;

I.e. 404 code will be preserved regardless of future processing.
This explains 404 status code.

Future processing will happen here:

location / {
proxy_next_upstream http_404 http_500 http_502 http_503 http_504 error
timeout;
proxy_pass http://switching_backend;
}

As proxy_pass without uri component is used, it will preserve
original request uri when asking backend. This explains correct
content.

Maxim D.

Maxim,

That was exactly it. I removed the location and all is well. Many
thanks!

–Austin

Posted at Nginx Forum: