Understanding location blocks and try files

Hi,

I’m trying to understand a problem I’m facing in a typical
frontend-controller application.

I’ve setup a test config with a single simple server [1], and ran a test
script with debugging enabled to show what happens [2].

What confuses me is why this example is a 404:

curl -i http://nginx.dev/apples.json
HTTP/1.1 404 Not Found
Server: nginx/1.4.4

As can be seen in the log [3] there is an invalid response from
/index.php.
If I disable the location block adding cache headers for json files [4]
though, the response is fine.

Can someone shed some light as to why this happens? Is there a way to
define
location blocks for static files - without that causing problems for
dynamic
requests for the same url pattern?

Any help appreciated,

Cheers,

AD

[1] https://github.com/AD7six/server-configs-nginx
[2] https://gist.github.com/AD7six/eafe7cc6fc655c3195c4
[3]
https://gist.github.com/AD7six/eafe7cc6fc655c3195c4#file-error-log-L424
[4]
https://github.com/AD7six/server-configs-nginx/blob/location-debug/h5bp/location/expires.conf#L12

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,246713,246713#msg-246713

On 22 January 2014 18:31, AD7six [email protected] wrote:

Can someone shed some light as to why this happens?
I was half-way through writing a response when I noticed I’d slightly
misunderstood part of what you wrote. I thought about going back and
checking what I’d missed but to be brutally honest, the horrible maze
of includes I’d had to work my way through once already put me /right/
off!

Suggestions:

  1. Have a very careful read through http://nginx.org/r/location and
    nginx.org/r/try_files.

  2. Simplify your problem so you can present it inline to the list,
    without external links and without all those includes! You’ll get
    significantly more interest in your problem that way, and may actually
    discover the solution yourself :wink:

Cheers,
Jonathan

Thanks for the reply,

I’ve read through those sections again - if I’m missing something
obvious
I’m afraid I need someone to point it out to me :expressionless:

Sorry about that I thought pointing at a working example would allow
close
scrutiny - didn’t think to remove the files/config that weren’t in use.
Only
the mentioned location block is relevant, inlining that and the fastcgi
config becomes [1]:

server {
    listen 80;
    server_name nginx.dev *.nginx.dev;

    access_log /tmp/access.log;
    error_log  /tmp/error.log debug;

    error_page 404 /404.html;

    root /etc/nginx/www;
    index index.php index.html index.htm;

    try_files $uri $uri.html /index.php?$args;

    send_timeout 600s;

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_param   QUERY_STRING        $query_string;
        fastcgi_param   REQUEST_METHOD      $request_method;
        fastcgi_param   CONTENT_TYPE        $content_type;
        fastcgi_param   CONTENT_LENGTH      $content_length;

        fastcgi_param   SCRIPT_FILENAME     $request_filename;
        fastcgi_param   SCRIPT_NAME     $fastcgi_script_name;
        fastcgi_param   REQUEST_URI     $request_uri;
        fastcgi_param   DOCUMENT_URI        $document_uri;
        fastcgi_param   DOCUMENT_ROOT       $document_root;
        fastcgi_param   SERVER_PROTOCOL     $server_protocol;

        fastcgi_param   GATEWAY_INTERFACE   CGI/1.1;
        fastcgi_param   SERVER_SOFTWARE     nginx/$nginx_version;

        fastcgi_param   REMOTE_ADDR     $remote_addr;
        fastcgi_param   REMOTE_PORT     $remote_port;
        fastcgi_param   SERVER_ADDR     $server_addr;
        fastcgi_param   SERVER_PORT     $server_port;
        fastcgi_param   SERVER_NAME     $server_name;

        fastcgi_param   HTTPS           $https if_not_empty;

        # PHP only, required if PHP was built with
--enable-force-cgi-redirect
        fastcgi_param   REDIRECT_STATUS     200;
        fastcgi_pass unix:/tmp/php-fpm.sock;
        fastcgi_index index.php;
        fastcgi_intercept_errors on; # to support 404s for PHP files not
found
        fastcgi_param SCRIPT_FILENAME 
$document_root$fastcgi_script_name;
        fastcgi_read_timeout 600s;
    }

    location ~* \.(?:manifest|appcache|html?|xml|json)$ {
      add_header section "expires.conf:13";
      expires -1;
    }
}

The behavior is identical to as originally described.

A valid response where the url is a file:

$ curl -i http://nginx.dev/foo.json
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Wed, 22 Jan 2014 19:28:34 GMT
Content-Type: application/json
Content-Length: 20
Last-Modified: Wed, 22 Jan 2014 18:01:51 GMT
Connection: keep-alive
ETag: “52e0078f-14”
Expires: Wed, 22 Jan 2014 19:28:33 GMT
Cache-Control: no-cache
section: expires.conf:13
Accept-Ranges: bytes

An invalid response when passed to php:

$ curl -i http://nginx.dev/apples.json
HTTP/1.1 404 Not Found
Server: nginx/1.4.4
Date: Wed, 22 Jan 2014 19:28:40 GMT
Content-Type: text/html
Content-Length: 8
Connection: keep-alive
ETag: “52dffeed-8”

OH DEAR

Am I missing something obvious or falling for a common misunderstanding?
Is there a way to define location blocks for static files - without that
causing problems for dynamic requests for the same url pattern?

Cheers,

AD

[1]
https://github.com/AD7six/server-configs-nginx/blob/location-debug/sites-available/nginx.dev

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,246713,246718#msg-246718

On Wed, Jan 22, 2014 at 02:54:35PM -0500, AD7six wrote:

Hi there,

location ~ \.php$ {
location ~* \.(?:manifest|appcache|html?|xml|json)$ {

A valid response where the url is a file:

$ curl -i http://nginx.dev/foo.json
HTTP/1.1 200 OK

An invalid response when passed to php:

$ curl -i http://nginx.dev/apples.json
HTTP/1.1 404 Not Found

Why do you think that this request is passed to php?

Am I missing something obvious or falling for a common misunderstanding?

One request is handled in one location.

The request “/apples.json” is handled in the second location above.
Which
says (by omission of any other directive) “serve it from the file system
or return 404”.

The error_page for 404 is also handled in that location, and the
appropriate file is found and returned.

Is there a way to define location blocks for static files - without that
causing problems for dynamic requests for the same url pattern?

“location” matches the uri. “static file” or “dynamic request” is
irrelevant at that point.

So “no”; but also “probably”, depending on what exactly you want to do.

(The general suggestion in nginx is to use prefix-match locations at
the top level, not regex-match ones.)

f

Francis D. [email protected]

On 22 January 2014 20:36, Francis D. [email protected] wrote:

HTTP/1.1 200 OK

An invalid response when passed to php:

$ curl -i http://nginx.dev/apples.json
HTTP/1.1 404 Not Found

Why do you think that this request is passed to php?

I /believe/ AD is thinking along these lines:

  • I have a server{} level “try_files”, which goes $uri, $uri/,
    /index.php?$args;
  • When file.json is present in the server{}-level root, it should be
    served (and is) due to try_files trying the “$uri” setting first;
  • When file.json is /missing/, the try_files setting should then
    result in nginx falling back to the php location, which AD then
    expects to do something meaningful with this request …

… and its this last step which isn’t working as expected. I don’t
quite have the explanation or docs to hand to say why this won’t work,
but this SO page seems to have an interestingly un-up-voted answer at
the bottom of the page:
http://stackoverflow.com/questions/13138318/nginx-try-files-outside-location

“You are probably under the delusion that try_files on server level
must work for every request. Not at all. Quite the contrary, it works
only for requests that match no location blocks.”

I’d be really interested to get confirmation that that statement is
unequivocally true!

Jonathan

On Wednesday 22 January 2014 21:07:13 Jonathan M. wrote:
[…]

I’d be really interested to get confirmation that that statement is
unequivocally true!

Well, actually I wrote this answer. So, I can confirm. =)

You may also find confirmations by Igor and Maxim by searching
through the mailing list.

There is a bit of history:
http://mailman.nginx.org/pipermail/nginx/2009-March/010749.html
http://mailman.nginx.org/pipermail/nginx/2011-June/027502.html
http://mailman.nginx.org/pipermail/nginx/2012-June/034389.html

Changes with nginx 0.7.44 23 Mar
2009

*) Feature: the "try_files" directive is now allowed on the server 

block
level.

but, personally I think it was a bad decision that eventually resulted
to a
lot of confusion around.

wbr, Valentin V. Bartenev

On Wed, Jan 22, 2014 at 09:07:13PM +0000, Jonathan M. wrote:

On 22 January 2014 20:36, Francis D. [email protected] wrote:

On Wed, Jan 22, 2014 at 02:54:35PM -0500, AD7six wrote:

Hi there,

location ~ \.php$ {
location ~* \.(?:manifest|appcache|html?|xml|json)$ {

An invalid response when passed to php:

$ curl -i http://nginx.dev/apples.json
HTTP/1.1 404 Not Found

Why do you think that this request is passed to php?

I /believe/ AD is thinking along these lines:

  • I have a server{} level “try_files”, which goes $uri, $uri/, /index.php?$args;

That is correct.

  • When file.json is present in the server{}-level root, it should be
    served (and is) due to try_files trying the “$uri” setting first;

That is incorrect.

Either check the debug log, or add something like

return 200 “location#2\n”;

to the second location{} to get a better idea of what really happens.

  • When file.json is /missing/, the try_files setting should then
    result in nginx falling back to the php location,

That is incorrect.

Check the debug log; or make a request for /file.nomatch to see that
kind of behaviour.

“You are probably under the delusion that try_files on server level
must work for every request. Not at all. Quite the contrary, it works
only for requests that match no location blocks.”

I’d be really interested to get confirmation that that statement is
unequivocally true!

The true documentation is in the directory called “src”. All else is
interpretations :slight_smile:

But if you don’t want to accept someone else’s word on it, the debug
log is usually reliable; and adding different “return” messages to each
location may help you figure out what nginx is doing.

(Especially when you realise that rewrite-module directives are
special.)

f

Francis D. [email protected]

On Thu, Jan 23, 2014 at 06:55:04AM -0500, AD7six wrote:

Hi there,

Allow me to rephrase: Is there a way to define rules for static files -
without that causing problems for dynamic requests matching the same url
pattern?

Asking the question that way, the answer is probably “no”.

Whatever rules you write are based on url patterns (prefix or
regex). nginx does not know whether files are involved until you tell
it to look.

If you want a completely generic (= arbitrary) uri/file layout, you
probably won’t find an nginx config you can drop in and have Just Work.

Nginx docs are full of warnings about If is evil and advocates using
try_files instead - yet doing that is not functionally/logically equivalent,
which leaves me kind of stuck.

“If is evil” inside a location.

You tested try_files outside a location.

That might make a difference.

I did also try using the front controller as a 404 handler which worked
exactly how I am trying to get things to work except of course everything
from the front controller is a http 404.

Why “of course”?

http://nginx.org/r/error_page

Might the suggestions in and around

http://forum.nginx.org/read.php?2,244818,244886

offer a hint to what might work for you?

One of the goals
of asking here was to eventually achieve a “just include this config file”
solution to optimize frontend performance. This post so far leads to the
conclusion that’s not possible and an application-specific config file is
required for all apps.

I suspect that that’s pretty much required, based on the idea that
generic = slow, specific = fast; and nginx being built to be fast.

Also “optimize” can mean different things, depending on what
specifically
is to be improved at the expense of what else.

An example of the kind of apache rule I’d like to emulate with nginx is:

<FilesMatch "\.(eot|otf|ttc|ttf|woff)$">
    Header set Access-Control-Allow-Origin "*"
</FilesMatch>

nginx does not have any equivalent to apache FilesMatch, which is
approximately “when any filename which matches this pattern is served,
apply this configuration as well”.

You may be able to get close, using nginx configuration. But it may not
be close enough for you to be happy with.

handled by wordpress - fails.
That configuration says “all requests that match these patterns (apart
from those that match a different location) are files to be served from
the filesystem”.

If you have a request that matches this location that is not a file
to be served from the filesystem, that configuration is not the one you
want to use.

(Use, for example, “location ^~ /wordpress/” and a request for
/wordpress/one.ogg will not match the regex location above.)

(Or use “error_page 404” to decide what to do about “file” requests
which are missing.)

it also means you can’t optimize static requests and do “funky caching” or
similar by which I mean handling the first request for something that
doesn’t exist, and caching the result in a path such that the next and all
subsequent requests are served by the webserver:

fastcgi_cache? Or try_files?

I suspect I’m missing something obvious, but why can’t you do this? If
you want the effect of “expires max”, why do you care whether you are
serving from file, from cache, or from php directly?

Only if it matters: could you describe how you want nginx to handle
different requests, and how it should know to do what you want?

Thanks,

f

Francis D. [email protected]

I did also try using the front controller as a 404 handler which worked
exactly how I am trying to get things to work except of course
everything

from the front controller is a http 404.

Why “of course”?

http://nginx.org/r/error_page

I said “of course” as when processed as a 404 - it didn’t seem possible
to
modify the http response code. So all content was served as desired BUT
as a
404, however I’d missed this in the docs:

If an error response is processed by a proxied server or a FastCGI server,
and the server may return different response codes (e.g., 200, 302, 401
or
404), it is possible to respond with the code it returns:

error_page 404 = /404.php;

Therefore using the following config:

server {
listen 80;
server_name nginx.dev *.nginx.dev;

try_files $uri $uri.html;
error_page 404 = @app; <-

root /etc/nginx/www;
index index.php index.html index.htm;

send_timeout 600s;

location @app {
    fastcgi_param   QUERY_STRING        $query_string;
    ...

    add_header section "app"; <-
}

location ~* \.(?:manifest|appcache|html?|xml|json)$ {
  expires -1;

  add_header section "static json files"; <-
}

}

Gives the folllowing results:

$ curl -I http://nginx.dev/doesnotexist.json
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Fri, 24 Jan 2014 15:39:19 GMT
Content-Type: text/html
Connection: keep-alive
Vary: Accept-Encoding
section: app <-

$ curl -I http://nginx.dev/static.json
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Fri, 24 Jan 2014 15:40:25 GMT
Content-Type: application/json
Content-Length: 20
Last-Modified: Wed, 22 Jan 2014 18:01:51 GMT
Connection: keep-alive
ETag: “52e0078f-14”
Expires: Fri, 24 Jan 2014 15:40:24 GMT
Cache-Control: no-cache
section: static json files <-
Accept-Ranges: bytes

Which is exactly what I’m aiming for, it means it’s possible to define
rules for static files without the risk of dynamically served content
with
the same url pattern being inaccessible.

I suspect that that’s pretty much required, based on the idea that generic
= slow, specific = fast; and nginx being built to be fast.

Optimizing frontend performance is pretty easy, doesn’t require detailed
analysis and has signifiant benefits (especially for mobile devices). To
a
certain extent it doesn’t matter if the webserver is fast if it is
having to
serve more work because of unoptimized frontend performance (or though
the
absense of other headers meaning the application simply doesn’t work).

It should be an obvious and easy step to optimize the serving of an
application’s static files rather than (as it had seemed right up until
your
last post) an arduous cumbersome fragile process which can lead to
applications breaking. It doesn’t matter if an image is /here.png
/over/here.png or /way/way/over/here.png - if it’s a static file it
should
be served with appropriate cache headers. needing to wrap rules for
static
files in all locations where they may occur is in many cases not viable

since even with a structured application the files may be in
sufficiently
varied locations to always overlap with dynamic requests.

I suspect I’m missing something obvious, but why can’t you do this? If you
want the effect of “expires max”, why do you care whether you are
serving
from file, from cache, or from php directly?

Serving files with php is slower, depending on exactly what it’s doing
possibly a lot slower, than letting the webserver do it. At the very
least
it means reimplementing cache header handling and partial responses in
the
application layer. This is not a point to be taken lightly - I’d go as
far
as to say doing that when you don’t need to is a flat out bad idea.

Here is an example where the same url may or may not be a static file
http://book.cakephp.org/2.0/en/views/themes.html#increasing-performance-of-plugin-and-theme-assets
That’s a pattern which is used by many (most?) web frameworks - whereby
static files are used if they exist but handled transparently if they do
not.

If it could only be static OR dynamic that would mean one of:

  1. The file must always be served by php with specific headers.
  2. it is impossible to generate the file dynamically, but if it exists
    it
    has specific headers.
  3. It is possible to generate the file dynamically, but will have no
    specific headers once generated.
  4. Additional server config is required to cache the response from php
    (hadn’t really thought of that - but that’s quite cumbersome)

None of which are attractive.

But now, that’s all moot =). Thanks very much for your help identifying
that
a dynamic response can be used as a 404 handler and define the
response
code. I may propose a change to make that more obvious to future
readers/users.

Regards,

AD

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,246713,246797#msg-246797

Hey, thanks for the responses!

Indeed your input has been very insightful, some follow ups:

@Jonathan/@Valentin
Your analysis is about right, the linked prior posts on the mailing list
were very informative.

@Francis

An invalid response when passed to php:

$ curl -i http://nginx.dev/apples.json
HTTP/1.1 404 Not Found

Why do you think that this request is passed to php?

Because I (mis)read the debug log =) (linked in the first post).
Re-reading
it I was looking at the log for a different response.

Is there a way to define location blocks for static files - without that
causing problems for dynamic requests for the same url pattern?

“location” matches the uri. “static file” or “dynamic request” is
irrelevant at that point.

Allow me to rephrase: Is there a way to define rules for static files -
without that causing problems for dynamic requests matching the same url
pattern?

Nginx docs are full of warnings about If is evil and advocates using
try_files instead - yet doing that is not functionally/logically
equivalent,
which leaves me kind of stuck.

Either check the debug log, or add something like
return 200 “location#2\n”;

I’ve been using add_header section "file:line"; for this purpose - a
useful technique imo.

My conclusion at this time is that it’s not possible to define rules for
static files without adding try_files to all location blocks; or by
using
if (!-f $request_filename) { or by adding a nested location block
e.g.:

location ~

^/(?:apple-touch-icon-precomposed[^/]*.png$|crossdomain.xml$|favicon.ico$|css/|files/|fonts/|img/|js/)
{
require static-files.conf;
}

I did also try using the front controller as a 404 handler which worked
exactly how I am trying to get things to work except of course
everything
from the front controller is a http 404. Is there a way to prevent the
http
response code being set…?

Cheers,

AD


Further info =)

I feel I should give some context as to what I’m trying to do in general
so
here goes:

I use nginx for everything, as do many of my peers and colleagues -
we’re
typically backend developers who simply appreciate performance.
However, I
also use and contribute to http://html5boilerplate.com /
https://github.com/h5bp/html5-boilerplate - which many frontend
developers
use as the basis for the projects. Most frontend developers are familiar
with and use apache - their combined knowledge distilled into
https://github.com/h5bp/html5-boilerplate/blob/master/.htaccess -
knowledge
which I’d like to apply by default to the use of nginx.

There is an nginx version of the apache rules here
https://github.com/h5bp/server-configs-nginx which I’d really like to
get
into a position to point at - right now it’s kind of dis-functional in
part
because of the problem I’m trying to solve in this thread. One of the
goals
of asking here was to eventually achieve a “just include this config
file”
solution to optimize frontend performance. This post so far leads to the
conclusion that’s not possible and an application-specific config file
is
required for all apps.

An example of the kind of apache rule I’d like to emulate with nginx is:

<FilesMatch "\.(eot|otf|ttc|ttf|woff)$">
    Header set Access-Control-Allow-Origin "*"
</FilesMatch>

Here’s a similar example as a further reference:

http://www.servermom.org/setup-nginx-virtual-host-for-wordpress-with-wp-super-cache/262/
(I don’t use wordpress but it serves the purpose as an example)

The suggestion there is to include the following:

# Cache static files for as long as possible
location ~*

.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$
{
expires max; log_not_found off; access_log off;
}

Which would mean any request for those file types which is intended to
be
handled by wordpress - fails.

it also means you can’t optimize static requests and do “funky
caching” or
similar by which I mean handling the first request for something that
doesn’t exist, and caching the result in a path such that the next and
all
subsequent requests are served by the webserver:

<?php //index.php .... $response = ....; file_put_contents($_SERVER['REQUEST_URI'], $response); echo $response; I hope that's sufficient, should anyone have any general or specific advice I'd be very grateful to hear it. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,246713,246751#msg-246751

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs