Forum: NGINX Clean-URL rewrite rule with nested "location" and alias directive

79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-19 16:37
(Received via mailing list)
Hi!

I am attempting to serve a staging website from a directory that is
outside of the web-server's document root, while at the same time making
the site accessible at a URL that "appears" to be a subdirectory of the
top-level domain. (I have to do this so that the SSL certificate for the
TLD can be used for the staging site.)

This works as expected, with one major exception: "clean-URL" rewriting
does not work. In other words, the homepage (/stage/) loads correctly,
but all sub-pages return a 404. As soon as I attempt to add "rewrite"
directives for clean-URLs, the server either returns a 500 (redirect
cycle) or "Primary script unknown".

I have tried adding "rewrite_log on;" in the "server" block, just before
this configuration snippet (and reloading the nginx config), but the
directive seems to have no effect (no rewrite information is logged to
error.log). So, I am unable to determine what is happening here.

Here's the configuration snippet:

location ^~ /stage/ {
  alias /var/www/example.com/private/stage/web/;

  if ($scheme = http) {
    return 301 https://$server_name$request_uri;
  }

  index index.php index.html index.htm;

  location ~ ^/stage/(.+\.php)$ {
    alias /var/www/example.com/private/stage/web/$1;

    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_param HTTPS on;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include /etc/nginx/fastcgi_params;

    # Trying to implement clean-URLs with this line,
    # but it causes "primary script unknown" when using
    # the "break" keyword, and redirect cycle when using
    # the "last" keyword.
    rewrite '^(.*)$'  /stage/index.php?q=$1  break;
  }
}

Any assistance with making this to work is hugely appreciated. Thanks in
advance!

-Ben
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-19 18:38
(Received via mailing list)
On Tue, Nov 19, 2013 at 10:36:53AM -0500, Ben Johnson wrote:

Hi there,

> This works as expected, with one major exception: "clean-URL" rewriting
> does not work. In other words, the homepage (/stage/) loads correctly,
> but all sub-pages return a 404. As soon as I attempt to add "rewrite"
> directives for clean-URLs, the server either returns a 500 (redirect
> cycle) or "Primary script unknown".

Can you give an example of a "clean-URL" that matches the location that
you use, and that does not do what you want it to do?

Note that your rewrite only happens for urls that end ".php".

  f
--
Francis Daly        francis@daoine.org
79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-19 19:46
(Received via mailing list)
On 11/19/2013 12:38 PM, Francis Daly wrote:
> Can you give an example of a "clean-URL" that matches the location that
> you use, and that does not do what you want it to do?
>
> Note that your rewrite only happens for urls that end ".php".
>
>   f
>

Thanks for your help, Francis!

Certainly: an example URL is /stage/my-account/ . Ultimately, I would
like for this URL to be rewritten to
/stage/index.php?q=/stage/my-account/ (the file "index.php" exists on
the filesystem at /var/www/example.com/private/stage/web/index.php).

Regarding the rewrite happening only for URLs that end in .php, that had
occurred to me, but if I try to move the rewrite "up one level", into
the block "location ^~ /stage/ {", nginx complains upon requesting the
URL /stage/ that '"alias" cannot be used in location "/stage/" where URI
was rewritten'.

Fundamentally, is the problem here that I need to be modifying the
"alias" directive dynamically (with regular expressions and capture
groups), instead of trying to achieve the same with the "rewrite"
directive?

Thanks again for your time,

-Ben
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-19 21:39
(Received via mailing list)
On Tue, Nov 19, 2013 at 01:45:15PM -0500, Ben Johnson wrote:
> On 11/19/2013 12:38 PM, Francis Daly wrote:
> > On Tue, Nov 19, 2013 at 10:36:53AM -0500, Ben Johnson wrote:

Hi there,

> >> This works as expected, with one major exception: "clean-URL" rewriting
> >> does not work. In other words, the homepage (/stage/) loads correctly,
> >> but all sub-pages return a 404.

Which nginx version do you use, where you get that output with the
config
snippet that you provided?

I've tried 1.4.3 and 1.5.7, and I don't get /stage/ to load correctly.

I suspect I'm just doing something silly.

> Certainly: an example URL is /stage/my-account/ . Ultimately, I would
> like for this URL to be rewritten to
> /stage/index.php?q=/stage/my-account/ (the file "index.php" exists on
> the filesystem at /var/www/example.com/private/stage/web/index.php).

I think that the problem may be due to the interaction between "alias"
and "$document_root" being not-always-obvious.

And $document_root is used in try_files, which is probably what you
want for "clean URLs" -- serve the file if present, else hand it to
a controller.

Does the following do what you want for all cases you care about?

===
      location ^~ /stage/ {
        alias /var/www/example.com/private/stage/web/;
        index index.php index.html index.htm;
        try_files $uri $uri/ /stage//stage/index.php?q=$uri;

        location ~ ^/stage/(.+\.php)$ {
                alias /var/www/example.com/private/stage/web/$1;
                try_files "" / /stage/index.php?q=$uri;
                fastcgi_pass unix:/var/run/php5-fpm.sock;
                fastcgi_param HTTPS on;
                fastcgi_param SCRIPT_FILENAME $request_filename;
                include /etc/nginx/fastcgi_params;
        }
      }
===

You may want to use something other than $uri in the last argument to
try_files, depending on how you want /stage/my-account/?key=value to
be processed.

And the curious-looking try_files lines are deliberate.

  f
--
Francis Daly        francis@daoine.org
79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-19 23:50
(Received via mailing list)
On 11/19/2013 3:39 PM, Francis Daly wrote:
> Which nginx version do you use, where you get that output with the config
> snippet that you provided?
>
> I've tried 1.4.3 and 1.5.7, and I don't get /stage/ to load correctly.
>
> I suspect I'm just doing something silly.
>

Given your level of expertise, I doubt that you're doing anything wrong.
:) I'm using nginx-1.1.19, because that's what's bundled with Ubuntu
12.04 LTS. It makes sense that without the rewrite rules in-place, only
"index.php" (via /stage/) returns a 200 response and not a 404.

> a controller.
>

I think that you're exactly right. I had tried try_files first, but was
unable to get it to work given that this site a) must be accessed via a
"subdirectory" relative to the domain-root URL, and b) is comprised of
files that live in a "private" directory that is outside of the
server-root on the filesystem.

>                 try_files "" / /stage/index.php?q=$uri;
>                 fastcgi_pass unix:/var/run/php5-fpm.sock;
>                 fastcgi_param HTTPS on;
>                 fastcgi_param SCRIPT_FILENAME $request_filename;
>                 include /etc/nginx/fastcgi_params;
>         }
>       }
> ===
>

Wow, that actually works! While I don't fully understand the try_files
magic, I didn't think try_files would ever serve the purpose because I
had read at http://wiki.nginx.org/HttpCoreModule#alias : "Note that
there is a longstanding bug that alias and try_files don't work
together" (with link to http://trac.nginx.org/nginx/ticket/97 ).

While I do realize that the above-cited Wiki is obsolete and the current
reference is at
http://nginx.org/en/docs/http/ngx_http_core_module... , the
referenced bug does not appear to have been closed.

Is the implication here that "alias" does indeed work with "try_files"
(even in my stale nginx-1.1.19 version)? At least in this particular
use-case?

There is one last trick to pull-off, which is to add very similar
clean-URL functionality for two other files (in addition to index.php),
but I am hoping that I will be able to adapt your working sample myself.

> You may want to use something other than $uri in the last argument to
> try_files, depending on how you want /stage/my-account/?key=value to
> be processed.
>

Understood; that makes sense.

> And the curious-looking try_files lines are deliberate.
>
>   f
>

You are a true guru, Francis. I can't thank you enough; I've been
struggling with this for days. This is the second time that you've
come-up huge for me.

Respectfully,

-Ben
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-20 10:11
(Received via mailing list)
On Tue, Nov 19, 2013 at 05:48:59PM -0500, Ben Johnson wrote:
> On 11/19/2013 3:39 PM, Francis Daly wrote:

Hi there,

> :) I'm using nginx-1.1.19, because that's what's bundled with Ubuntu

It's good to know that this configuration works on older versions too.

> files that live in a "private" directory that is outside of the
> server-root on the filesystem.

Generally it shouldn't be a problem -- file space and url space are
different, and one of the main points of web server configuration is to
map between them.

If you do have free choice in the matter, some things work more easily
within nginx if you can use "root" and not "alias" -- so if you want
files to be accessible below the url /stage/, having them directly in
a directory called /stage/ is convenient.

(So if you can either get rid of the web/ directory and move all
contents
up one, or add a stage/ below web/ and move all contents down one,
then set "root" appropriately and the remaining configuration may become
simpler.)

> >                 fastcgi_param HTTPS on;
> >                 fastcgi_param SCRIPT_FILENAME $request_filename;
> >                 include /etc/nginx/fastcgi_params;
> >         }
> >       }
> > ===

> had read at http://wiki.nginx.org/HttpCoreModule#alias : "Note that
> there is a longstanding bug that alias and try_files don't work
> together" (with link to http://trac.nginx.org/nginx/ticket/97 ).

> Is the implication here that "alias" does indeed work with "try_files"
> (even in my stale nginx-1.1.19 version)? At least in this particular
> use-case?

I'd say rather that this configuration works with the current
implementation of the defect.

So if the defect is fixed, this configuration will start to fail, but a
more obvious working configuration will be available; but if the defect
is changed or partly fixed, this configuration may start to fail without
an equivalent workaround.

The ticket-#97 page currently (last modified date 2013-08-23) lists
three aspects in the description.

The second aspect is, as I understand it, that in a prefix location with
alias, if the fallback contains a $variable and begins with the location
prefix, then the location prefix is stripped before the internal rewrite
is done. In this configuration, that's why the extra /stage/ is in the
first try_files.

The third aspect is, as I understand it, that in a regex location
with alias, $document_root is set to the alias, which in this case is
equivalent to the wanted filename. That's why an argument of "" finds
the file, if it is present, in the second try_files.

I strongly suspect that the second argument there, /, can safely be
dropped -- it can only apply if there is a directory called
something.php;
and in that case, I see no difference with the / there or not (in
1.5.7).

> There is one last trick to pull-off, which is to add very similar
> clean-URL functionality for two other files (in addition to index.php),
> but I am hoping that I will be able to adapt your working sample myself.

I don't fully follow what you mean there; but once you can isolate your
requests in a location, then you should be able to set the fallback to
whatever you want.

It may get to the point that it is clearer to use a @named location as
the fallback, and just hardcode SCRIPT_FILENAME in them. You'll see when
you do it, no doubt.

> > You may want to use something other than $uri in the last argument to
> > try_files, depending on how you want /stage/my-account/?key=value to
> > be processed.

The usual caveats apply here regarding url escaping -- you've copied
an unescaped string directly into query string, so if there were any
characters like % or + or & involved, they may cause confusion.

Cheers,

  f
--
Francis Daly        francis@daoine.org
79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-21 04:32
(Received via mailing list)
On 11/20/2013 4:10 AM, Francis Daly wrote:

> If you do have free choice in the matter, some things work more easily
> within nginx if you can use "root" and not "alias" -- so if you want
> files to be accessible below the url /stage/, having them directly in
> a directory called /stage/ is convenient.
>
> (So if you can either get rid of the web/ directory and move all contents
> up one, or add a stage/ below web/ and move all contents down one,
> then set "root" appropriately and the remaining configuration may become
> simpler.)
>

Ultimately, I have control of the configuration, so I could go either
route.

But in the first case, my hesitation is two-fold:

a.) I prefer to maintain identical directory structures within my
staging and production environments. This allows my build scripts and
relative paths to remain identical (provided that I set a "root path"
variable or similar in each shell script), and it helps me to remember
to where I must "cd" in order to execute a build script.

b.) I really prefer to keep PHP and PHP-template files out of the
document root (on the filesystem). I don't want user-agents to be able
to request .tpl files directly, for example, by requesting
/templates/home.tpl. The same applies to configuration files, e.g.,
/config.inc.php. Most folks have seen what happens when a
misconfiguration allows a "sensitive configuration file" to be
downloaded as a plaintext file. For these reasons, and as a matter of
course, I build my applications such that every request is routed
through a single point that has complete authority over how the request
is handled.

In the second case, this seems risky; if I were to put the "stage"
directory within the production site's document root, it's conceivable
that doing so could have unintended consequences, such as the staging
site being deleted inadvertently during the build process (think rsync
with the --delete flag, as an
off-the-cuff-though-not-something-I-would-actually-do-myself example of
a crude "clean build process"). Issue b.) from above applies here, too.

>>>                 fastcgi_param HTTPS on;
>> Is the implication here that "alias" does indeed work with "try_files"
>
> with alias, $document_root is set to the alias, which in this case is
> equivalent to the wanted filename. That's why an argument of "" finds
> the file, if it is present, in the second try_files.
>
> I strongly suspect that the second argument there, /, can safely be
> dropped -- it can only apply if there is a directory called something.php;
> and in that case, I see no difference with the / there or not (in 1.5.7).
>

Your ability to digest exactly what's happening here is inspiring. And
humbling. :) I'm glad that your explanation is now on-record (for my own
reference, if no one else's).

>> There is one last trick to pull-off, which is to add very similar
>> clean-URL functionality for two other files (in addition to index.php),
>> but I am hoping that I will be able to adapt your working sample myself.
>
> I don't fully follow what you mean there; but once you can isolate your
> requests in a location, then you should be able to set the fallback to
> whatever you want.
>

Oops, I misspoke!

I meant to say that there are certain cases in which I want index.php to
dispatch the request to dedicated CSS and JS controllers. So, rather
than request /css.php?file=home.css in the HTML, for example, I would
request /css2/home.css (the /css2/ is to distinguish "virtual" requests
from "real" requests for un-processed files in the standard /css/
directory). In these cases, index.php looks for /css2/ and dispatches
the request to the CSS controller, which in-turn obtains the requested
file from the URL (everything after /css2/) and returns the output when
the template file exists and is valid.

Maybe it would be helpful for me to demonstrate how I achieve this when
the "site" is not accessed in a subdirectory (with respect to the URL)
and the files actually exist in the virtual host's document root. (This
is counter to the scenario that we've thus far addressed in this
thread.) You actually helped me arrive at this solution a few months
back.

###############################################################

location / {
  try_files $uri $uri/ @virtual;
}

location @virtual {
  include fastcgi_params;
  fastcgi_pass unix:/var/lib/php5-fpm/web1.sock;
  fastcgi_index index.php;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
  fastcgi_intercept_errors on;

  if ($uri ~ '/download/.*/$') {
    rewrite ^/(.*)/$ /$1 permanent;
  }

  if ($uri !~ '(/|\.[a-zA-Z0-9]{1,12}|/download/.*)$') {
    return 301 $uri/$is_args$args;
  }

  if ($uri ~ '(/|/download/.*|/css2/.*|/js2/.*)$') {
    rewrite  ^(.*)$  /index.php?q=$1  last;
  }
}

###############################################################

This is all I'm trying to do. I just want to replicate the above with
the "less-friendly" directory structure that we've been discussing.

Regarding the rewrite statements (in order of appearance):

1.) If the URI is requesting a download and a trailing slash was
included, remove the trailing slash.

2.) If the URI doesn't end with a trailing slash, a file extension, or
contain "/download/...", redirect and add a trailing slash.

3.) If the URI ends with a trailing slash or contains "/download/...",
pass the request to PHP controller. Note that URLs ending in a file
extension with a trailing slash will be passed to the PHP controller,
too.

If there is a better, more concise, or more secure way to accomplish the
above, then I am eager to learn.

> It may get to the point that it is clearer to use a @named location as
> the fallback, and just hardcode SCRIPT_FILENAME in them. You'll see when
> you do it, no doubt.
>

Hmm, I think I see what you're getting at here. But is there no means by
which to achieve the desired configuration with my particular directory
structure? I'm a bit surprised at how difficult it is to decouple the
request location from the filesystem directory from which it is served,
while maintaining this clean-URL setup.

>>> You may want to use something other than $uri in the last argument to
>>> try_files, depending on how you want /stage/my-account/?key=value to
>>> be processed.
>
> The usual caveats apply here regarding url escaping -- you've copied
> an unescaped string directly into query string, so if there were any
> characters like % or + or & involved, they may cause confusion.
>

Understood. The snippet that I pasted above does everything that I
want/need with respect to escaping (or not), so if that can be made to
work with minimal "tweaking", it would be ideal.

> Cheers,
>
>   f
>

Thanks again for your time, dedication, and assistance. I'm very
grateful!

-Ben
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-21 21:45
(Received via mailing list)
On Wed, Nov 20, 2013 at 10:31:07PM -0500, Ben Johnson wrote:
> On 11/20/2013 4:10 AM, Francis Daly wrote:

Hi there,

> > If you do have free choice in the matter, some things work more easily
> > within nginx if you can use "root" and not "alias" -- so if you want
> > files to be accessible below the url /stage/, having them directly in
> > a directory called /stage/ is convenient.

> a.) I prefer to maintain identical directory structures within my
> staging and production environments. This allows my build scripts and
> relative paths to remain identical (provided that I set a "root path"
> variable or similar in each shell script), and it helps me to remember
> to where I must "cd" in order to execute a build script.

That's perfectly sensible.

And the web server configurations for production and staging would
also be very similar, if they were served at similar points in the
hierarchy of the web domain or domains.

But because you serve one at "/" and one at "/stable/", you end up
having different configurations. And because of nginx's implementation
of "alias", and your use of a filesystem hierarchy which means you need
"alias", you end up having very different configurations.

> b.) I really prefer to keep PHP and PHP-template files out of the
> document root (on the filesystem).

That seems mostly unrelated, unless I'm missing something. Put your php
files in "php/" which is parallel to "web/" if you like, nginx won't
care. In fact, if you don't use try_files, nginx won't even look. All
it needs to know is what filename to tell the fastcgi server to read.

> I don't want user-agents to be able
> to request .tpl files directly, for example, by requesting
> /templates/home.tpl.

I confess that I thought that was standard -- the php file itself should
have written in it "read your associated files from this directory
location", which should not be web-accessible (unless you specifically
want it to be).

> course, I build my applications such that every request is routed
> through a single point that has complete authority over how the request
> is handled.

If you really mean "every", then the only nginx config you need is

  location ^~ /myapp/ {
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /var/myapp.php;
    fastcgi_pass unix:php.sock;
  }

Anything else is the web server getting in the way (and implementing
things that it can probably do more efficiently than your application
can).

So the time spent arranging the web server configuration you want can
be counted as time saved not implementing the extra features in your
application.

> In the second case, this seems risky; if I were to put the "stage"
> directory within the production site's document root,

I may have been unclear, or I may be misunderstanding you now.

What I intended to suggest was that: currently all of
your staging web content is visible on the filesystem in
/var/www/example.com/private/stage/web/;

if you either move or copy (or possibly even symlink) it to be visible
in /var/www/example.com/private/stage/web/stage/, then you will be able
to configure nginx using "root" instead of "alias", and your "staging"
configuration can look much more like your "production" one.

      location ^~ /stage/ {
        root /var/www/example.com/private/stage/web/;
        # The files are read from
/var/www/example.com/private/stage/web/stage/
        index index.php index.html index.htm;
        try_files $uri $uri/ /stage/index.php?q=$uri;

        location ~ \.php$ {
          # use a different "root" here if you want; but make sure the
php
          # files can be read from within "stage/" below that root.
          try_files $uri /stage/index.php?q=$uri;
          fastcgi_pass unix:/var/run/php5-fpm.sock;
          fastcgi_param SCRIPT_FILENAME
$document_root$fastcgi_script_name;
          include /etc/nginx/fastcgi_params;
        }
      }


> I meant to say that there are certain cases in which I want index.php to
> dispatch the request to dedicated CSS and JS controllers.

If "dispatch to the controller" means "something within php, and
therefore
nginx doesn't have to know or care about it", then it probably should
Just Work.

> Maybe it would be helpful for me to demonstrate how I achieve this when
> the "site" is not accessed in a subdirectory (with respect to the URL)
> and the files actually exist in the virtual host's document root.

> location / {
>   try_files $uri $uri/ @virtual;
> }

That'll become "/stage/" and "@stagevirtual", I guess. But it'll
probably
want a suitable "root" added -- your "location /" can safely inherit
the server one, while "/stage/" can't.

> location @virtual {

>   fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

@stagevirtual will need to know its version of $document_root. So set
a suitable "root" there too.

>   if ($uri ~ '/download/.*/$') {
>     rewrite ^/(.*)/$ /$1 permanent;

"/stage/" will already be there in both parts of the rewrite, so no
change needed.

>   if ($uri !~ '(/|\.[a-zA-Z0-9]{1,12}|/download/.*)$') {
>     return 301 $uri/$is_args$args;

"/stage/" will already be there in $uri, so no change needed.

>   if ($uri ~ '(/|/download/.*|/css2/.*|/js2/.*)$') {
>     rewrite  ^(.*)$  /index.php?q=$1  last;

That should become "/stage/index.php".

(Note that in this specific rewrite, $1 == $uri, so you could avoid the
match-and-capture.)

> This is all I'm trying to do. I just want to replicate the above with
> the "less-friendly" directory structure that we've been discussing.

I'm not seeing many changes being needed. Provided you avoid
"alias". Which requires a filesystem change. Which hopefully won't break
the non-nginx part of your workflow.

(Actually, within your @stagevirtual, the only place root/alias seems to
matter is in the SCRIPT_FILENAME thing. So if you can build your own
from
the variables you have to hand, you can probably get away with whatever
file structure you like.)

> If there is a better, more concise, or more secure way to accomplish the
> above, then I am eager to learn.

In your staging web directory: ln -s . stage. In your staging php
directory: ln -s . stage. Then set "root" in your "location ^~/stage/"
and in your "location @stagevirtual".

So long as everything that tries to recurse into the directories can
recognise the symlink loop, it should be fine.

Then in production, either "rm stage", or don't create the link in the
first place.

> Hmm, I think I see what you're getting at here. But is there no means by
> which to achieve the desired configuration with my particular directory
> structure? I'm a bit surprised at how difficult it is to decouple the
> request location from the filesystem directory from which it is served,
> while maintaining this clean-URL setup.

If you want nginx, and you want a directory structure that requires
nginx's "alias", then you get to deal with nginx's "alias", which has
some imperfections.

Change one of your wants, and the problem disappears.

All the best,

  f
--
Francis Daly        francis@daoine.org
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-22 03:16
(Received via mailing list)
On Thu, Nov 21, 2013 at 08:45:02PM +0000, Francis Daly wrote:
> On Wed, Nov 20, 2013 at 10:31:07PM -0500, Ben Johnson wrote:

...and one more possibility...

If your application directory structure is such that:

  /var/www/myapp/web/

contains only static files that should be served as-is if requested,
with appropriate "index" files for some directories if wanted; and

  /var/www/myapp/php/

contains "myapp.php", which is the single controlling script that will
always be called if the web request is not for a static file; and

whatever other scripts, templates, and other things that are necessary
for myapp.php to refer to are here, or are anywhere other than in
/var/www/myapp/web/,

and the application should be accessible from the web url "/app1/",

then possibly all the locations you need are

===
    # all requests for this app
    location ^~ /app1/ {
      alias /var/www/myapp/web/;
      error_page 404 403 = @app1;
    }
    # the main controller. Set whatever you want in the include file
    location @app1 {
      fastcgi_pass unix:php.sock;
      fastcgi_param SCRIPT_FILENAME /var/www/myapp/php/myapp.php;
      include fastcgi_params;
    }
===

> >   if ($uri ~ '/download/.*/$') {
> >     rewrite ^/(.*)/$ /$1 permanent;

> >   if ($uri !~ '(/|\.[a-zA-Z0-9]{1,12}|/download/.*)$') {
> >     return 301 $uri/$is_args$args;

> >   if ($uri ~ '(/|/download/.*|/css2/.*|/js2/.*)$') {
> >     rewrite  ^(.*)$  /index.php?q=$1  last;

If you want to keep those "rewrites" within nginx.conf, just use the
first two directly within @app1, and I think the third will not be
needed.

myapp.php should be able to find all it needs in
_SERVER["DOCUMENT_URI"],
so it shouldn't need _REQUEST["q"].

And a staging version would merely be another application -- copy the
two locations, change the prefix and name, change the alias, error_page,
and SCRIPT_FILENAME. All done.

An application at / or at /sub/url/ would be no different. You might be
able to use "root" instead of "alias" in some cases, but I don't think
it matters here.

> > If there is a better, more concise, or more secure way to accomplish the
> > above, then I am eager to learn.

The above is more concise -- partly because it tries to do less. The
previous version allowed you to access the urls /app1/one.php
and /app1/two.php in the client, and the separate files would be
processed. This version would return the php files directly from web/,
or would let php/myapp.php decide what to do if the files aren't in
web/.

That's the key to the smaller config -- nginx only tells the fastcgi
server to process a single file.

  f
--
Francis Daly        francis@daoine.org
79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-23 18:37
(Received via mailing list)
On 11/21/2013 3:45 PM, Francis Daly wrote:
>> a.) I prefer to maintain identical directory structures within my
>
> But because you serve one at "/" and one at "/stable/", you end up
> having different configurations. And because of nginx's implementation
> of "alias", and your use of a filesystem hierarchy which means you need
> "alias", you end up having very different configurations.
>

I see; this seems to be the crux of my struggle. :)

>> /templates/home.tpl.
>
> I confess that I thought that was standard -- the php file itself should
> have written in it "read your associated files from this directory
> location", which should not be web-accessible (unless you specifically
> want it to be).
>

It *should* be standard, but consider frameworks such as WordPress,
Joomla, Drupal, etc. They all require (or at least assume) everything to
be within the virtual host's document root (the "web" directory, per our
discussion here). Not directly relevant; just an observation that we
seem to be "ahead of the pack" in this best-practice.

>   }
>
> Anything else is the web server getting in the way (and implementing
> things that it can probably do more efficiently than your application
> can).
>

You're right; I don't mean *every* request -- I mean only those that are
not for a "real file" and that match the format of a resource that I may
serve via PHP.

> your staging web content is visible on the filesystem in
> /var/www/example.com/private/stage/web/;
>
> if you either move or copy (or possibly even symlink) it to be visible
> in /var/www/example.com/private/stage/web/stage/, then you will be able
> to configure nginx using "root" instead of "alias", and your "staging"
> configuration can look much more like your "production" one.
>

Yes, this was a simple misunderstanding; I thought you meant to move
everything to /var/www/example.com/web/stage/. Thanks for clarifying
this point.

This seems to be "the hot ticket". I have no reason for not doing this.

>           fastcgi_pass unix:/var/run/php5-fpm.sock;
> nginx doesn't have to know or care about it", then it probably should
> That'll become "/stage/" and "@stagevirtual", I guess. But it'll probably
>>   if ($uri ~ '/download/.*/$') {
>>   if ($uri ~ '(/|/download/.*|/css2/.*|/js2/.*)$') {
> I'm not seeing many changes being needed. Provided you avoid
>
> In your staging web directory: ln -s . stage. In your staging php
> directory: ln -s . stage. Then set "root" in your "location ^~/stage/"
> and in your "location @stagevirtual".
>
> So long as everything that tries to recurse into the directories can
> recognise the symlink loop, it should be fine.
>
> Then in production, either "rm stage", or don't create the link in the
> first place.
>

Okay, I'm trying to implement this, but something seems to have gone
terribly awry. I apologize for derailing our progress, but until this is
resolved, I have no way to test your recommendations.

It's bizarre. At some point while meddling with the configuration,
requests for /stage/ began causing the browser to download index.php
(and I can open the file and see the PHP code). And no matter what I
change in the configuration, this behavior persists.

I went so far as to rename the "stage" directory to "staging", and
changed all references in nginx's configuration accordingly. Yet, for
some reason, requests for /stage/ still download the index.php file! I'm
not even sure where this index.php file is coming from, given that nginx
should have absolutely no knowledge of this file's new/current location.

The weirdest part is that the downloaded file is *not* the file that
exists at /index.php (the "main site" index file). I confirmed this by
adding some commented PHP code at the bottom of /index.php, and the
comments do not appear in the downloaded file.

Hell, I even tried deleting the entire "staging" directory and this
still happens! I've restarted nginx, php5-fpm, etc. and nothing changes.
Where is this file coming from???

> Change one of your wants, and the problem disappears.
>
> All the best,
>
>   f
>

Thanks again for seeing me through this, Francis. A bottle of your
favorite spirit is in order once this whole affair is resolved!

-Ben
79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-23 19:12
(Received via mailing list)
On 11/23/2013 12:36 PM, Ben Johnson wrote:
> changed all references in nginx's configuration accordingly. Yet, for
> still happens! I've restarted nginx, php5-fpm, etc. and nothing changes.
> Where is this file coming from???

I removed everything related to this effort (setting-up the staging
site) from nginx's configuration, and deleted all of the staging site
files from the filesystem, yet nginx is still serving an "index.php"
file whenever I request /stage/! How is this possible? This "index.php"
is indeed "mine"; it contains the code from my application. But, as I
said, if I add some random, commented PHP code to the bottom of the
"real index file" at /index.php, it's not present in the downloaded
file. I just don't see where this file could be coming from at this
point.

If I request some other URL, e.g., /stag/, this does not happen; the
problem is specific to the location /stage/.

grep -ir "stage" /etc/nginx does not return any results, either, which
proves that all references to this location have been expunged from the
nginx configuration.

Furthermore, any effort to modify the configuration to use a different
location, such as /staging/ has no effect. It's like the nginx
configuration is stuck in some cached state. And restarting nginx
doesn't fix the problem.

Am I out of my mind?

Thanks for any insight as to what could be happening here...

-Ben
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-23 21:48
(Received via mailing list)
On Sat, Nov 23, 2013 at 01:11:46PM -0500, Ben Johnson wrote:
> On 11/23/2013 12:36 PM, Ben Johnson wrote:

Hi there,

> > It's bizarre. At some point while meddling with the configuration,
> > requests for /stage/ began causing the browser to download index.php
> > (and I can open the file and see the PHP code). And no matter what I
> > change in the configuration, this behavior persists.

I tend to test with "curl", because it avoids any chance of a browser
cache being involved.

Any intermediate proxies or caches might still be in the way, of
course. But the nginx access log and/or debug log should show what nginx
thinks is happening.

> > I went so far as to rename the "stage" directory to "staging", and
> > changed all references in nginx's configuration accordingly. Yet, for
> > some reason, requests for /stage/ still download the index.php file! I'm
> > not even sure where this index.php file is coming from, given that nginx
> > should have absolutely no knowledge of this file's new/current location.

The usual reason for this kind of thing is that the nginx.conf that you
are writing and the nginx.conf that nginx is reading are not the same.

> > The weirdest part is that the downloaded file is *not* the file that
> > exists at /index.php (the "main site" index file). I confirmed this by
> > adding some commented PHP code at the bottom of /index.php, and the
> > comments do not appear in the downloaded file.

The other likely reason is that the web server you are talking to and
the web server you think you are talking to are not the same.

(Either a different machine entirely, or else a different server{}
block in the same config file.)

> > Hell, I even tried deleting the entire "staging" directory and this
> > still happens! I've restarted nginx, php5-fpm, etc. and nothing changes.

Possibly you did a "nginx -s reload", but an error in the conf file
meant that nginx didn't stop using the older file, or something like
that?

> I removed everything related to this effort (setting-up the staging
> site) from nginx's configuration, and deleted all of the staging site
> files from the filesystem, yet nginx is still serving an "index.php"
> file whenever I request /stage/! How is this possible? This "index.php"
> is indeed "mine"; it contains the code from my application. But, as I
> said, if I add some random, commented PHP code to the bottom of the
> "real index file" at /index.php, it's not present in the downloaded
> file. I just don't see where this file could be coming from at this point.

Add something like

  location = /stage/ { return 200 "Just checking...\n"; }

and restart nginx. If you don't see that response for that request,
you're not using that conf file.

If that much does work, then either read the debug log, or trace through
nginx.conf to see what should happen for this request -- server-level
rewrite module directives, plus the one location that the request
"/stage/" should be handled in, are the most likely sources.

> Thanks for any insight as to what could be happening here...

Make sure nginx is really completely stopped.

Then start it again with a known config file.

And look at the responses to a few requests, and see which are not what
you expect.

I'm afraid I'm reduced to generalities with the information available.

Good luck with it,

  f
--
Francis Daly        francis@daoine.org
79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-24 17:15
(Received via mailing list)
On 11/23/2013 3:47 PM, Francis Daly wrote:
> I tend to test with "curl", because it avoids any chance of a browser
> cache being involved.
>

That's very good advice, and yielded a somewhat surprising result when I
added your test string (from your previous reply, down below) to my
configuration:

location = /stage/ { return 200 "Just checking...\n"; }

When I request the /stage/ URL via curl, *from* the web-server on which
this problem is occurring, I see our test string, "Just checking...". I
am using the FQDN when I perform the curl test.

And, when I hit the URL with curl from my local workstation, I see "Just
checking..."

Yet, when I hit the same URL in a web-browser (Chrome in Incognito mode,
Firefox, doesn't seem to matter), from my local workstation, I am
presented with a binary file download for this stale index.php that is
coming from who-knows-where. The MIME-type registers as
application/octet-stream.

I tried hitting the URL in a browser on another machine that is in
another physical location and the result is the same.

When I ping the FQDN from my home workstation, the IP address is as I
expect.

> Any intermediate proxies or caches might still be in the way, of
> course. But the nginx access log and/or debug log should show what nginx
> thinks is happening.
>

The last time a DNS change was made for this domain was several weeks
ago. But because it may be relevant, that DNS change did directly affect
web services (the change was made as part of a server migration -- this
problem is on the new server).

>>> I went so far as to rename the "stage" directory to "staging", and
>>> changed all references in nginx's configuration accordingly. Yet, for
>>> some reason, requests for /stage/ still download the index.php file! I'm
>>> not even sure where this index.php file is coming from, given that nginx
>>> should have absolutely no knowledge of this file's new/current location.
>
> The usual reason for this kind of thing is that the nginx.conf that you
> are writing and the nginx.conf that nginx is reading are not the same.
>

I tried intentionally throwing bad syntax into the nginx config file for
this vhost and restarting nginx, and indeed nginx complains about the
problem and won't restart, which is expected.

This seems to confirm that the configuration file that I am editing is
indeed effective.

>
I would be surprised if I am still hitting the "old" server, away from
which I migrated over three weeks ago. Every machine from which I "ping"
the FQDN returns the expected IP address (including the machine on which
this is happening).

>>> Hell, I even tried deleting the entire "staging" directory and this
>>> still happens! I've restarted nginx, php5-fpm, etc. and nothing changes.
>
> Possibly you did a "nginx -s reload", but an error in the conf file
> meant that nginx didn't stop using the older file, or something like that?
>

I'm using my OS-specific command, "service nginx reload" (which uses
/etc/init.d/nginx) to test configuration changes. Restarting with
"service nginx restart" doesn't seem to make any difference.
Historically, "reload" has always been sufficient to render my changes
effective.

No errors appear in the log, unless I intentionally insert invalid
syntax into the configuration.

>
>   location = /stage/ { return 200 "Just checking...\n"; }
>
> and restart nginx. If you don't see that response for that request,
> you're not using that conf file.
>

When I hit the URL with curl, the request is indeed logged to the
vhost's access log:

"GET /stage/ HTTP/1.1" 200 17 "-" "curl/7.22.0 (i686-pc-linux-gnu)
libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"

When I hit the URL with the browser, the request is logged, too:

"GET /stage/ HTTP/1.1" 200 17 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57
Safari/537.36"

Given that the requests look identical, excepting the user-agent, I'm
left wondering why the browser downloads some stale index.php, yet curl
displays our test message.

Again, I've tried hitting the URL from different computers in different
locations, using different browsers, and the behavior is consistent.
This implies that the problem is with the server, not the user-agents
with which I'm testing.

> If that much does work, then either read the debug log, or trace through
> nginx.conf to see what should happen for this request -- server-level
> rewrite module directives, plus the one location that the request
> "/stage/" should be handled in, are the most likely sources.
>

So, I just went to comment-out the test location and see what curl
returns in that scenario, given that a browser is clearly unreliable,
and now we're back to normal! All I did was comment-out the test line in
the vhost's config file and "service nginx reload"!

>
> Good luck with it,
>
>   f
>

In summary, all I did today was comment-out one line of the config and
reload nginx, and now all is back to normal. Needless to say, I had
tried all of this and much more yesterday, and nothing I tried had any
effect.

I see no explanation for this behavior other than nginx caching its
configuration in some fashion. Maybe after 20-something hours and a
reload, it "let go" of whatever cached version it was using.

If this ever happens again, what is the first avenue to explore? There
has to be a better way to troubleshoot this.

Maybe one of the developers can comment as to how nginx stores its
configuration while running, and how to prevent this issue going forward
(or at least fix it whenever it happens).

Thanks again for your excellent suggestions and sound troubleshooting
logic, Francis.

Now, back to the original problem... :P

-Ben
79b9e05b03252fbca412505e42ce23c8?d=identicon&s=25 Ben Johnson (Guest)
on 2013-11-24 22:32
(Received via mailing list)
On 11/23/2013 12:36 PM, Ben Johnson wrote:
>>           fastcgi_pass unix:/var/run/php5-fpm.sock;
>>           fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
>>           include /etc/nginx/fastcgi_params;
>>         }
>>       }

Francis,

Yes! After making the changes you recommended on the filesystem
(enabling me to ditch "alias"), the staging site is now working
perfectly with a slight variation of the above configuration.

For the sake of academic curiosity, I would like to try some of the
alternate configurations that you cooked-up, but I'll save that for
later. For now, I'm just thrilled to have this working!

Again, I can't thank you enough for your patience, thoroughness, and
generosity with your time.

Best regards, and cheers to a working configuration!

-Ben
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.