I am having problems with proxy being unable to store pages that do not have a file extension (such as a directory or "nice url"). 1. User requests http://domain.com/page/hello/ 2. nginx looks in the root, can not find the page. 3. nginx uses the error page, which then calls a proxy pass inside a location 4. nginx fetches the page. 5. nginx cannot save the results, because it is /page/hello/ nginx does create the /page/hello/ directory inside the proxy_store directory though. #### It could very well be a problem with my configuration. It works for urls with file extensions. location ^~ /fetch/ { internal; include proxy.conf; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:9001/; proxy_store /var/www/data/domain.com$uri; proxy_store_access user:rw group:rw all:r; } location / { root /var/www/data/doman.com/fetch; error_page 404 = /fetch$uri; }
on 27.01.2008 04:28
on 28.01.2008 06:19
Hi Cactus! On Sat, 2008-01-26 at 19:18 -0800, eliott wrote: > I am having problems with proxy being unable to store pages that do > not have a file extension (such as a directory or "nice url"). Have you tried with just proxy_store on; rather than explicitly specifying a path? Unfortunately I don't have a way to test this at the moment, but it's worth a try. Cliff
on 28.01.2008 07:23
On 1/27/08, Cliff Wells <cliff@develix.com> wrote: > Hi Cactus! Hello! :) > Have you tried with just > > proxy_store on; > > rather than explicitly specifying a path? Unfortunately I don't have a > way to test this at the moment, but it's worth a try. I haven't tried that. I will give it a shot tomorrow and see if it changes the behavior at all.
on 28.01.2008 07:29
On Sat, Jan 26, 2008 at 07:18:26PM -0800, eliott wrote: > > proxy_store /var/www/data/domain.com$uri; > proxy_store_access user:rw group:rw all:r; > } > > location / { > root /var/www/data/doman.com/fetch; > error_page 404 = /fetch$uri; > } You may try location / { root /var/www/data/doman.com/fetch; set $fetch /fetch$uri; error_page 404 = $fetch; } location ~ /$ { index index.html; root /var/www/data/doman.com/fetch; set $fetch /fetch${uri}index.html; error_page 403 404 = $fetch; } location ^~ /fetch/ { internal; include proxy.conf; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:9001/; proxy_store /var/www/data/domain.com$fetch; proxy_store_access user:rw group:rw all:r; }
on 28.01.2008 07:37
That might work with some tweaking. My problem is that the upstream cache doesn't have a target for a http://domain.com/foo/index.html request, just the http://domain.com/foo/ (django app with the django url routing) Maybe I can add another rewrite clause in the /fetch/ location, to strip off an ending index.html before the request hits the proxy.
on 28.01.2008 08:06
On Sun, Jan 27, 2008 at 10:31:54PM -0800, eliott wrote: > That might work with some tweaking. > > My problem is that the upstream cache doesn't have a target for a > http://domain.com/foo/index.html request, just the > http://domain.com/foo/ > (django app with the django url routing) Then: location / { root /var/www/data/doman.com/fetch; error_page 404 = /fetch$uri; } location /fetch/ { internal; include proxy.conf; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:9001/; proxy_store /var/www/data/domain.com$uri; proxy_store_access user:rw group:rw all:r; } location ~ /$ { index index.html; root /var/www/data/doman.com/fetch; error_page 403 404 = @fetch; } location @fetch { internal; include proxy.conf; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:9001; proxy_store /var/www/data/domain.com/fetch${uri}index.html; proxy_store_access user:rw group:rw all:r; }
on 28.01.2008 10:42
eliott wrote: > I am having problems with proxy being unable to store pages that do > not have a file extension (such as a directory or "nice url"). > > 1. User requests http://domain.com/page/hello/ > 2. nginx looks in the root, can not find the page. > 3. nginx uses the error page, which then calls a proxy pass inside a location > 4. nginx fetches the page. > 5. nginx cannot save the results, because it is /page/hello/ > nginx does create the /page/hello/ directory inside the proxy_store > directory though. location /{ proxy_pass http://somewhere.com/; proxy_set_header X-Real-IP $remote_addr; proxy_store /home/arcade/www2/$uri#; proxy_set_header Host $host; } Trick is to add some symbol to any path making any url a valid file link. If we do select symbol which would not come in uri at any case we can use as escape symbol. Works for me.
on 28.01.2008 10:44
On Mon, Jan 28, 2008 at 10:55:19AM +0200, Volodymyr Kostyrko wrote: > > nginx does create the /page/hello/ directory inside the proxy_store > link. If we do select symbol which would not come in uri at any case we > can use as escape symbol. > > Works for me. I do no think that http://domain.com/page/hello/%23 is "nice URL".
on 28.01.2008 11:19
On Mon, 2008-01-28 at 12:36 +0300, Igor Sysoev wrote: > > >4. nginx fetches the page. > > > > Trick is to add some symbol to any path making any url a valid file > > link. If we do select symbol which would not come in uri at any case we > > can use as escape symbol. > > > > Works for me. > > I do no think that http://domain.com/page/hello/%23 is "nice URL". But how will the browser ever see the %23 in this case? I assume it doesn't, so does it make any difference how "nice" it is? Regards, Cliff
on 28.01.2008 11:56
Igor Sysoev пишет: >>> 4. nginx fetches the page. >> Trick is to add some symbol to any path making any url a valid file >> link. If we do select symbol which would not come in uri at any case we >> can use as escape symbol. >> >> Works for me. > > I do no think that http://domain.com/page/hello/%23 is "nice URL". This seem to work anyway. The file is stored on the local disk and and in the remote logs i see response 200 turned to 304. The browser is unaffected and keep displaying the original url.
on 29.01.2008 04:15
I actually tested both methods presented above, and got them both to
work.
########
method 1 was copied nearly verbatim from Igor's example.
Thanks Igor!
location / {
root /var/www/data/example.com/fetch;
error_page 404 = /fetch$uri;
}
location ^~ /fetch/ {
internal;
include proxy.conf;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://127.0.0.1:9001;
proxy_store /var/www/data/example.com$uri;
proxy_store_access user:rw group:rw all:r;
}
location ~ /$ {
index index.html;
root /var/www/data/example.com/fetch;
error_page 403 404 = @fetch;
}
location @fetch {
internal;
include proxy.conf;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://127.0.0.1:9001;
proxy_store /var/www/data/example.com/fetch${uri}index.html;
proxy_store_access user:rw group:rw all:r;
}
########
Method 2 required some minor adjustment
Thanks Volodymyr!
location / {
root /var/www/data/example.com/fetch;
if (-f $request_filename/_cache.html) {
rewrite (.*) $1/_cache.html;
}
error_page 403 404 = /fetch$uri;
}
location ^~ /fetch/ {
internal;
include proxy.conf;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://127.0.0.1:9001;
proxy_store /var/www/data/example.com${uri}_cache.html;
proxy_store_access user:rw group:rw all:r;
}
########
I am not sure which method is better. The first method results in a
proper site structure that you could tar up and move to another box,
or serve from another webserver, and it would be functional.
The second method is shorter and simpler, while it is tied more
closely to custom rewrite rules and an odd file extension.
I think for now I want the first method, for the ability to be more
highly portable.
I can rsync the output around if I want to.
Thanks for all the help in this thread.
It was a nice first experience on the mailing list for me.
:)