Forum: NGINX Set-Cookie is missing via proxy

Hi,


I am using nginx as a reverse proxy for apache.
I have the following symptom: Sessions of users get mixed up.
The original html from the apache looks like this via curl:

[...]
<a href="/ueber-uns/?acmeID=e58927c4c728307aa0596f23a87c8136" 
class="to_page
right_btn"><img src="/layout/cyt/img/blind.gif" alt="" /></a></div>
<div class="teaser teaser4a">
[...]

If I get this page through nginx, it looks like this:
[...]
<a href="/ueber-uns/" class="to_page right_btn"><img
src="/layout/cyt/img/blind.gif" alt="" /></a></div>
[...]

So obviously the query string containing the identifing ID is missing. I
guess
this is where sessions get mixed up. Now I had
a look a the headers, and there I got via apache:
curl --head http://www.foobar-shop.de/
HTTP/1.1 200 OK
Date: Wed, 05 Dec 2012 10:19:48 GMT
Server: Apache
X-Powered-By: PHP/5.2.17-0.dotdeb.0
Set-Cookie: acmeID=48dceed9217eea2b691f75e26276bdb8; expires=Fri,
15-Mar-2013
10:19:48 GMT; path=/; domain=.foobar-shop.de
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Pragma: no-cache
Content-Type: text/html;charset=UTF-8

and via nginx:
HTTP/1.1 200 OK
Server: nginx/1.2.5
Date: Wed, 05 Dec 2012 10:45:54 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 945
Connection: keep-alive
Last-Modified: Fri, 13 Apr 2012 02:58:01 GMT
ETag: "132b39c-3b1-4bd86a3f74040"
Accept-Ranges: bytes
Vary: Accept-Encoding

So the "Set-Cookie" header is missing via nginx. I guess this is the
reason why the query
string is also missing in the html. ( Odly though, if I browse the site 
with
firefox, I still get the cookie ...? )
I tried setting
proxy_pass_header Set-Cookie;
but with no success.

Now I am a bit clueless.

My config:

/etc/nginx/nginx.conf
user www-data;
worker_processes 16;
pid /var/run/nginx.pid;

events {
         worker_connections 2000;
         # multi_accept on;
}

http {

         ##
         # Basic Settings
         ##

         sendfile on;
         tcp_nopush on;
         tcp_nodelay on;
         keepalive_timeout 65;
         types_hash_max_size 2048;
         # server_tokens off;

         # server_names_hash_bucket_size 64;
         # server_name_in_redirect off;

         include /etc/nginx/mime.types;
         default_type application/octet-stream;

         ##
         # Logging Settings
         ##

         access_log /var/log/nginx/access.log;
         error_log /var/log/nginx/error.log;

         ##
         # Gzip Settings
         ##

         gzip on;
         gzip_disable "msie6";
        # gzip_vary on;
         # gzip_proxied any;
         # gzip_comp_level 6;
         # gzip_buffers 16 8k;
         # gzip_http_version 1.1;
         # gzip_types text/plain text/css application/json
         # application/x-javascript text/xml application/xml
application/xml+rss
         # text/javascript;

         # Because we have a lot of server_names, we need to increase
         # server_names_hash_bucket_size
         # (http://nginx.org/en/docs/http/server_names.html)
         server_names_hash_max_size 6000;
         server_names_hash_bucket_size 512;
                                                            # raise 
default
                                                            # values for 
php
         client_max_body_size 20M;
         client_body_buffer_size 128k;

         ##
         # Virtual Host Configs
         ##
         include /var/www3/acme_cache/load_balancer/upstream.conf;
         include /etc/nginx/conf.d/*.conf;
         include /etc/nginx/sites-enabled/*;
         include /etc/nginx/proxy_params;

         index index.html index.htm ;

         ##
         # Proxy Settings
         ##

         # include hostname in request to backend
         proxy_set_header Host $host;

         # only honor internal Caching policies
         proxy_ignore_headers X-Accel-Expires Expires Cache-Control;

         # let cookies from the backend pass
         proxy_pass_header Set-Cookie;

}


/etc/nginx/proxy_params
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

/etc/nginx/conf.d/proxy_paths.conf
proxy_temp_path   /var/lib/nginx/proxy/tmp;
proxy_cache_path  /var/lib/nginx/proxy/cache levels=2:2:2
keys_zone=acme-cache:800m max_size=55000m inactive=10m;

/etc/nginx/sites-enabled/foobar-shop.de_p80
server
{
         server_name www.foobar-shop.de foobar-shop.de foobar-beta.de;
         listen 80;
         access_log /var/log/www/asdf/foobar/log/access.log;
         error_log /var/log/nginx/vhost_error.log;
         proxy_cache acme-cache;
         proxy_cache_key "$scheme$host$proxy_host$uri$is_args$args";
         proxy_cache_valid       200 302 60m;
         proxy_cache_valid       404             10m;

         location ~* \.(jpg|gif|png|css|js)
         {
                 try_files $uri @proxy;
         }

         location @proxy
         {
                 proxy_pass http://backend-all-apaches;
         }

         location /
         {
                 proxy_pass http://backend-all-apaches;
         }

}

Any Ideas?


Isaac

Hello!

On Wed, Dec 05, 2012 at 12:00:51PM +0100, Isaac Hailperin wrote:

> If I get this page through nginx, it looks like this:
> HTTP/1.1 200 OK
> Content-Type: text/html;charset=UTF-8
> Accept-Ranges: bytes
> Vary: Accept-Encoding
>
> So the "Set-Cookie" header is missing via nginx. I guess this is the
> reason why the query
> string is also missing in the html. ( Odly though, if I browse the site with
> firefox, I still get the cookie ...? )

I would suggest it's backend code/configs which needs
investigation.  It looks like the backend returns static file
(note Content-Length, ETag and no X-Powered-By) to nginx for some
reason.

[...]

--
Maxim Dounin
http://nginx.com/support.html

On 12/05/2012 01:23 PM, Maxim Dounin wrote:

> I would suggest it's backend code/configs which needs
> investigation.  It looks like the backend returns static file
> (note Content-Length, ETag and no X-Powered-By) to nginx for some
> reason.
Am I understanding you correctly: you suggest that for some reason the
backend is delivering a page without the query string, so nginx has no
chance to deliver it?

Isaac

Also, at least from the config, the html should not be cached, right?
I will also check the communication with the backend(there currently is
a different problem which prevents this check) to see which headers are
passed between nginx and the backend, and compare them with the headers
sent between the client and nginx.

Anything else I could do to investigate?

Isaac

Hello!

On Wed, Dec 05, 2012 at 09:12:09PM +0100, Isaac Hailperin wrote:

> Also, at least from the config, the html should not be cached, right?

I don't see anything in the config you provided which will prevent
caching.

[...]

> >>I would suggest it's backend code/configs which needs
> >>investigation.  It looks like the backend returns static file
> >>(note Content-Length, ETag and no X-Powered-By) to nginx for some
> >>reason.
> >Am I understanding you correctly: you suggest that for some reason the
> >backend is delivering a page without the query string, so nginx has no
> >chance to deliver it?

Yes.

--
Maxim Dounin
http://nginx.com/support.html

On 06.12.2012 14:37, Maxim Dounin wrote:

>> Also, at least from the config, the html should not be cached,
>> right?
>
> I don't see anything in the config you provided which will prevent
> caching.

location ~* \.(jpg|gif|png|css|js)
         {
                 try_files $uri @proxy;
         }

         location @proxy
         {
                 proxy_pass http://backend-all-apaches;
         }

         location /
         {
                 proxy_pass http://backend-all-apaches;
         }

As far as my understanding goes, this will only cache
jpg|gif|png|css|js files, and send the rest directly to the backend.
Please correct me if I am wrong.

Isaac

Hello!

On Thu, Dec 06, 2012 at 04:16:50PM +0100, Isaac Hailperin wrote:

>                 try_files $uri @proxy;
>         }
>
> As far as my understanding goes, this will only cache
> jpg|gif|png|css|js files, and send the rest directly to the backend.
> Please correct me if I am wrong.

You are wrong.  It tries to lookup jpg/gif/png/css/js files
directly on the file system as static files, but it's

1) Not cache.  Cache is activated with proxy_cache directive, see
http://nginx.org/r/proxy_cache.

2) Not related to html files you talked about.

3) Not related to the response in question as headers clearly
indicate it was originally returned by Apache (note ETag in Apache
format).

Whether or not the response in question was cached can't be
concluded from the information provided.  Most likely it was as
config says to cache all 200 responses (without cookies) for 60
minutes.

--
Maxim Dounin
http://nginx.com/support.html

On 12/06/2012 05:02 PM, Maxim Dounin wrote:
>>          location /
>
> concluded from the information provided.  Most likely it was as
> config says to cache all 200 responses (without cookies) for 60
> minutes.
Thank you Maxim, that was very helpfull! So indeed, my config was wrong,
so in debugging the original problem, I made wrong assumptions.

Now for the original problem:
The Set-Cookie was not missing, but it turned out that having
proxy_set_header Host $host;
twice as I did accidentally, "confused" the backend, which is why it did
not deliver the Set-Cookie header. Removing one instance solved that
problem. But that did not solve the session mixing. The session mixing
was due to
proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
I set this earlier, because not every page set those headers correctly,
resulting in suboptimal cacheing. But it turns out that its really
needed for other pages, so I removed that line again. This solved the
session mixing problem.

Isaac