Set-Cookie is missing via proxy

Hi,

I am using nginx as a reverse proxy for apache.
I have the following symptom: Sessions of users get mixed up.
The original html from the apache looks like this via curl:

[…]

[...]

If I get this page through nginx, it looks like this:
[…]


[…]

So obviously the query string containing the identifing ID is missing. I
guess
this is where sessions get mixed up. Now I had
a look a the headers, and there I got via apache:
curl --head http://www.foobar-shop.de/
HTTP/1.1 200 OK
Date: Wed, 05 Dec 2012 10:19:48 GMT
Server: Apache
X-Powered-By: PHP/5.2.17-0.dotdeb.0
Set-Cookie: acmeID=48dceed9217eea2b691f75e26276bdb8; expires=Fri,
15-Mar-2013
10:19:48 GMT; path=/; domain=.foobar-shop.de
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Pragma: no-cache
Content-Type: text/html;charset=UTF-8

and via nginx:
HTTP/1.1 200 OK
Server: nginx/1.2.5
Date: Wed, 05 Dec 2012 10:45:54 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 945
Connection: keep-alive
Last-Modified: Fri, 13 Apr 2012 02:58:01 GMT
ETag: “132b39c-3b1-4bd86a3f74040”
Accept-Ranges: bytes
Vary: Accept-Encoding

So the “Set-Cookie” header is missing via nginx. I guess this is the
reason why the query
string is also missing in the html. ( Odly though, if I browse the site
with
firefox, I still get the cookie …? )
I tried setting
proxy_pass_header Set-Cookie;
but with no success.

Now I am a bit clueless.

My config:

/etc/nginx/nginx.conf
user www-data;
worker_processes 16;
pid /var/run/nginx.pid;

events {
worker_connections 2000;
# multi_accept on;
}

http {

     ##
     # Basic Settings
     ##

     sendfile on;
     tcp_nopush on;
     tcp_nodelay on;
     keepalive_timeout 65;
     types_hash_max_size 2048;
     # server_tokens off;

     # server_names_hash_bucket_size 64;
     # server_name_in_redirect off;

     include /etc/nginx/mime.types;
     default_type application/octet-stream;

     ##
     # Logging Settings
     ##

     access_log /var/log/nginx/access.log;
     error_log /var/log/nginx/error.log;

     ##
     # Gzip Settings
     ##

     gzip on;
     gzip_disable "msie6";
    # gzip_vary on;
     # gzip_proxied any;
     # gzip_comp_level 6;
     # gzip_buffers 16 8k;
     # gzip_http_version 1.1;
     # gzip_types text/plain text/css application/json
     # application/x-javascript text/xml application/xml

application/xml+rss
# text/javascript;

     # Because we have a lot of server_names, we need to increase
     # server_names_hash_bucket_size
     # (http://nginx.org/en/docs/http/server_names.html)
     server_names_hash_max_size 6000;
     server_names_hash_bucket_size 512;
                                                        # raise 

default
# values for
php
client_max_body_size 20M;
client_body_buffer_size 128k;

     ##
     # Virtual Host Configs
     ##
     include /var/www3/acme_cache/load_balancer/upstream.conf;
     include /etc/nginx/conf.d/*.conf;
     include /etc/nginx/sites-enabled/*;
     include /etc/nginx/proxy_params;

     index index.html index.htm ;

     ##
     # Proxy Settings
     ##

     # include hostname in request to backend
     proxy_set_header Host $host;

     # only honor internal Caching policies
     proxy_ignore_headers X-Accel-Expires Expires Cache-Control;

     # let cookies from the backend pass
     proxy_pass_header Set-Cookie;

}

/etc/nginx/proxy_params
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

/etc/nginx/conf.d/proxy_paths.conf
proxy_temp_path /var/lib/nginx/proxy/tmp;
proxy_cache_path /var/lib/nginx/proxy/cache levels=2:2:2
keys_zone=acme-cache:800m max_size=55000m inactive=10m;

/etc/nginx/sites-enabled/foobar-shop.de_p80
server
{
server_name www.foobar-shop.de foobar-shop.de foobar-beta.de;
listen 80;
access_log /var/log/www/asdf/foobar/log/access.log;
error_log /var/log/nginx/vhost_error.log;
proxy_cache acme-cache;
proxy_cache_key “$scheme$host$proxy_host$uri$is_args$args”;
proxy_cache_valid 200 302 60m;
proxy_cache_valid 404 10m;

     location ~* \.(jpg|gif|png|css|js)
     {
             try_files $uri @proxy;
     }

     location @proxy
     {
             proxy_pass http://backend-all-apaches;
     }

     location /
     {
             proxy_pass http://backend-all-apaches;
     }

}

Any Ideas?

Isaac

Hello!

On Wed, Dec 05, 2012 at 12:00:51PM +0100, Isaac H. wrote:

If I get this page through nginx, it looks like this:
HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Accept-Ranges: bytes
Vary: Accept-Encoding

So the “Set-Cookie” header is missing via nginx. I guess this is the
reason why the query
string is also missing in the html. ( Odly though, if I browse the site with
firefox, I still get the cookie …? )

I would suggest it’s backend code/configs which needs
investigation. It looks like the backend returns static file
(note Content-Length, ETag and no X-Powered-By) to nginx for some
reason.

[…]


Maxim D.

On 12/05/2012 01:23 PM, Maxim D. wrote:

I would suggest it’s backend code/configs which needs
investigation. It looks like the backend returns static file
(note Content-Length, ETag and no X-Powered-By) to nginx for some
reason.
Am I understanding you correctly: you suggest that for some reason the
backend is delivering a page without the query string, so nginx has no
chance to deliver it?

Isaac

Also, at least from the config, the html should not be cached, right?
I will also check the communication with the backend(there currently is
a different problem which prevents this check) to see which headers are
passed between nginx and the backend, and compare them with the headers
sent between the client and nginx.

Anything else I could do to investigate?

Isaac

On 06.12.2012 14:37, Maxim D. wrote:

Also, at least from the config, the html should not be cached,
right?

I don’t see anything in the config you provided which will prevent
caching.

location ~* .(jpg|gif|png|css|js)
{
try_files $uri @proxy;
}

     location @proxy
     {
             proxy_pass http://backend-all-apaches;
     }

     location /
     {
             proxy_pass http://backend-all-apaches;
     }

As far as my understanding goes, this will only cache
jpg|gif|png|css|js files, and send the rest directly to the backend.
Please correct me if I am wrong.

Isaac

Hello!

On Wed, Dec 05, 2012 at 09:12:09PM +0100, Isaac H. wrote:

Also, at least from the config, the html should not be cached, right?

I don’t see anything in the config you provided which will prevent
caching.

[…]

I would suggest it’s backend code/configs which needs
investigation. It looks like the backend returns static file
(note Content-Length, ETag and no X-Powered-By) to nginx for some
reason.
Am I understanding you correctly: you suggest that for some reason the
backend is delivering a page without the query string, so nginx has no
chance to deliver it?

Yes.


Maxim D.

Hello!

On Thu, Dec 06, 2012 at 04:16:50PM +0100, Isaac H. wrote:

            try_files $uri @proxy;
    }

As far as my understanding goes, this will only cache
jpg|gif|png|css|js files, and send the rest directly to the backend.
Please correct me if I am wrong.

You are wrong. It tries to lookup jpg/gif/png/css/js files
directly on the file system as static files, but it’s

  1. Not cache. Cache is activated with proxy_cache directive, see
    Module ngx_http_proxy_module.

  2. Not related to html files you talked about.

  3. Not related to the response in question as headers clearly
    indicate it was originally returned by Apache (note ETag in Apache
    format).

Whether or not the response in question was cached can’t be
concluded from the information provided. Most likely it was as
config says to cache all 200 responses (without cookies) for 60
minutes.


Maxim D.

On 12/06/2012 05:02 PM, Maxim D. wrote:

     location /

concluded from the information provided. Most likely it was as
config says to cache all 200 responses (without cookies) for 60
minutes.
Thank you Maxim, that was very helpfull! So indeed, my config was wrong,
so in debugging the original problem, I made wrong assumptions.

Now for the original problem:
The Set-Cookie was not missing, but it turned out that having
proxy_set_header Host $host;
twice as I did accidentally, “confused” the backend, which is why it did
not deliver the Set-Cookie header. Removing one instance solved that
problem. But that did not solve the session mixing. The session mixing
was due to
proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
I set this earlier, because not every page set those headers correctly,
resulting in suboptimal cacheing. But it turns out that its really
needed for other pages, so I removed that line again. This solved the
session mixing problem.

Isaac