Forum: NGINX nginx + cookies + firefox = 400 Bad Request

Hi guys, this week is was about to move partially to nginx for all
static content. The full switch was planned in about a month on a new
server. I followed this guide
http://www.dikant.de/2008/07/10/nginx-as-a-reverse... and
it worked instantly. After some minutes I got some t-calls about saying
that the page isnt reachable with a 400 - Bad Request for them. It runs
a vbulletin-forum and a Mantis-Bugtracker, it happens on both.

Some hours later I could nail that problem down to nginx (same with 0.5x
from etch and 0.632 from lenny) and firefox users (versions 2&3). It
works again if the related domain-cookies are deleted. apache2 doesnt
cause any trouble, same for IE/Opera-users.

Tried a quick google search and I found endless posts on a lot of sites
with the same 400 problem and ff after they switched to nginx (even
famous ones like electronicarts). Sadly none if these contain a solution
besides cleaning the cookies but that doesnt seem to help for long
according to that posts.

As I have over 60% FF-users Im a little afraid of the results and
switched back to pure apache for the moment. Is there any known solution
to this problem on serverside? May a selfcompiled-0.7-version help on
this? Couldnt find anything related in the changelog.

Many thanks in advance
Oliver

Hello!

On Sun, Dec 14, 2008 at 01:26:59PM +0100, Oliver Pestring wrote:

> works again if the related domain-cookies are deleted. apache2 doesnt
> to this problem on serverside? May a selfcompiled-0.7-version help on
> this? Couldnt find anything related in the changelog.

You should tune large_client_header_buffers in your nginx config, see
http://wiki.codemongers.com/NginxHttpCoreModule#la...
for details.

The problem AFAIK is vbulletin which sets enormous number of
various cookies.

Maxim Dounin

It might be but I don't have that problem with my moderately busy 
vBulletin
board and 31% of my visitors (including myself) use Firefox. The 
difference
is that I use php-fcgi not Apache. Looking at Google search the problem
appears with other apps but the common denominator in most cases seems 
to be
when nginx is being used as a reverse proxy and only with Firefox. The
problem is more likely with Firefox than with vBulletin. As for number 
of
cookies, my browser has 11 from my vBulletin installation. The average 
user
has one fewer since they don't have an admin control panel cookie. In
contrast, I have 28 cookies from CNN.com (which uses nginx to serve some 
of
the content that I browse), 30 from VerizonWireless (my cell phone 
company),
and 16 from American Express, so 11 (or 10 as a typical user might have)
does not seem "enormous".

Consider a brief test proxying your php requests to php-cgi (I use 
php-fpm
but you can use spawn-fcgi from lighttpd and you probably won't need to
recompile php for this test). There are lots of "recipes" online for 
doing
so. If the problem resolves then you can make a more informed decision. 
It
appears that the problem has existed in at least some versions of nginx
0.7.x - see 
http://www.linuxquestions.org/questions/project.ph...
so a recompile may not help. I don't know if it's been rectified at that
site but I have visited it without issue. They are still using nginx but 
how
it is configured I do not know. If you decide to do this, also consider
running some benchmarks using a simple php script: Apache alone, nginx 
as
reverse proxy to Apache, nginx using fastcgi.

Good luck!

Jim

Hello!

On Sun, Dec 14, 2008 at 09:17:02PM -0500, Jim Ohlstein wrote:

> It might be but I don't have that problem with my moderately busy vBulletin
> board and 31% of my visitors (including myself) use Firefox. The difference
> is that I use php-fcgi not Apache. Looking at Google search the problem
> appears with other apps but the common denominator in most cases seems to be
> when nginx is being used as a reverse proxy and only with Firefox. The
> problem is more likely with Firefox than with vBulletin. As for number of

Other browsers seems to have lower limits on total cookie size, and
hence the problem doesn't manifest itself (instead, some cookies
probably just discarded by other browsers).

> cookies, my browser has 11 from my vBulletin installation. The average user
> has one fewer since they don't have an admin control panel cookie. In
> contrast, I have 28 cookies from CNN.com (which uses nginx to serve some of
> the content that I browse), 30 from VerizonWireless (my cell phone company),
> and 16 from American Express, so 11 (or 10 as a typical user might have)
> does not seem "enormous".

Yep, I was wrong in my assumption (just did a quick-review of
vbulletin code).  It doesn't create many cookies, it just uses
several ones that may grow really big.

Anyway, the problem is that client sent too long Cookie header to
nginx.

Quick fix is to enlarge large_client_header_buffers.  And the next
thing to tune is proxy_buffer_size, which in turn will require
enlargement.

More correct fix would be to limit backend somehow, but it's out
of scope of this list.

> Consider a brief test proxying your php requests to php-cgi (I use php-fpm
> but you can use spawn-fcgi from lighttpd and you probably won't need to
> recompile php for this test). There are lots of "recipes" online for doing
> so. If the problem resolves then you can make a more informed decision. It
> appears that the problem has existed in at least some versions of nginx
> 0.7.x - see http://www.linuxquestions.org/questions/project.ph...
> so a recompile may not help. I don't know if it's been rectified at that

BTW, looks like vbulletin on www.linuxquestions.org was patched to
avoid bbthread_lastview cookie (the one that grows for each forum
tread visited by user).

Maxim Dounin

That would explain why it doesn't happen on my installation as I use
database to track thread/forum read info for users, not cookie. That's a
quick setting change in the vBulletin admin panel to fix though it does 
put
a bit more load on the processor.

Maxim Dounin wrote:
> You should tune large_client_header_buffers in your nginx config, see
> http://wiki.codemongers.com/NginxHttpCoreModule#la...
> for details.
> 
> The problem AFAIK is vbulletin which sets enormous number of
> various cookies.


Many thanks guys! It seems that the issue doesnt occur with the 
0.7.26-version. I compiled it manually yesterday evening and vbulletin 
and Mantis work perfectly with the previous config until now.
Just to be sure Ive now followed your advices and added some larger 
values for large_client_header_buffers and proxy_buffer_size too. Also 
switched from cookie-based activity to db-based for vb.

My testsystem is still on lennys 0.6.32 and here an increased 
large_client_header_buffers was even enough to get it working again. 
Perfect ;)

I'm continuing to also see issues with nginx returning 400s, I posted
another thread about this earlier.  The suggestions here are worth
trying, hopefully they help!

For the large_client_header_buffers setting, what values are 
recommended?

Also, for proxy_buffers, I have the following:
            proxy_buffer_size          4k;
            proxy_buffers              4 32k;
            proxy_busy_buffers_size    64k;
            proxy_temp_file_write_size 64k;

Any suggestions if those should be tweaked in any manner?

Thanks!

On Mon, Dec 15, 2008 at 05:59:07PM -0800, Neil Sheth wrote:

> I'm continuing to also see issues with nginx returning 400s, I posted
> another thread about this earlier.  The suggestions here are worth
> trying, hopefully they help!
> 
> For the large_client_header_buffers setting, what values are recommended?

To see 400' reasons you should set error_log level to info:

error_log  /path/to/log  info;

They may be not associated with bufers at all.

As to large_client_header_buffers, Apache's default one header limit is 
8K:
http://httpd.apache.org/docs/2.0/mod/core.html#lim...

nginx's defaults are

client_header_buffer_size    1k;
large_client_header_buffers  4 4k;
                        # or 4 8k for sparc64/ppc64 (but not amd64)

You will see in log that these buffers are no enough, you may set

client_header_buffer_size    1k;
large_client_header_buffers  8 8k;

Setting the error_log to info, I'm seeing that a lot of the 400
requests have a corresponding entry in the error log saying "client
closed prematurely connection while reading client request line" - any
suggestions from that?

Thanks!

Tried increasing the large_client_header_buffers setting as suggested,
but it had no effect on my 400s.  Any other suggestions?  A bit at a
loss at this point, so open to any suggestions.  Thanks!

On Fri, Dec 19, 2008 at 12:36:55AM -0800, Neil Sheth wrote:

> Tried increasing the large_client_header_buffers setting as suggested,
> but it had no effect on my 400s.  Any other suggestions?  A bit at a
> loss at this point, so open to any suggestions.  Thanks!

If causes of these 400s is "client closed prematurely connection
while reading client request line", then you can not do anything
on nginx side. These may be user request cancels when he clicks on
a link on a page while browser is still loading images for the page.
These may be network problems.