# Upstream sent too big header while reading response header from upstream

**URL:** <https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974>\
**Category:** NGINX\
**Created:** [February 6, 2014, 12:49am UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974 "2014-02-06T00:49:52Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![musicdenotation](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea666f/32.png) [@musicdenotation](https://www.ruby-forum.com/u/musicdenotation)\
**Post date:** [February 6, 2014, 12:49am UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/1 "2014-02-06T00:49:52Z")

</div>

After I added some CORS headers to my API, one of the users of my  
nginx-based system complained about occasional errors with:

upstream sent too big header while reading response header from upstream

He also reported to have worked around the issue using:

proxy\_buffers 8 512k;  
proxy\_buffer\_size 2024k;  
proxy\_busy\_buffers\_size 2024k;  
proxy\_read\_timeout 3000;

However unfortunately I was unable to reproduce this problem myself. I  
also had a hard time figuring out what the exact problem is.

Some questions:

- What exactly does this error mean? Does it mean that response  
contained too many headers? How many is too many?
- Is it wise to increase the buffer sizes as the user reported? What  
would be sensible defaults?

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 6, 2014, 1:19pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/2 "2014-02-06T13:19:23Z")

</div>

Hello!

On Wed, Feb 05, 2014 at 03:48:50PM -0800, Jeroen O. wrote:

> proxy\_read\_timeout 3000;
> 
> However unfortunately I was unable to reproduce this problem myself. I  
> also had a hard time figuring out what the exact problem is.
> 
> Some questions:
> 
> - What exactly does this error mean? Does it mean that response  
> contained too many headers? How many is too many?

Response headers should fit into proxy\_buffer\_size, see  
[Module ngx\_http\_proxy\_module](http://nginx.org/r/proxy_buffer_size). If they don’t, the error  
is reported.

> - Is it wise to increase the buffer sizes as the user reported? What  
> would be sensible defaults?

Certainly no. In most cases defaults used (4k on most platforms)  
are appropriate. If big cookies are expected to be returned by a  
proxied server, something like 32k or 64k will be good enough. If  
larger values are needed, it indicate backend problem.

–  
Maxim D.

> **[nginx](http://nginx.org/)**

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 6, 2014, 6:12pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/3 "2014-02-06T18:12:41Z")

</div>

On Thu, Feb 6, 2014 at 4:18 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote

> Response headers should fit into proxy\_buffer\_size, see  
> [Module ngx\_http\_proxy\_module](http://nginx.org/r/proxy_buffer_size). If they don’t, the error  
> is reported.

In which the “size” refers to the number of characters that appear up  
till the blank line that separates the headers from the body in the  
response? It looks like it would be around 4k. So perhaps I’ll  
increase proxy\_buffer\_size to 8k.

Is it also necessary to increase modify proxy\_busy\_buffers\_size and  
proxy\_buffers to deal with responses with many headers?

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 6, 2014, 6:19pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/4 "2014-02-06T18:19:07Z")

</div>

Hello!

On Thu, Feb 06, 2014 at 09:11:31AM -0800, Jeroen O. wrote:

> increase proxy\_buffer\_size to 8k.  
> Yes.

> Is it also necessary to increase modify proxy\_busy\_buffers\_size and  
> proxy\_buffers to deal with responses with many headers?

No.

–  
Maxim D.

> **[nginx](http://nginx.org/)**

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 10, 2014, 11:55am UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/5 "2014-02-10T11:55:51Z")

</div>

Hi Maxim & Jeroen,

I’m the user Jeroen mentioned. I’m sorry for only being to produce  
sporadic  
errors earlier, I now made a test case which reliably produces  
the error, both on our server and Jeroen’s server (so it’s hopefully not  
just my amateur status with nginx).

Of course, the ridiculously large proxy settings were only chosen in  
desperation, but I can now report that no increase of proxy buffer  
settings solves the problem at all (or shifts the treshold at which the  
error messages occur). So I think it’s safe to say the error message  
upstream sent too big header while reading response header from upstream  
is misleading (unless the CORS headers that Jeroen added are somehow  
inflated through the POST request, wouldn’t know why that would be).

I can also now rule out that’s its due to the headers sent or received.

I receive only the following headers when the 502s occur

HTTP/1.1 100 Continue

HTTP/1.1 502 Bad Gateway  
Server: nginx/1.4.4  
Date: Mon, 10 Feb 2014 10:46:57 GMT  
Content-Type: text/html  
Content-Length: 172  
Connection: keep-alive

And I sent only these

POST /ocpu/library/base/R/identity HTTP/1.1  
Host: ourhost.xx  
Accept: _/_  
Content-Length: 3545  
Content-Type: application/x-www-form-urlencoded  
Expect: 100-continue

I do send a large request, 2241 characters (inlined JSON-ified data),  
but  
that is not near the upper limit of POST request as I know them.

Here’s the issue on the openCPU github, where I uploaded my test cases  
(sorry for it being a bit primitively done, I’m in a bit of a tight  
spot  
time-wise here and just an amateur):

> <https://github.com/opencpu/opencpu/issues/76>
>
> See comment at https://public.opencpu.org/posts/opencpu-release-1.2/

Posted at Nginx Forum:

> **[Re: upstream sent too big header while reading response header from upstream](https://forum.nginx.org/read.php?2,247230,247327#msg-247327)**
>
> After I added some CORS headers to my API, one of the users of my nginx-based system complained about occasional errors with: upstream sent too big header while reading response header from upstream He also reported to have worked around the issue...

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 10, 2014, 1:51pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/6 "2014-02-10T13:51:18Z")

</div>

Hi,

after some further testing I discovered that I had the order in which  
various nginx config files are called wrong. Because location {} isn’t  
merged, but overridden, my directives never ‘took’.

Setting proxy\_buffer\_size 8k; kept the errors from occurring.

As I wrote on Github [proxy\_buffer\_size · Issue #76 · opencpu/opencpu · GitHub](https://github.com/jeroenooms/opencpu/issues/76) ,  
it  
still seems like the error message was misleading here, because the  
headers  
sent were identical (except for the exact Content-Length) and the  
headers  
received were pretty much the same as well. There are no cookies at all  
involved here.

Thanks for the advice and nginx!

Posted at Nginx Forum:

> **[Re: upstream sent too big header while reading response header from upstream](https://forum.nginx.org/read.php?2,247230,247334#msg-247334)**
>
> After I added some CORS headers to my API, one of the users of my nginx-based system complained about occasional errors with: upstream sent too big header while reading response header from upstream He also reported to have worked around the issue...

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 10, 2014, 12:14pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/7 "2014-02-10T12:14:01Z")

</div>

Hello,  
try this

> <https://stackoverflow.com/questions/13894386/upstream-too-big-nginx-codeigniter>

Regards,  
Basti

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 10, 2014, 1:57pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/8 "2014-02-10T13:57:10Z")

</div>

Hi Basti,

thanks, I found the SO post myself. I had not set up the directives  
properly, so thought the fix didn’t work. It does now. I also think they  
described a different problem, as in my case no cookies were sent,  
headers  
were fairly small and two requests with pretty much identical headers  
sent/received had different results (only 502ed).

I’m very interested in learning what exactly may have caused the message  
in  
my case, so as to know the boundaries in which my requests will work. I  
now  
set the buffer\_size to 8K and I don’t see failures with Content-Lengths  
that  
go far beyond that (though maybe compression has to be considered?).

Best regards,

Ruben

Posted at Nginx Forum:

> **[Re: upstream sent too big header while reading response header from upstream](https://forum.nginx.org/read.php?2,247230,247335#msg-247335)**
>
> After I added some CORS headers to my API, one of the users of my nginx-based system complained about occasional errors with: upstream sent too big header while reading response header from upstream He also reported to have worked around the issue...

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 10, 2014, 3:00pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/9 "2014-02-10T15:00:28Z")

</div>

Yes, that’s quite probably it!  
Then I guess it’s on Jerome to weigh in, I don’t know the exact  
reasoning  
for doing so, I would have thought it would be more appropriate to hash  
the  
request body in the cache key, but maybe that’s not possibly using  
nginx?  
OpenCPU allows for request bodies up to 500M by default, what kind of  
settings would be necessary to make that (or something more reasonable  
like  
50M) play with the buffer? I’m guessing the buffer doesn’t have to be as  
large as the maximal request, right?

Posted at Nginx Forum:

> **[Re: upstream sent too big header while reading response header from upstream](https://forum.nginx.org/read.php?2,247230,247352#msg-247352)**
>
> After I added some CORS headers to my API, one of the users of my nginx-based system complained about occasional errors with: upstream sent too big header while reading response header from upstream He also reported to have worked around the issue...

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 10, 2014, 6:46pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/10 "2014-02-10T18:46:41Z")

</div>

On Mon, Feb 10, 2014 at 5:15 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> it is likely the cause, as the config includes the following lines:
> 
> ```
> proxy_cache_methods POST;
> proxy_cache_key "$request_method$request_uri$request_body";
> 
> ```

Yikes I was not aware that the cache key gets stored into the buffers  
as well. Is this mentioned in the manual anywhere?

So we need to set proxy\_buffer\_size to a value greater than the sum of  
client\_body\_buffer\_size + header size? Or alternatively, is there a  
way that to use a fixed length hash of the request body in the  
proxy\_cache\_key?

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 10, 2014, 2:15pm UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/11 "2014-02-10T14:15:55Z")

</div>

Hello!

On Mon, Feb 10, 2014 at 07:56:22AM -0500, rubenarslan wrote:

> set the buffer\_size to 8K and I don’t see failures with Content-Lengths that  
> go far beyond that (though maybe compression has to be considered?).

Another possible cause may be use of $request\_body in  
proxy\_cache\_key. Cache header, including cache key, is placed  
into proxy buffer if caching is enabled, and effectively reduces  
proxy\_buffer\_size available to read response headers.

Assuming you are using configs like this one:

[https://github.com/jeroenooms/opencpu-deb/blob/master/opencpu-cache/nginx/opencpu-ocpu.conf](https://github.com/jeroenooms/opencpu-deb/blob/master/opencpu-cache/nginx/opencpu-ocpu.conf)

it is likely the cause, as the config includes the following lines:

```
proxy_cache_methods POST;
proxy_cache_key "$request_method$request_uri$request_body";

```

–  
Maxim D.

> **[nginx](http://nginx.org/)**

---

<div class="post-metadata">

**Author:** ![Jeroen\_O](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/7c8e57/32.png) [@Jeroen\_O](https://www.ruby-forum.com/u/Jeroen_O)\
**Post date:** [February 11, 2014, 11:55am UTC](https://www.ruby-forum.com/t/upstream-sent-too-big-header-while-reading-response-header-from-upstream/238974/12 "2014-02-11T11:55:23Z")

</div>

Hello!

On Mon, Feb 10, 2014 at 09:45:50AM -0800, Jeroen O. wrote:

> On Mon, Feb 10, 2014 at 5:15 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > it is likely the cause, as the config includes the following lines:
> > 
> > ```
> > proxy_cache_methods POST;
> > proxy_cache_key "$request_method$request_uri$request_body";
> > 
> > ```
> 
> Yikes I was not aware that the cache key gets stored into the buffers  
> as well. Is this mentioned in the manual anywhere?

Likely no. It’s mostly proxy cache implementation detail - it needs a  
way to write a cache header (which includes key) to a cache file,  
and places it to the buffer just before response headers from  
an upstream.

> So we need to set proxy\_buffer\_size to a value greater than the sum of  
> client\_body\_buffer\_size + header size?

If you use $request\_body in the cache key - yes. And don’t forget  
to add other variables in proxy\_cache\_key.

> Or alternatively, is there a  
> way that to use a fixed length hash of the request body in the  
> proxy\_cache\_key?

As of now hashes may be calculated using, e.g., the embedded perl  
or 3rd party modules.

–  
Maxim D.

> **[nginx](http://nginx.org/)**