GZIP problem with backend hosts

Afternoon all.

I first of all thought I had a problem with our backend hosts not
sending GZIP content, but it appears NGINX is at fault.

We use nginx infront of a lot of Amazon instances. The backend
instances, prepare the content in GZIP’d form.

However nginx, is taking this content, and then decompressing it, and
giving out the decompressed version back to the originating client!!

I do not want nginx to do the compressing; the load can be better
handled with the backend instances. I just want nginx to proxy what its
given and not do too much to the stream coming back.

Advice please?

I’ve tried some of the steps from the wiki along with some info i found
online but unable to get this working here’s some of the errors i get
when
i launch nginx

I’ve tried some of the suggestions but still getting
the same error

here’s snip of my config

location /videos {
accesskey on;
accesskey_hashmethod md5;
accesskey_arg “key”;
accesskey_signature “myPass$remote_addr”;

[root@static /usr/local/chtest]# ./sbin/nginx -t -c conf/nginx.conf
2008/08/26 14:58:28 [emerg] 25808#0: unknown directive “accesskey” in
conf/nginx.conf:53
2008/08/26 14:58:28 [emerg] 25808#0: the configuration file
conf/nginx.conf
test failed

no one got anything input here?

seems to be a fairly big failing in NGINX here. Why would it spend
extra time and resources decompressing outgoing content

On Fri, Aug 29, 2008 at 09:41:42AM +0100, Alan W. wrote:

no one got anything input here?

seems to be a fairly big failing in NGINX here. Why would it spend
extra time and resources decompressing outgoing content

nginx can not decompress proxied response.
In your case probably backend sends uncompressed response.

Igor S. wrote:

On Fri, Aug 29, 2008 at 09:41:42AM +0100, Alan W. wrote:

no one got anything input here?

seems to be a fairly big failing in NGINX here. Why would it spend
extra time and resources decompressing outgoing content

nginx can not decompress proxied response.
In your case probably backend sends uncompressed response.

okay so lets ask the question another way.

Scenario#1:
Hit Backend#1 directly; get GZIP’d content

Scenario#2:
Hit nginx which then proxies to Backend#1; get raw content

Soooo … either … nginx is decompressing, which you claim it is not,
which leaves that nginx isn’t passing on all the HTTP headers to the
backend, including the all important:

“Accept-Encoding: gzip,deflate”

?

If you did not set
proxy_set_header Accept-Encoding “”;

then nginx will pass it backend.

i do not understand this. Are you suggesting i need to set this in
order for nginx to send that header to the back end?

What about the other headers that get sent with the request (cookie etc)
we don’t manually set them do we? They simply get passed to the
backend.

It always bothers me that nginx doesn’t pass “Host:” one and i have to
set this manually, makes me wonder what other headers nginx is silently
rejecting and not passing on to the backend.

can you tell me what the definitive answer is here?

------ snippet -------
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header x-forwarded-for $remote_addr;
proxy_pass http://backends;
}

So if a client sends in the header that he can accept GZIP encoding,
nginx is removing this and not passing it through.

So how can i add it back in, but for only the clients that can accept
it; a bit silly to ALWAYS add it in especially if its not been set.

I do not know conditions when Amazon may refuse to compresses response,
it may have simillar limits as nginx has:

this has nothing to do with Amazon. I merely was illustrating
that our infastructure runs within the EC2 cloud.

On Fri, Aug 29, 2008 at 12:08:21PM +0100, Alan W. wrote:

If you did not set
proxy_set_header Accept-Encoding “”;

then nginx will pass it backend.

i do not understand this. Are you suggesting i need to set this in
order for nginx to send that header to the back end?

No, if you set this then nginx will never pass Accept-Encoding to
backend.

What about the other headers that get sent with the request (cookie etc)
we don’t manually set them do we? They simply get passed to the backend.

By default all headers goes to backend as is.
There are two exceptions:

  1. Host: nginx sets it to $proxy_host value,
  2. Connection: nginx sets it “close”

It always bothers me that nginx doesn’t pass “Host:” one and i have to
set this manually, makes me wonder what other headers nginx is silently
rejecting and not passing on to the backend.

nginx always sends Host. However, it sets to a value that backend
usually
expects to get. In this configuration:

 server {
     server_name  ONE;
     location / {
         proxy_pass http://TWO;
     }
 }

the TWO server usually expects to get “Host: TWO”, but not “Host: ONE”
as the ONE server expects. If your TWO server requires “Host: ONE”,
then you have to add manually

 proxy_set_header Host $host;

or
proxy_set_header Host ONE;

can you tell me what the definitive answer is here?

------ snippet -------
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header x-forwarded-for $remote_addr;
proxy_pass http://backends;
}

Try to remove

 proxy_set_header x-forwarded-for $remote_addr;

This may be treated as proxied request on backend side.

So if a client sends in the header that he can accept GZIP encoding,
nginx is removing this and not passing it through.

So how can i add it back in, but for only the clients that can accept
it; a bit silly to ALWAYS add it in especially if its not been set.

As I have said before nginx does not remove “Accept-Enconding” by
default.

I do not know conditions when Amazon may refuse to compresses response,
it may have simillar limits as nginx has:

this has nothing to do with Amazon. I merely was illustrating
that our infastructure runs within the EC2 cloud.

As I understand EC2 cloud are Amazon servers. Are your backends EC2
servers ?

Alan W. a écrit :

I do not know conditions when Amazon may refuse to compresses response,
it may have simillar limits as nginx has:

this has nothing to do with Amazon. I merely was illustrating
that our infastructure runs within the EC2 cloud.

NginX do only “HTTP/1.0” requests with backend, so if Amazan do
compression only for “HTTP/1.1” requests (like NginX seems to do), then
the result will not be compressed.

By default all headers goes to backend as is.
There are two exceptions:

  1. Host: nginx sets it to $proxy_host value,
  2. Connection: nginx sets it “close”

okay thats that bit cleared up. thank you.

As I have said before nginx does not remove “Accept-Enconding” by default.

okay, i think the clue may lie in the fact nginx proxies in only
“HTTP/1.0”. So i would have to override the backend to break the
protocol slightly here, and if receive a request in HTTP/1.0 but still
can accept GZIP then give it.

Why only 1.0 incidentally?

As I understand EC2 cloud are Amazon servers. Are your backends EC2 servers ?

yes, but i suspect you don’t have much experence in this world. Amazon
has nothing to do with it, they are merely a cloud provider. The
instances are running OUR software, on OUR Linux setup. I shouldn’t
have brought it up, as it appears to be ‘clouding’ the issue! :slight_smile:

On Fri, Aug 29, 2008 at 11:04:57AM +0100, Alan W. wrote:

backend, including the all important:

“Accept-Encoding: gzip,deflate”

?

If you did not set
proxy_set_header Accept-Encoding “”;

then nginx will pass it backend.

I do not know conditions when Amazon may refuse to compresses response,
it may have simillar limits as nginx has:

  1. gzip_http_version: nginx by default compresses HTTP/1.1 requests
    only,
    Amazon may do the same. nginx talks to backend using HTTP/1.0 while
    browsers usually use HTTP/1.1.

  2. gzip_proxied: nginx by default disables compression if a request has
    “Via” header.

On Fri, Aug 29, 2008 at 12:47:15PM +0100, Alan W. wrote:

okay, i think the clue may lie in the fact nginx proxies in only
“HTTP/1.0”. So i would have to override the backend to break the
protocol slightly here, and if receive a request in HTTP/1.0 but still
can accept GZIP then give it.

Why only 1.0 incidentally?

Why does nginx talk to backend using HTTP/1.0 ? Because it does not
support
chunked responses which are allowable in HTTP/1.1.

As I understand EC2 cloud are Amazon servers. Are your backends EC2
servers ?

yes, but i suspect you don’t have much experence in this world. Amazon
has nothing to do with it, they are merely a cloud provider. The
instances are running OUR software, on OUR Linux setup. I shouldn’t
have brought it up, as it appears to be ‘clouding’ the issue! :slight_smile:

You are right, I do not know this area absolutely.
What do you use at your backends ?