Forum: NGINX problem with echo_before when proxying a server which sends gzipped content

Caec75a457b17eb669102b54929ff522?d=identicon&s=25 Carsten Germer (Guest)
on 2014-04-02 14:49
(Received via mailing list)
Hi everyone,
currently I'm, trying to configure NGINX as a proxy for JSON from the
iTunes API.
It's for a small game, iTunes is slow sometimes and the data for the
game is mostly the same for a good length of time, anyway.
The JSON from iTunes is to be padded with the original requests callback
parameters. For this there are many good posts out on the net, but I
can't seem to get the basic echo_* to work.

I boiled my configuration of nginx down to the point where I just use
echo_before or echo_after and proxy_pass.

If I append something with echo_after it works fine in browsers and in
jQuery.
If I prepend anything with echo_before the answer can't be read by
browsers, "curl --compressed" throws "curl: (23) Error while processing
content unencoding: invalid block type".

If I configure Firefox with "about:config" to
"network.http.accept-encoding:true" it fixes fixes display in Firefox.

When I look in the network tab of chrome console I see that requesting
".../echo-after/" closes the request after 2Xms.
Requesting ".../echo-before" also gets 200 ok but never arrives fully,
is shown as "pending" indefinitely.

My best bet is, that it has something to do with gzip-compressed answer
from iTunes but I can't find any solution or even hint for my level of
understanding of the inner workings of nginx.

One more info: I tried many different combinations of "gzip on|off" and
other directives, basically I indiscriminately tried everything I found
mentioned somewhere. But, nothing changed the behavior much as far as I
could see, so I stripped it out of the configuration again for
readability.

<nginx configuration for testing>
# Proxy iTunes for bug hunting
# test by accessing
http://json.musiguess.com/itunes/[raw|echo-after|e...

location /itunes/raw/ {
  # suppress proxying for testing completely
  proxy_cache off;
  # getting a json containing N current top albums from itunes
  proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/e...
}
location /itunes/echo-after/ {
  proxy_cache off;
  proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/e...
  # echo something after body
  echo_after_body ");";
}
location /itunes/echo-before/ {
  proxy_cache off;
  # echo something before body
  echo_before_body -n "abc(";
  proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/e...
}
location /itunes/echo-beforeNafter/ {
  proxy_cache off;
  echo_before_body -n "abc(";
  proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/e...
  echo_after_body ");";
}
</nginx configuration for testing>

You are very welcome to access http://json.musiguess.com/itunes/... if
it may help you assess my problem.


<info about my installation>
~# nginx -V
nginx version: nginx/1.5.10
built by gcc 4.7.2 (Debian 4.7.2-5)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_dav_module --with-http_flv_module --with-http_mp4_module
--with-http_gunzip_module --with-http_gzip_static_module
--with-http_random_index_module --with-http_secure_link_module
--with-http_stub_status_module --with-http_auth_request_module
--add-module=/var/cache/nginx/nginx-1.5.10/src/echo --with-mail
--with-mail_ssl_module --with-file-aio --w
 ith-http_spdy_module --with-cc-opt='-g -O2 -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security'
--with-ld-opt=-Wl,-z,relro --with-ipv6

Echo v0.51
</info about my installation>

Any help is greatly appreciated!
Cheers /Carsten

---
Carsten Germer
Creative Director
intolabs GmbH
http://www.intolabs.net/
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2014-04-02 15:47
(Received via mailing list)
Hello!

On Wed, Apr 02, 2014 at 02:48:33PM +0200, Carsten Germer wrote:

> just use echo_before or echo_after and proxy_pass.
>
> When I look in the network tab of chrome console I see that
> requesting ".../echo-after/" closes the request after 2Xms.
> Requesting ".../echo-before" also gets 200 ok but never arrives
> fully, is shown as "pending" indefinitely.
>
> My best bet is, that it has something to do with gzip-compressed
> answer from iTunes but I can't find any solution or even hint
> for my level of understanding of the inner workings of nginx.

Something like

    proxy_set_header Accept-Encoding "";

in relevant location should help.

BTW, you may try add_before_body / add_after_body as available in
standard addition filter module instead, see here:

http://nginx.org/en/docs/http/ngx_http_addition_module.html

--
Maxim Dounin
http://nginx.org/
Caec75a457b17eb669102b54929ff522?d=identicon&s=25 Carsten Germer (Guest)
on 2014-04-04 10:17
(Received via mailing list)
Hey Maxim,

yes, it works with suppressing gzip between nginx and source-server with
"proxy_set_header Accept-Encoding "deflate";"
Thanks a bunch!
I was aiming for a solution that preserves the gzip-compression between
source and cache, but I'm caching long time, anyway.

> add_before_body / add_after_body
I did look at those before but I'd still have to find a way to get the
$arg_callback to the URIs and output them with echo for the whole
solution.

Works very fine with the neutered compression, thanks again!

Cheers /Carsten

---
Carsten Germer
Creative Director
intolabs GmbH
http://www.intolabs.net/

Am 02.04.2014 um 19:21 schrieb nginx-request@nginx.org:
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2014-04-29 21:26
(Received via mailing list)
Hello!

On Thu, Apr 3, 2014 at 8:14 AM, Carsten Germer wrote:
> yes, it works with suppressing gzip between nginx and source-server with
"proxy_set_header Accept-Encoding "deflate";"
> Thanks a bunch!
> I was aiming for a solution that preserves the gzip-compression between source
and cache, but I'm caching long time, anyway.
>
>> add_before_body / add_after_body
> I did look at those before but I'd still have to find a way to get the
$arg_callback to the URIs and output them with echo for the whole solution.

As the author of both the ngx_echo and ngx_xss modules, I'd recommend
ngx_xss module for your JSONP use case:

    https://github.com/openresty/xss-nginx-module#readme

It should be a bit faster and also safer for this very case.

Regards,
-agentzh
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.