Problem with echo_before when proxying a server which sends gzipped content

Server NGINX
1

Hi everyone,
currently I’m, trying to configure NGINX as a proxy for JSON from the
iTunes API.
It’s for a small game, iTunes is slow sometimes and the data for the
game is mostly the same for a good length of time, anyway.
The JSON from iTunes is to be padded with the original requests callback
parameters. For this there are many good posts out on the net, but I
can’t seem to get the basic echo_* to work.

I boiled my configuration of nginx down to the point where I just use
echo_before or echo_after and proxy_pass.

If I append something with echo_after it works fine in browsers and in
jQuery.
If I prepend anything with echo_before the answer can’t be read by
browsers, “curl --compressed” throws “curl: (23) Error while processing
content unencoding: invalid block type”.

If I configure Firefox with “about:config” to
“network.http.accept-encoding:true” it fixes fixes display in Firefox.

When I look in the network tab of chrome console I see that requesting
“…/echo-after/” closes the request after 2Xms.
Requesting “…/echo-before” also gets 200 ok but never arrives fully,
is shown as “pending” indefinitely.

My best bet is, that it has something to do with gzip-compressed answer
from iTunes but I can’t find any solution or even hint for my level of
understanding of the inner workings of nginx.

One more info: I tried many different combinations of “gzip on|off” and
other directives, basically I indiscriminately tried everything I found
mentioned somewhere. But, nothing changed the behavior much as far as I
could see, so I stripped it out of the configuration again for
readability.

# Proxy iTunes for bug hunting # test by accessing http://json.musiguess.com/itunes/[raw|echo-after|echo-before]/

location /itunes/raw/ {

suppress proxying for testing completely

proxy_cache off;

getting a json containing N current top albums from itunes

proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;
}
location /itunes/echo-after/ {
proxy_cache off;
proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;

echo something after body

echo_after_body “);”;
}
location /itunes/echo-before/ {
proxy_cache off;

echo something before body

echo_before_body -n “abc(”;
proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;
}
location /itunes/echo-beforeNafter/ {
proxy_cache off;
echo_before_body -n “abc(”;
proxy_pass
http://itunes.apple.com/de/rss/topalbums/limit=2/explicit=true/json;
echo_after_body “);”;
}
</nginx configuration for testing>

You are very welcome to access Home - Ganske.de… if
it may help you assess my problem.

~# nginx -V nginx version: nginx/1.5.10 built by gcc 4.7.2 (Debian 4.7.2-5) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --add-module=/var/cache/nginx/nginx-1.5.10/src/echo --with-mail --with-mail_ssl_module --with-file-aio --w ith-http_spdy_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' --with-ld-opt=-Wl,-z,relro --with-ipv6

Echo v0.51
</info about my installation>

Any help is greatly appreciated!
Cheers /Carsten

Carsten Germer
Creative Director
intolabs GmbH
http://www.intolabs.net/

2

Hello!

On Wed, Apr 02, 2014 at 02:48:33PM +0200, Carsten Germer wrote:

just use echo_before or echo_after and proxy_pass.

When I look in the network tab of chrome console I see that
requesting “…/echo-after/” closes the request after 2Xms.
Requesting “…/echo-before” also gets 200 ok but never arrives
fully, is shown as “pending” indefinitely.

My best bet is, that it has something to do with gzip-compressed
answer from iTunes but I can’t find any solution or even hint
for my level of understanding of the inner workings of nginx.

Something like

proxy_set_header Accept-Encoding "";

in relevant location should help.

BTW, you may try add_before_body / add_after_body as available in
standard addition filter module instead, see here:

http://nginx.org/en/docs/http/ngx_http_addition_module.html

–
Maxim D.
http://nginx.org/

3

Hey Maxim,

yes, it works with suppressing gzip between nginx and source-server with
“proxy_set_header Accept-Encoding “deflate”;”
Thanks a bunch!
I was aiming for a solution that preserves the gzip-compression between
source and cache, but I’m caching long time, anyway.

add_before_body / add_after_body
I did look at those before but I’d still have to find a way to get the
$arg_callback to the URIs and output them with echo for the whole
solution.

Works very fine with the neutered compression, thanks again!

Cheers /Carsten

Carsten Germer
Creative Director
intolabs GmbH
http://www.intolabs.net/

Am 02.04.2014 um 19:21 schrieb [email protected]:

4

Hello!

On Thu, Apr 3, 2014 at 8:14 AM, Carsten Germer wrote:

yes, it works with suppressing gzip between nginx and source-server with
“proxy_set_header Accept-Encoding “deflate”;”
Thanks a bunch!
I was aiming for a solution that preserves the gzip-compression between source
and cache, but I’m caching long time, anyway.

add_before_body / add_after_body
I did look at those before but I’d still have to find a way to get the
$arg_callback to the URIs and output them with echo for the whole solution.

As the author of both the ngx_echo and ngx_xss modules, I’d recommend
ngx_xss module for your JSONP use case:

https://github.com/openresty/xss-nginx-module#readme

It should be a bit faster and also safer for this very case.

Regards,
-agentzh