Perl module get an empty body

Server NGINX
1

Hi all,

I’m using Nginx as a reverse proxy and I use Nginx perl module to
inspect my requests and redirect to a different upstream.
It works well but some of request_body are empty and others just make
nginx timeout.

Here is my nginx.conf:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
worker_connections 768;

multi_accept on;

}

http {

Basic Settings

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;

server_tokens off;

server_names_hash_bucket_size 64;

server_name_in_redirect off;

include /etc/nginx/mime.types;
default_type application/octet-stream;

Logging Settings

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

Gzip Settings

gzip on;
gzip_disable “msie6”;

gzip_vary on;

gzip_proxied any;

gzip_comp_level 6;

gzip_buffers 16 8k;

gzip_http_version 1.1;

gzip_types text/plain text/css application/json

application/x-javascript text/xml application/xml application/xml+rss
text/javascript;

Virtual Host Configs

    perl_modules /opt/test/perl/;
    perl_require switcher_test.pm;
    perl_set $test switcher_test::handler;

    log_format cestate '$remote_addr,$http_x_forwarded_for -

$remote_user [$time_local] “$request” $status $body_bytes_sent
“$http_referer” “$http_user_agent” $request_time # $test’;
access_log /var/log/nginx/cestate.log cestate;

include /etc/nginx/conf.d/.conf;
include /etc/nginx/sites-enabled/;
}

My proxy.conf:
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
client_max_body_size 21474836470;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 3600;
proxy_buffers 32 8k;
proxy_max_temp_file_size 0;

My script:
#!/usr/bin/perl

package switcher_test;
use nginx;

sub handler {
my $r = shift;
if($r->has_request_body(sub{})) { return 3 };
return 4;

}

1;
END

My vhost on test the value of $test and proxy_pass to the good
upstream.

When Nginx timeout, I have the following log:
2012/08/10 11:56:15 [alert] 30570#0: *1 zero size buf in output t:1 r:0
f:0 0000000001481728 0000000001481728-0000000001481728 0000000000000000
0-0 while sending request to upstream, client: xxx.xxx.xxx.xxx, server:
www.test.com, request: “POST /test.php HTTP/1.1”, upstream:
“http://xxx.xxx.xxx.xxx:80/test.php”, host: “www.test.com”
2012/08/10 11:56:41 [info] 30580#0: Using 32768KiB of shared memory for
push module in /etc/nginx/nginx.conf:63

Has someone any idea of what’s going on ?

Thanks.

Rémi

Posted at Nginx Forum:

2

I forgot to mention. I’m using the following nginx version:

nginx -vV

nginx version: nginx/1.2.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-client-body-temp-path=/var/lib/nginx/body
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi
–http-log-path=/var/log/nginx/access.log
–http-proxy-temp-path=/var/lib/nginx/proxy
–http-scgi-temp-path=/var/lib/nginx/scgi
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi
–lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid
–with-pcre-jit --with-debug --with-http_addition_module
–with-http_dav_module --with-http_flv_module --with-http_geoip_module
–with-http_gzip_static_module --with-http_image_filter_module
–with-http_mp4_module --with-http_perl_module
–with-http_random_index_module --with-http_realip_module
–with-http_secure_link_module --with-http_stub_status_module
–with-http_ssl_module --with-http_sub_module --with-http_xslt_module
–with-ipv6 --with-sha1=/usr/include/openssl
–with-md5=/usr/include/openssl --with-mail --with-mail_ssl_module
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-auth-pam
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/chunkin-nginx-module
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/headers-more-nginx-module
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-development-kit
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-echo
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-http-push
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-lua
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-upload-module
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-upload-progress
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-upstream-fair
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-dav-ext-module

Posted at Nginx Forum:

3

It can be reproduced by commenting/uncommenting the append in the
following code:

#!/usr/bin/python

-- coding: utf-8 --

import urllib
import urllib2

url = ‘https://www.test.com/test.php’
params = { ‘name’: ‘Luke’, ‘location’: ‘Tatooine’ }
pList = []
for paramKey, paramValue in params.items():
pList.append(urllib.quote_plus(unicode(‘%s=%s’ % (paramKey,
paramValue))))
#pList.append(‘%s=%s’ % (paramKey,
urllib.quote_plus(unicode(paramValue))))

post_params = ‘&’.join(pList)
req = urllib2.Request(url, data=post_params)
f = urllib2.urlopen(req)

print f.read()

Posted at Nginx Forum:

4

Hi Maxim,

Thanks for your reply.

I don’t understand what you mean by “content handler” as I’m a noob in
nginx perl usage (and I’m not sure it was clear enough for you). I
tested http_content_length but it’s empty also.
I really need to have the calling request body (I deleted the major part
of my code but I parse some stuff in).

Let’s say I want to return 3 to perl_set if the request body match
“test” and 4 otherwise (and log it), I would do:

#!/usr/bin/perl

package switcher_test;
use nginx;
use Sys::Syslog;

my $facility = ‘local2’;

sub handler {
my $r = shift;

    openlog('tester', 'ndelay', $facility);

    $r->has_request_body(sub{});
    if ($r->request_body =~ /.*test.*/) { return 3 };

    syslog(LOG_DEBUG, 'request_body: '.$r->request_body);

    return 4;

}

1;
END

It works but when another program (developed by another company who
don’t want to change the code) POST to my Nginx using:

pList.append(‘%s=%s’ % (paramKey,
urllib.quote_plus(unicode(paramValue))))

(cf. my test program above)

it doesn’t works and Nginx timeouts.

According to your previous answer how would you do it ?

Thanks.

Rémi

Posted at Nginx Forum:

5

“and I’m not sure it was clear enough for you” - I meant my post wasn’t
clear enough

Posted at Nginx Forum:

6

Hello!

On Fri, Aug 10, 2012 at 09:44:14AM -0400, darkweaver871 wrote:

Let’s say I want to return 3 to perl_set if the request body match
“test” and 4 otherwise (and log it), I would do:

You have to use perl directive to handle request. Then you’ll be
able to use $r->has_request_body(), see here:

http://nginx.org/en/docs/http/ngx_http_perl_module.html#methods

For conditional processing by other modules you may use
$r->internal_redirect() method.

    openlog('tester', 'ndelay', $facility);

    $r->has_request_body(sub{});
    if ($r->request_body =~ /.*test.*/) { return 3 };

This is just wrong: $r->request_body will be only available when
body handler function is called (“sub{}” in the above code).
Using it here won’t work.

[…]

Maxim D.

7

Hello!

On Fri, Aug 10, 2012 at 06:03:14AM -0400, darkweaver871 wrote:

Hi all,

I’m using Nginx as a reverse proxy and I use Nginx perl module to
inspect my requests and redirect to a different upstream.
It works well but some of request_body are empty and others just make
nginx timeout.

[…]

    perl_modules /opt/test/perl/;
    perl_require switcher_test.pm;
    perl_set $test switcher_test::handler;

[…]

sub handler {
my $r = shift;
if($r->has_request_body(sub{})) { return 3 };

This doesn’t work as $r->has_request_body() currently assumes it’s
called from a perl content handler, not from a perl variable
handler, and does bad things if called from a variable handler.

You may try to test $http_content_length instead.

Maxim D.

8

OK, so is this better ?

#!/usr/bin/perl

package switcher_test;
use nginx;

use Sys::Syslog;

my $facility = ‘local2’;

sub handler {
my $r = shift;

    return $r->has_request_body(\& test);

}

sub test {
my $r = shift;
if ($r->request_body =~ /.test./) { return 3 };
openlog(‘switcher’, ‘ndelay’, $facility);
syslog(LOG_DEBUG, 'request_body: '.$r->request_body);
syslog(LOG_DEBUG, 'request_body_file: '.$r->request_body_file);
return 4;
}

1;
END

It still doesn’t work this way.

Posted at Nginx Forum:

9

Hello!

On Fri, Aug 10, 2012 at 10:12:56AM -0400, darkweaver871 wrote:

    openlog('switcher', 'ndelay', $facility);
    syslog(LOG_DEBUG, 'request_body: '.$r->request_body);
    syslog(LOG_DEBUG, 'request_body_file: '.$r->request_body_file);
    return 4;

}

1;
END

It still doesn’t work this way.

Again: you can’t use this code in perl_set directive. You have to
use perl directive:

perl switcher_test::handler;

And in perl package (mostly cut-n-paste from docs, with an
additional test for body):

package switcher_test;

use nginx;

sub handler {
my $r = shift;

if ($r->request_method ne "POST") {
    return DECLINED;
}

if ($r->has_request_body(\&post)) {
    return OK;
}

return HTTP_BAD_REQUEST;

}

sub post {
my $r = shift;

$r->send_http_header;

$r->print("request_body: \"", $r->request_body, "\"<br/>");
$r->print("request_body_file: \"", $r->request_body_file,

“”
\n");

if ($r->request_body =~ /test/) {
    return HTTP_BAD_REQUEST;
}

return OK;

}

1;

Maxim D.

10

Hello Maxim,

OK got it now :wink:
It’s working, you saved my day :smiley:

So to put it in a nutshell:

  • I used perl switcher_test::handler in my location /
  • in the handler method I call $r->has_request_body(& test) and in test
    method according to what I found in the body I do an internal_redirect
    (as far as I understood it can only redirect to a location in the same
    vhost) to /proxy1 (or /proxy2 depending) and return.
  • in /proxyX I do a rewrite /proxyX(.*) $1 break; proxy_pass
    http://my_upstream

It’s working perfectly but I just wonder if there is a more
elegant/efficient way to do it ?

Rémi

Posted at Nginx Forum: