Forum: NGINX nginx segfaulting with mod_security

111a08d93ad6b7c5b4f5eb09c45c746d?d=identicon&s=25 Robert Paprocki (Guest)
on 2014-04-13 01:45
(Received via mailing list)
Hello,

I have compiled nginx-1.5.13 with modsecurity-2.7.7 and am seeing
occasional segfaults when sending requests to the server. mod_security
was compiled as a standalone module per the instructions made available
at
https://github.com/SpiderLabs/ModSecurity/wiki/Ref....
The segfaults appear sporadic and do not seem to match up with any given
request. Below is my nginx configuration:

[root@poseidon src]# nginx -V
nginx version: nginx/1.5.13
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-debug --with-http_ssl_module
--with-http_realip_module --with-http_addition_module
--with-http_sub_module --with-http_dav_module --with-http_flv_module
--with-http_mp4_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module
--with-mail --with-mail_ssl_module --with-file-aio --with-ipv6
--with-cc-opt='-g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
-mtune=generic -fasynchronous-unwind-tables -g -O0'
--add-module=../modsecurity-apache_2.7.7/nginx/modsecurity/

Also, a backtrace of the core dump:
(gdb) bt
#0  0x080a1827 in ngx_http_write_filter (r=0x83bb078, in=0x8baaa6c) at
src/http/ngx_http_write_filter_module.c:121
#1  0x080bc0d4 in ngx_http_chunked_body_filter (r=0x83bb078,
in=0x8baaa6c)
    at src/http/modules/ngx_http_chunked_filter_module.c:111
#2  0x080c462b in ngx_http_gzip_body_filter (r=0x83bb078, in=0x8baaa6c)
    at src/http/modules/ngx_http_gzip_filter_module.c:325
#3  0x080c5fb3 in ngx_http_postpone_filter (r=0x83bb078, in=0x8baaa6c)
    at src/http/ngx_http_postpone_filter_module.c:82
#4  0x080c6581 in ngx_http_ssi_body_filter (r=0x83bb078, in=0x8baaa6c)
    at src/http/modules/ngx_http_ssi_filter_module.c:408
#5  0x080cc021 in ngx_http_charset_body_filter (r=0x83bb078,
in=0x8baaa6c)
    at src/http/modules/ngx_http_charset_filter_module.c:553
#6  0x080ce31f in ngx_http_sub_body_filter (r=0x83bb078, in=0x8baaa6c)
    at src/http/modules/ngx_http_sub_filter_module.c:201
#7  0x080cf730 in ngx_http_addition_body_filter (r=0x83bb078,
in=0x8baaa6c)
    at src/http/modules/ngx_http_addition_filter_module.c:147
#8  0x080cfc78 in ngx_http_gunzip_body_filter (r=0x83bb078,
in=0x8baaa6c)
    at src/http/modules/ngx_http_gunzip_filter_module.c:184
#9  0x081146bd in ngx_http_modsecurity_body_filter (r=0x83bb078,
in=0xbf7ff8b4)
    at
../modsecurity-apache_2.7.7/nginx/modsecurity//ngx_http_modsecurity.c:1252
#10 0x08055381 in ngx_output_chain (ctx=0x8baa9b8, in=0xbf7ff8b4) at
src/core/ngx_output_chain.c:66
#11 0x080a253c in ngx_http_copy_filter (r=0x83bb078, in=0xbf7ff8b4) at
src/http/ngx_http_copy_filter_module.c:143
#12 0x080bd477 in ngx_http_range_body_filter (r=0x83bb078,
in=0xbf7ff8b4)
    at src/http/modules/ngx_http_range_filter_module.c:594
#13 0x0808e81e in ngx_http_output_filter (r=0x83bb078, in=0xbf7ff8b4) at
src/http/ngx_http_core_module.c:1964
#14 0x0809c72f in ngx_http_send_special (r=0x83bb078, flags=1) at
src/http/ngx_http_request.c:3332
#15 0x080b5737 in ngx_http_upstream_finalize_request (r=0x83bb078,
u=0x83bbab0, rc=0)
    at src/http/ngx_http_upstream.c:3551
#16 0x080b4a77 in ngx_http_upstream_process_request (r=0x83bb078) at
src/http/ngx_http_upstream.c:3159
#17 0x080b477e in ngx_http_upstream_process_upstream (r=0x83bb078,
u=0x83bbab0) at src/http/ngx_http_upstream.c:3090
#18 0x080b329a in ngx_http_upstream_send_response (r=0x83bb078,
u=0x83bbab0) at src/http/ngx_http_upstream.c:2493
#19 0x080b1937 in ngx_http_upstream_process_header (r=0x83bb078,
u=0x83bbab0) at src/http/ngx_http_upstream.c:1735
#20 0x080b02ef in ngx_http_upstream_handler (ev=0x8b31f5c) at
src/http/ngx_http_upstream.c:977
#21 0x080726fd in ngx_event_process_posted (cycle=0x83b45a8,
posted=0x81c495c) at src/event/ngx_event_posted.c:40
#22 0x080708c2 in ngx_process_events_and_timers (cycle=0x83b45a8) at
src/event/ngx_event.c:275
#23 0x0807c629 in ngx_worker_process_cycle (cycle=0x83b45a8, data=0x0)
at src/os/unix/ngx_process_cycle.c:816
#24 0x080795a4 in ngx_spawn_process (cycle=0x83b45a8, proc=0x807c48e
<ngx_worker_process_cycle>, data=0x0,
    name=0x815e33b "worker process", respawn=-3) at
src/os/unix/ngx_process.c:198
#25 0x0807b720 in ngx_start_worker_processes (cycle=0x83b45a8, n=2,
type=-3) at src/os/unix/ngx_process_cycle.c:364
#26 0x0807aecf in ngx_master_process_cycle (cycle=0x83b45a8) at
src/os/unix/ngx_process_cycle.c:136
#27 0x080500c5 in main (argc=3, argv=0xbf7ffe54) at src/core/nginx.c:407

Unfortunately I am not skilled at reading c backtraces. I was going to
attach the debug log but it's very large and I don't want to make thi
message much larger :p Below is my nginx coniguration:

user  nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;
worker_rlimit_core  500M;
working_directory   /tmp;

events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local]
"$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
    fastcgi_buffers 256 4k;
    client_max_body_size 64m;
    #client_body_buffer_size 16m;
    server_tokens off;
}

server {
    listen       23.226.226.175:80;
    server_name  cryptobells.com www.cryptobells.com;
    root /var/www/cryptobells;
    rewrite     ^   https://$server_name$request_uri? permanent;
    location / {
        index  index.php index.html index.htm;
        try_files $uri $uri/ /index.php?$args;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    location ~* \.php$ {
        fastcgi_index   index.php;
        fastcgi_pass  unix:/var/run/php-fpm/php-fpm.sock;
        include         fastcgi_params;
        fastcgi_param   SCRIPT_FILENAME
$document_root$fastcgi_script_name;
        fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
    }
}

server {
    listen       23.226.226.175:443 ssl;
    server_name  cryptobells.com www.cryptobells.com;

    ssl_certificate      /etc/ssl/certs/cryptobells.com.crt;
    ssl_certificate_key  /etc/ssl/certs/cryptobells.com.key;

    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  5m;

    ssl_ciphers
ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH;
    ssl_prefer_server_ciphers   on;

    root /var/www/cryptobells;
  ModSecurityEnabled on;
        ModSecurityConfig /etc/modsecurity/modsecurity.conf;

    location / {
        index  index.php index.html index.htm;
        try_files $uri $uri/ /index.php?$args;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    location ~* \.php$ {
        fastcgi_index   index.php;
        fastcgi_pass  unix:/var/run/php-fpm/php-fpm.sock;
        include         fastcgi_params;
        fastcgi_param   SCRIPT_FILENAME
$document_root$fastcgi_script_name;
        fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
    }

Please let me know if anyone is able to help identify what could be
causing segfaults, ro if there is any more information I can provide.
Thank you!
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2014-04-13 12:18
(Received via mailing list)
Hello!

On Sat, Apr 12, 2014 at 04:44:28PM -0700, Robert Paprocki wrote:

> Hello,
>
> I have compiled nginx-1.5.13 with modsecurity-2.7.7 and am seeing
> occasional segfaults when sending requests to the server. mod_security
> was compiled as a standalone module per the instructions made available
> at
>
https://github.com/SpiderLabs/ModSecurity/wiki/Ref....
> The segfaults appear sporadic and do not seem to match up with any given
> request. Below is my nginx configuration:

[...]

> Also, a backtrace of the core dump:
> (gdb) bt
> #0  0x080a1827 in ngx_http_write_filter (r=0x83bb078, in=0x8baaa6c) at
> src/http/ngx_http_write_filter_module.c:121

This points to the following code line:

        cl->buf = ln->buf;

That is, dereferencing ln->buf fails, which may only happen if the
buffer chain ("in" argument) is broken.

[...]

> #8  0x080cfc78 in ngx_http_gunzip_body_filter (r=0x83bb078, in=0x8baaa6c)
>     at src/http/modules/ngx_http_gunzip_filter_module.c:184
> #9  0x081146bd in ngx_http_modsecurity_body_filter (r=0x83bb078,
> in=0xbf7ff8b4)
>     at
> ../modsecurity-apache_2.7.7/nginx/modsecurity//ngx_http_modsecurity.c:1252
> #10 0x08055381 in ngx_output_chain (ctx=0x8baa9b8, in=0xbf7ff8b4) at
> src/core/ngx_output_chain.c:66

And this clearly shows that the buffer chain was chaned by
mod_security output body filter.  Note "in" argument of
mod_security ("in=0xbf7ff8b4") and gunzip filter which follows it
("in=0x8baaa6c").

That is, from the backtrace it looks like mod_security changed the
buffer chain and did it wrong, with a segfault as a result.

--
Maxim Dounin
http://nginx.org/
E49afde41ff1b58837f111e14f35f7d4?d=identicon&s=25 Henry Suhatman (Guest)
on 2014-04-13 12:28
(Received via mailing list)
B.
111a08d93ad6b7c5b4f5eb09c45c746d?d=identicon&s=25 Robert Paprocki (Guest)
on 2014-04-14 05:43
(Received via mailing list)
Hi Maxim!

Thank you for your response, always nice to actually hear back from
someone knowledgeable. Once thing I had noticed while looking at
backtraces (coredumps seem to indicate segfaults occurring in a number
of places, not just filter_module.c:121) was that every bt seemed to
include gzip modules as well. i disabled gzip in both my server and http
sections but this did not stop the segfaults (which is interesting, but
not entirely surprising, given that even when I had set SecRuleEngine to
off in my modsecurity.conf file, segfaults still occured... so even the
mere presence of ModSecurityEnabled in the nginx configuration was
leading to a break).

I have since recompiled nginx without both gunzip module and gzip static
module, and have not yet gotten any segfaults. i will continue to
research this and update if I have any more information
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2014-04-14 12:45
(Received via mailing list)
Hello!

On Sun, Apr 13, 2014 at 08:42:04PM -0700, Robert Paprocki wrote:

> Hi Maxim!
>
> Thank you for your response, always nice to actually hear back from
> someone knowledgeable. Once thing I had noticed while looking at
> backtraces (coredumps seem to indicate segfaults occurring in a number
> of places, not just filter_module.c:121) was that every bt seemed to
> include gzip modules as well. i disabled gzip in both my server and http
> sections but this did not stop the segfaults (which is interesting, but

All filters, including gzip, are expected to be in backtraces, as
they are always called.  Depending on configuration, they either
do something or not.

> not entirely surprising, given that even when I had set SecRuleEngine to
> off in my modsecurity.conf file, segfaults still occured... so even the
> mere presence of ModSecurityEnabled in the nginx configuration was
> leading to a break).
>
> I have since recompiled nginx without both gunzip module and gzip static
> module, and have not yet gotten any segfaults. i will continue to
> research this and update if I have any more information

As per backtrace, it's more or less obvious that the problem is in
modsecurity.  Compiling out other modules may result in less
frequent segfaults, but it won't fix the bug in modsecurity.

If you want to actually fix the problem, you should either
switch off / compile out modsecurity, or find and fix the bug in
it.

> >> was compiled as a standalone module per the instructions made available
> >> src/http/ngx_http_write_filter_module.c:121
> >> #8  0x080cfc78 in ngx_http_gunzip_body_filter (r=0x83bb078, in=0x8baaa6c)
> > mod_security ("in=0xbf7ff8b4") and gunzip filter which follows it
> > ("in=0x8baaa6c").
> >
> > That is, from the backtrace it looks like mod_security changed the
> > buffer chain and did it wrong, with a segfault as a result.
> >
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

--
Maxim Dounin
http://nginx.org/
111a08d93ad6b7c5b4f5eb09c45c746d?d=identicon&s=25 Robert Paprocki (Guest)
on 2014-04-14 22:30
(Received via mailing list)
I realized that I spoke too soon about gzip shortly after sending this.
My apologies for making a silly assumption; I am a sysadmin by trade and
not so skilled at developing or troubleshooting complex server software.
I've contact mod_security mailing list but haven't heard back from them.
Thank you for your time and patience in answering my questions!

Sincerely,
Robert Paprocki
2974d09ac2541e892966b762aad84943?d=identicon&s=25 mex (Guest)
on 2014-04-15 00:24
(Received via mailing list)
hi robert,


if you dont depend on mod_security's advanced features like
output-filtering'n'stuff you might want to try naxsi
https://github.com/nbs-system/naxsi/wiki

its stable, its fast, rules are easy to create and understand and it
provides a set of basic features for a waf.

the community is responsive and open for feature-requests or bugreports.



regards,

mex

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249248,249294#msg-249294
111a08d93ad6b7c5b4f5eb09c45c746d?d=identicon&s=25 Robert Paprocki (Guest)
on 2014-04-15 00:51
(Received via mailing list)
Hi mex!

Thanks for the tip! I've known about naxsi for a while. I'm researching
various WAF options that will scale well for my Master's thesis, and
mod_security + nginx interested me (my other research is pointing
towards using Varnish, for which several WAF solutions have already been
somewhat implemented). Naxsi doesn't seem to offer the extensive logging
and detailed features like state tracking that mod_sec does, so I have
been wary to research further into it. But thanks for the suggestion!

Sincerely,
Robert Paprocki
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.