Nginx segfaulting with mod_security

[musicdenotation]

Hello,

I have compiled nginx-1.5.13 with modsecurity-2.7.7 and am seeing
occasional segfaults when sending requests to the server. mod_security
was compiled as a standalone module per the instructions made available
at
Home · SpiderLabs/ModSecurity Wiki · GitHub.
The segfaults appear sporadic and do not seem to match up with any given
request. Below is my nginx configuration:

[root@poseidon src]# nginx -V
nginx version: nginx/1.5.13
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
–lock-path=/var/run/nginx.lock
–http-client-body-temp-path=/var/cache/nginx/client_temp
–http-proxy-temp-path=/var/cache/nginx/proxy_temp
–http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
–http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
–http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
–group=nginx --with-debug --with-http_ssl_module
–with-http_realip_module --with-http_addition_module
–with-http_sub_module --with-http_dav_module --with-http_flv_module
–with-http_mp4_module --with-http_gunzip_module
–with-http_gzip_static_module --with-http_random_index_module
–with-http_secure_link_module --with-http_stub_status_module
–with-mail --with-mail_ssl_module --with-file-aio --with-ipv6
–with-cc-opt=‘-g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
-mtune=generic -fasynchronous-unwind-tables -g -O0’
–add-module=…/modsecurity-apache_2.7.7/nginx/modsecurity/

Also, a backtrace of the core dump:
(gdb) bt
#0 0x080a1827 in ngx_http_write_filter (r=0x83bb078, in=0x8baaa6c) at
src/http/ngx_http_write_filter_module.c:121
#1 0x080bc0d4 in ngx_http_chunked_body_filter (r=0x83bb078,
in=0x8baaa6c)
at src/http/modules/ngx_http_chunked_filter_module.c:111
#2 0x080c462b in ngx_http_gzip_body_filter (r=0x83bb078, in=0x8baaa6c)
at src/http/modules/ngx_http_gzip_filter_module.c:325
#3 0x080c5fb3 in ngx_http_postpone_filter (r=0x83bb078, in=0x8baaa6c)
at src/http/ngx_http_postpone_filter_module.c:82
#4 0x080c6581 in ngx_http_ssi_body_filter (r=0x83bb078, in=0x8baaa6c)
at src/http/modules/ngx_http_ssi_filter_module.c:408
#5 0x080cc021 in ngx_http_charset_body_filter (r=0x83bb078,
in=0x8baaa6c)
at src/http/modules/ngx_http_charset_filter_module.c:553
#6 0x080ce31f in ngx_http_sub_body_filter (r=0x83bb078, in=0x8baaa6c)
at src/http/modules/ngx_http_sub_filter_module.c:201
#7 0x080cf730 in ngx_http_addition_body_filter (r=0x83bb078,
in=0x8baaa6c)
at src/http/modules/ngx_http_addition_filter_module.c:147
#8 0x080cfc78 in ngx_http_gunzip_body_filter (r=0x83bb078,
in=0x8baaa6c)
at src/http/modules/ngx_http_gunzip_filter_module.c:184
#9 0x081146bd in ngx_http_modsecurity_body_filter (r=0x83bb078,
in=0xbf7ff8b4)
at
…/modsecurity-apache_2.7.7/nginx/modsecurity//ngx_http_modsecurity.c:1252
#10 0x08055381 in ngx_output_chain (ctx=0x8baa9b8, in=0xbf7ff8b4) at
src/core/ngx_output_chain.c:66
#11 0x080a253c in ngx_http_copy_filter (r=0x83bb078, in=0xbf7ff8b4) at
src/http/ngx_http_copy_filter_module.c:143
#12 0x080bd477 in ngx_http_range_body_filter (r=0x83bb078,
in=0xbf7ff8b4)
at src/http/modules/ngx_http_range_filter_module.c:594
#13 0x0808e81e in ngx_http_output_filter (r=0x83bb078, in=0xbf7ff8b4) at
src/http/ngx_http_core_module.c:1964
#14 0x0809c72f in ngx_http_send_special (r=0x83bb078, flags=1) at
src/http/ngx_http_request.c:3332
#15 0x080b5737 in ngx_http_upstream_finalize_request (r=0x83bb078,
u=0x83bbab0, rc=0)
at src/http/ngx_http_upstream.c:3551
#16 0x080b4a77 in ngx_http_upstream_process_request (r=0x83bb078) at
src/http/ngx_http_upstream.c:3159
#17 0x080b477e in ngx_http_upstream_process_upstream (r=0x83bb078,
u=0x83bbab0) at src/http/ngx_http_upstream.c:3090
#18 0x080b329a in ngx_http_upstream_send_response (r=0x83bb078,
u=0x83bbab0) at src/http/ngx_http_upstream.c:2493
#19 0x080b1937 in ngx_http_upstream_process_header (r=0x83bb078,
u=0x83bbab0) at src/http/ngx_http_upstream.c:1735
#20 0x080b02ef in ngx_http_upstream_handler (ev=0x8b31f5c) at
src/http/ngx_http_upstream.c:977
#21 0x080726fd in ngx_event_process_posted (cycle=0x83b45a8,
posted=0x81c495c) at src/event/ngx_event_posted.c:40
#22 0x080708c2 in ngx_process_events_and_timers (cycle=0x83b45a8) at
src/event/ngx_event.c:275
#23 0x0807c629 in ngx_worker_process_cycle (cycle=0x83b45a8, data=0x0)
at src/os/unix/ngx_process_cycle.c:816
#24 0x080795a4 in ngx_spawn_process (cycle=0x83b45a8, proc=0x807c48e
<ngx_worker_process_cycle>, data=0x0,
name=0x815e33b “worker process”, respawn=-3) at
src/os/unix/ngx_process.c:198
#25 0x0807b720 in ngx_start_worker_processes (cycle=0x83b45a8, n=2,
type=-3) at src/os/unix/ngx_process_cycle.c:364
#26 0x0807aecf in ngx_master_process_cycle (cycle=0x83b45a8) at
src/os/unix/ngx_process_cycle.c:136
#27 0x080500c5 in main (argc=3, argv=0xbf7ffe54) at src/core/nginx.c:407

Unfortunately I am not skilled at reading c backtraces. I was going to
attach the debug log but it’s very large and I don’t want to make thi
message much larger Below is my nginx coniguration:

user nginx;
worker_processes 2;

error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;
worker_rlimit_core 500M;
working_directory /tmp;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local]

“$request” ’
'$status $body_bytes_sent “$http_referer” ’
‘“$http_user_agent” “$http_x_forwarded_for”’;

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  65;

#gzip  on;

include /etc/nginx/conf.d/*.conf;
fastcgi_buffers 256 4k;
client_max_body_size 64m;
#client_body_buffer_size 16m;
server_tokens off;

}

server {
listen 23.226.226.175:80;
server_name cryptobells.com www.cryptobells.com;
root /var/www/cryptobells;
rewrite ^ https://$server_name$request_uri? permanent;
location / {
index index.php index.html index.htm;
try_files $uri $uri/ /index.php?$args;
}

error_page   500 502 503 504  /50x.html;
location = /50x.html {
    root   /usr/share/nginx/html;
}

location ~* \.php$ {
    fastcgi_index   index.php;
    fastcgi_pass  unix:/var/run/php-fpm/php-fpm.sock;
    include         fastcgi_params;
    fastcgi_param   SCRIPT_FILENAME

$document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
}
}

server {
listen 23.226.226.175:443 ssl;
server_name cryptobells.com www.cryptobells.com;

ssl_certificate      /etc/ssl/certs/cryptobells.com.crt;
ssl_certificate_key  /etc/ssl/certs/cryptobells.com.key;

ssl_session_cache shared:SSL:1m;
ssl_session_timeout  5m;

ssl_ciphers

ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH;
ssl_prefer_server_ciphers on;

root /var/www/cryptobells;

ModSecurityEnabled on;
ModSecurityConfig /etc/modsecurity/modsecurity.conf;

location / {
    index  index.php index.html index.htm;
    try_files $uri $uri/ /index.php?$args;
}

error_page   500 502 503 504  /50x.html;
location = /50x.html {
    root   /usr/share/nginx/html;
}

location ~* \.php$ {
    fastcgi_index   index.php;
    fastcgi_pass  unix:/var/run/php-fpm/php-fpm.sock;
    include         fastcgi_params;
    fastcgi_param   SCRIPT_FILENAME

$document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
}

Please let me know if anyone is able to help identify what could be
causing segfaults, ro if there is any more information I can provide.
Thank you!

[robertpostill]

Hello!

On Sat, Apr 12, 2014 at 04:44:28PM -0700, Robert P. wrote:

Hello,

I have compiled nginx-1.5.13 with modsecurity-2.7.7 and am seeing
occasional segfaults when sending requests to the server. mod_security
was compiled as a standalone module per the instructions made available
at

Home · SpiderLabs/ModSecurity Wiki · GitHub.

The segfaults appear sporadic and do not seem to match up with any given
request. Below is my nginx configuration:

[…]

Also, a backtrace of the core dump:
(gdb) bt
#0 0x080a1827 in ngx_http_write_filter (r=0x83bb078, in=0x8baaa6c) at
src/http/ngx_http_write_filter_module.c:121

This points to the following code line:

    cl->buf = ln->buf;

That is, dereferencing ln->buf fails, which may only happen if the
buffer chain (“in” argument) is broken.

[…]

#8 0x080cfc78 in ngx_http_gunzip_body_filter (r=0x83bb078, in=0x8baaa6c)
at src/http/modules/ngx_http_gunzip_filter_module.c:184
#9 0x081146bd in ngx_http_modsecurity_body_filter (r=0x83bb078,
in=0xbf7ff8b4)
at
…/modsecurity-apache_2.7.7/nginx/modsecurity//ngx_http_modsecurity.c:1252
#10 0x08055381 in ngx_output_chain (ctx=0x8baa9b8, in=0xbf7ff8b4) at
src/core/ngx_output_chain.c:66

And this clearly shows that the buffer chain was chaned by
mod_security output body filter. Note “in” argument of
mod_security (“in=0xbf7ff8b4”) and gunzip filter which follows it
(“in=0x8baaa6c”).

That is, from the backtrace it looks like mod_security changed the
buffer chain and did it wrong, with a segfault as a result.

–
Maxim D.
http://nginx.org/

[robertpostill]

Hi Maxim!

Thank you for your response, always nice to actually hear back from
someone knowledgeable. Once thing I had noticed while looking at
backtraces (coredumps seem to indicate segfaults occurring in a number
of places, not just filter_module.c:121) was that every bt seemed to
include gzip modules as well. i disabled gzip in both my server and http
sections but this did not stop the segfaults (which is interesting, but
not entirely surprising, given that even when I had set SecRuleEngine to
off in my modsecurity.conf file, segfaults still occured… so even the
mere presence of ModSecurityEnabled in the nginx configuration was
leading to a break).

I have since recompiled nginx without both gunzip module and gzip static
module, and have not yet gotten any segfaults. i will continue to
research this and update if I have any more information

[robertpostill]

B.

[robertpostill]

Hello!

On Sun, Apr 13, 2014 at 08:42:04PM -0700, Robert P. wrote:

Hi Maxim!

Thank you for your response, always nice to actually hear back from
someone knowledgeable. Once thing I had noticed while looking at
backtraces (coredumps seem to indicate segfaults occurring in a number
of places, not just filter_module.c:121) was that every bt seemed to
include gzip modules as well. i disabled gzip in both my server and http
sections but this did not stop the segfaults (which is interesting, but

All filters, including gzip, are expected to be in backtraces, as
they are always called. Depending on configuration, they either
do something or not.

not entirely surprising, given that even when I had set SecRuleEngine to
off in my modsecurity.conf file, segfaults still occured… so even the
mere presence of ModSecurityEnabled in the nginx configuration was
leading to a break).

I have since recompiled nginx without both gunzip module and gzip static
module, and have not yet gotten any segfaults. i will continue to
research this and update if I have any more information

As per backtrace, it’s more or less obvious that the problem is in
modsecurity. Compiling out other modules may result in less
frequent segfaults, but it won’t fix the bug in modsecurity.

If you want to actually fix the problem, you should either
switch off / compile out modsecurity, or find and fix the bug in
it.

was compiled as a standalone module per the instructions made available
src/http/ngx_http_write_filter_module.c:121
#8 0x080cfc78 in ngx_http_gunzip_body_filter (r=0x83bb078, in=0x8baaa6c)
mod_security (“in=0xbf7ff8b4”) and gunzip filter which follows it
(“in=0x8baaa6c”).

That is, from the backtrace it looks like mod_security changed the
buffer chain and did it wrong, with a segfault as a result.

---

nginx mailing list
[email protected]
nginx Info Page

–
Maxim D.
http://nginx.org/

[robertpostill]

hi robert,

if you dont depend on mod_security’s advanced features like
output-filtering’n’stuff you might want to try naxsi

Home

its stable, its fast, rules are easy to create and understand and it
provides a set of basic features for a waf.

the community is responsive and open for feature-requests or bugreports.

regards,

mex

Posted at Nginx Forum:

OT / Re: nginx segfaulting with mod_security

[robertpostill]

I realized that I spoke too soon about gzip shortly after sending this.
My apologies for making a silly assumption; I am a sysadmin by trade and
not so skilled at developing or troubleshooting complex server software.
I’ve contact mod_security mailing list but haven’t heard back from them.
Thank you for your time and patience in answering my questions!

Sincerely,
Robert P.

[robertpostill]

Hi mex!

Thanks for the tip! I’ve known about naxsi for a while. I’m researching
various WAF options that will scale well for my Master’s thesis, and
mod_security + nginx interested me (my other research is pointing
towards using Varnish, for which several WAF solutions have already been
somewhat implemented). Naxsi doesn’t seem to offer the extensive logging
and detailed features like state tracking that mod_sec does, so I have
been wary to research further into it. But thanks for the suggestion!

Sincerely,
Robert P.