Nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Server NGINX
#1

Dear all

Fascinated by nginx, I attempted to integrate it with modsecurity.

Unfortunately, ever when modsecurity is enabled, nginx reports a
sefault in sysmessages.

Searching the web did not reveal any solution, i.e. I switched off
SecAudit* and even started modsecurity without rules – it continued
crashing.

Thank you for any hint on solving this issue.

Please find next information related to my setup including some logs.

wbr, Lukas

==

My current setup:

Platform: Linux/4.3.3 running on Debian/wheezy

nginx: self-compiled from sources according to
https://blog.stickleback.dk/nginx-modsec-on-ubuntu-14-04-lts/

modsecurity: installed and configured according to
https://www.howtoforge.com/tutorial/install-nginx-with-mod_security-on-ubuntu-15-04/

Relevant Logs:

$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.9.9
built by gcc 4.7.2 (Debian 4.7.2-5)
built with OpenSSL 1.0.1e 11 Feb 2013
TLS SNI support enabled
configure arguments: --user=www-data --group=www-data --with-pcre-jit
–with-ipv6 --with-http_ssl_module
–add-module=…/modsecurity-2.9.0/nginx/modsecurity
–conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log

$ tail error.log
2016/01/10 13:13:34 [notice] 10256#0: ModSecurity: LIBXML compiled
version=“2.8.0”
2016/01/10 13:13:34 [notice] 10256#0: ModSecurity: Status engine is
currently disabled, enable it by set SecStatusEngine to On.
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity for nginx
(STABLE)/2.9.0 (http://www.modsecurity.org/) configured.
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: APR compiled
version=“1.4.6”; loaded version=“1.4.6”
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: PCRE compiled
version="8.30 "; loaded version=“8.30 2012-02-04”
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: LIBXML compiled
version=“2.8.0”
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: Status engine is
currently disabled, enable it by set SecStatusEngine to On.
2016/01/10 13:13:38 [alert] 10261#0: worker process 10267 exited on
signal 11
2016/01/10 13:13:38 [alert] 10261#0: worker process 10264 exited on
signal 11
2016/01/10 13:13:38 [alert] 10261#0: worker process 10265 exited on
signal 11

$ dmesg
[605432.202671] nginx[10267]: segfault at 70 ip 08093ba1 sp bfc9a7c0
error 4 in nginx[8048000+123000]
[605432.385414] nginx[10264]: segfault at 70 ip 08093ba1 sp bfc9a7c0
error 4 in nginx[8048000+123000]
[605432.409089] nginx[10265]: segfault at 70 ip 08093ba1 sp bfc9a7c0
error 4 in nginx[8048000+123000]

–
Lukas Ruf http://www.lpr.ch | Ad Personam
Consecom http://www.consecom.com | Ad Laborem

#2

SecAudit* and even started modsecurity without rules – it continued
crashing.

Thank you for any hint on solving this issue.

Please find next information related to my setup including some logs.

By chance, I tried to get this to work just yesterday and also got only
SIGSEGV from it.
(nginx 1.8, FreeBSD 10.1-amd64, ap22-mod_security-2.9.0, all from my own
repository)

I found this:

https://github.com/SpiderLabs/ModSecurity/issues/839

So, you need to set

proxy_force_ranges on;

in the location you want to protect with mod_security.

It didn’t segfault any more after this - but I haven’t had time to check
how well it actually works.

Rainer

#3

Hi Lukas,

You may want to use the ModSecurity’s nginx_refactoring branch instead
of
the master branch. Here is the link to the branch:

https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

Br.,
Felipe Z.
Lead dev for ModSecurity

#4

crashing.

So, you need to set

proxy_force_ranges on;

in the location you want to protect with mod_security.

It didn???t segfault any more after this - but I haven???t had time to check how
well it actually works.

Thanks for your hint, Rainer. Now it has not crahed anymore…

wbr
Lukas

#5

Hi Felipe

Felipe Z. [email protected] [2016-01-10 14:49]:

You may want to use the ModSecurity’s nginx_refactoring branch instead of
the master branch. Here is the link to the branch:

https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

Thanks for your hint.

I found that recommendation. Since I also read that it would not be
fully compatible with OWASP/CRS I have not given it a try.

What is the situation regrading OWASP/CRS?

wbr
Lukas

#6

Hi Lukas,

On Sun, Jan 10, 2016 at 11:05 AM Lukas [email protected] wrote:

I found that recommendation. Since I also read that it would not be
fully compatible with OWASP/CRS I have not given it a try.

What is the situation regrading OWASP/CRS?

Currently there are three different versions of ModSecurity for nginx:

  • Version 2.9.0: That is the last released version, I think that is the
    one
    that you are using.
  • nginx_refactoring: That version contains some fixes on the top of
    v2.9.0,
    but those fixes may lead to instabilities depending on your
    configuration.
  • ModSecurity-connector: That is something that still under development
    and
    we have some work to do, to be exactly:

https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20documentation
https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20features
https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20operators
https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20transformation
https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20variables

Only use the ModSecurity-connector if you understands well the
ModSecurity
rules and the consequences of the missing pieces.

Further information about libModSecurity can be found here:
http://blog.zimmerle.org/2016/01/an-overview-of-upcoming-libmodsecurity.html
or:
https://www.trustwave.com/Resources/SpiderLabs-Blog/An-Overview-of-the-Upcoming-libModSecurity/

Br.,
Felipe.

#7

Hi Felipe

Currently there are three different versions of ModSecurity for nginx:

https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20operators

Thanks for pointing this out.

What worries me a “little bit” is that nginx started crashing with an
Out-of-Memory Exception when ModSecurity 2.9.0 with OWASP/CRS was
activated.

Have others experienced similar problems?

Isn’t there at least a run-time control in nginx that kills
subprocesses like ModSecurity as soon as they start overconsuming
resources/execution time?

Thanks.

wbr
Lukas

#8

Hi Robert

Robert P. [email protected] [2016-01-20 00:14]:

ModSecurity isn’t a sub-process, it’s compiled into the nginx binary and
runs as part of the worker process(es). Nginx doesn’t have a concept of
spawning children in the manner you’re referencing, so there’s nothing to
be monitored wrt. resource consumption. Any resource monitoring would be
done by the kernel, and the target would be nginx itself.

Thanks for clarifying.

If you’re running into an OOM condition with the nginx worker process, it
sounds like a leak within one of the modules (possible, but not definitely,
ModSecurity, if it only happens when you load the OWASP CRS).

I have not had the time to test different variants yet. The
proxy-setup, however, works perfectly fine with “ModSecurityEnabled
off;” but crashes otherwise.

My current config:

server {
listen 443 ssl;
listen [::]:443 ssl;

root /var/www/html;

index index.html index.htm index.nginx-debian.html;

server_name foobar;

ssl on;
ssl_certificate     crt.stack.pem;
ssl_certificate_key key.pem;

ssl_session_timeout 5m;
ssl_session_cache  builtin:1000  shared:SSL:10m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers

HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;

location / {

  ModSecurityEnabled off;
  ModSecurityConfig modsecurity/modsecurity_crs_10_setup.conf;

  proxy_force_ranges on;
  proxy_set_header Host                $host;
  proxy_set_header X-Real-IP           $remote_addr;
  proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto   $scheme;

  proxy_pass      http://ip.ad.dr.ess:80;
  proxy_redirect  http://ip.ad.dr.ess:80 https://$host$request_uri;

  client_max_body_size       10m;
  client_body_buffer_size    128k;

  client_body_temp_path      /var/cache/nginx/client_body_temp;

  proxy_connect_timeout      90;
  proxy_send_timeout         90;
  proxy_read_timeout         90;

  proxy_buffer_size          4k;
  proxy_buffers              4 32k;
  proxy_busy_buffers_size    64k;
  proxy_temp_file_write_size 64k;

  proxy_temp_path            /var/cache/nginx/proxy_temp;

}

location ~ /\.ht {
  deny all;
}

access_log  /var/log/nginx/access.log upstreamlog;
error_log   /var/log/nginx/error.log debug;

}

Thanks.

wbr.
Lukas

#9

ModSecurity isn’t a sub-process, it’s compiled into the nginx binary and
runs as part of the worker process(es). Nginx doesn’t have a concept of
spawning children in the manner you’re referencing, so there’s nothing
to
be monitored wrt. resource consumption. Any resource monitoring would be
done by the kernel, and the target would be nginx itself.

If you’re running into an OOM condition with the nginx worker process,
it
sounds like a leak within one of the modules (possible, but not
definitely,
ModSecurity, if it only happens when you load the OWASP CRS).

#10

Dear all

Lukas [email protected] [2016-01-10 14:39]:

Fascinated by nginx, I attempted to integrate it with modsecurity.

Unfortunately, ever when modsecurity is enabled, nginx reports a
sefault in sysmessages.

I tried debugging the issue a bit further (from a user perspective)
with common web-page and CalDAV with the following results:

  • nginx with modsecurity switched off works perfectly as a proxy nginx
  • nginx with modsecurity switched on with one owasp rule-set
    (modsecurity_crs_20_protocol_violations.conf) works for common
    web-pages with multi-media content (quick test without any errors
    reported)
  • nginx with modsecurity switched on with one owasp rule-set
    (modsecurity_crs_20_protocol_violations.conf) does not work for
    CalDAV.
    error.log: 2016/01/23 01:19:07 [emerg] 4844#0: *7 posix_memalign(16,
    4096) failed (12: Cannot allocate memory) while logging request
  • nginx with modsecurity switched on without any ruleset
    does not work for CalDAV – same error
  • nginx with modsecurity switched off without any ruleset
    does work for CalDAV perfectly.

With modsecurity switched on, an Out-of-Memory exception took place
always reporting:

[876715.533926] nginx invoked oom-killer: gfp_mask=0x280da, order=0,
oom_score_adj=0
[876715.533930] nginx cpuset=/ mems_allowed=0
[876715.533936] CPU: 0 PID: 4844 Comm: nginx Not tainted
4.3.3-consecom-ag #1
[876715.533937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS debian/1.7.5-1-0-g506b58d-dirty-20140812_231322-gandalf 04/01/2014
[876715.533939] f5a53ed0 d52542a6 f5a6b7c0 d5110792 d55a6db0 f5a6bab4
000280da 00000000
[876715.533943] 00000000 ffffffff 0d3f1361 00031d5e f4929cb8 00200282
f4929cb8 f4929cb0
[876715.533946] d50babb7 00200206 d525956e 00000002 00000002 f5020840
f5020bc4 d55a5702
[876715.533949] Call Trace:
[876715.533955] [] ? dump_stack+0x3e/0x58
[876715.533959] [] ? dump_header.isra.8+0x65/0x1be
[876715.533963] [] ? delayacct_end+0x47/0xa0
[876715.533967] [] ? ___ratelimit+0x7e/0xe0
[876715.533970] [] ? oom_kill_process+0x1d9/0x380
[876715.533973] [] ? security_capable_noaudit+0x3a/0x60
[876715.533977] [] ? has_ns_capability_noaudit+0xb/0x20
[876715.533979] [] ? oom_badness+0x96/0x100
[876715.533981] [] ? out_of_memory+0x252/0x320
[876715.533984] [] ? __alloc_pages_nodemask+0x77e/0x7a0
[876715.533989] [] ? handle_mm_fault+0xd54/0xf50
[876715.533990] [] ? vma_merge+0x1bf/0x280
[876715.533992] [] ? do_brk+0x1ca/0x2b0
[876715.533995] [] ? __do_page_fault+0x137/0x3a0
[876715.533998] [] ? vmalloc_sync_all+0x130/0x130
[876715.534001] [] ? error_code+0x5a/0x60
[876715.534003] [] ? vmalloc_sync_all+0x130/0x130
[876715.534004] Mem-Info:
[876715.534008] active_anon:543864 inactive_anon:208884 isolated_anon:0
[876715.534008] active_file:54 inactive_file:77 isolated_file:0
[876715.534008] unevictable:0 dirty:1 writeback:0 unstable:0
[876715.534008] slab_reclaimable:326 slab_unreclaimable:997
[876715.534008] mapped:88 shmem:4 pagetables:957 bounce:0
[876715.534008] free:21502 free_pcp:289 free_cma:0
[876715.534014] DMA free:12152kB min:64kB low:80kB high:96kB
active_anon:1676kB inactive_anon:1928kB active_file:8kB
inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15992kB managed:15916kB mlocked:0kB dirty:0kB writeback:0kB
mapped:8kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:76kB
kernel_stack:8kB pagetables:20kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:120
all_unreclaimable? yes
[876715.534016] lowmem_reserve[]: 0 839 3023 3023
[876715.534021] Normal free:73380kB min:3528kB low:4408kB high:5292kB
active_anon:386788kB inactive_anon:386844kB active_file:208kB
inactive_file:276kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:892920kB managed:859928kB mlocked:0kB
dirty:4kB writeback:0kB mapped:324kB shmem:0kB slab_reclaimable:1288kB
slab_unreclaimable:3912kB kernel_stack:672kB pagetables:3808kB
unstable:0kB bounce:0kB free_pcp:564kB local_pcp:564kB free_cma:0kB
writeback_tmp:0kB pages_scanned:115004 all_unreclaimable? yes
[876715.534022] lowmem_reserve[]: 0 0 17471 17471
[876715.534027] HighMem free:476kB min:512kB low:2808kB high:5104kB
active_anon:1786992kB inactive_anon:446764kB active_file:0kB
inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:2236296kB managed:2236296kB mlocked:0kB dirty:0kB writeback:0kB
mapped:20kB shmem:16kB slab_reclaimable:0kB slab_unreclaimable:0kB
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:592kB
local_pcp:592kB free_cma:0kB writeback_tmp:0kB pages_scanned:7836
all_unreclaimable? yes
[876715.534028] lowmem_reserve[]: 0 0 0 0
[876715.534030] DMA: 44kB (E) 78kB (UE) 516kB (UEM) 332kB (U) 264kB
(EM) 2128kB (EM) 3256kB (UEM) 1512kB (E) 21024kB (UE) 22048kB (UE)
14096kB (M) = 12152kB
[876715.534039] Normal: 1494kB (UEM) 1088kB (UEM) 6316kB (UE) 3232kB
(UEM) 1064kB (UE) 11128kB (UEM) 5256kB (UE) 2512kB (EM) 21024kB
(UM) 32048kB (UEM) 144096kB (M) = 73380kB
[876715.534047] HighMem: 14kB (U) 18kB (U) 116kB (M) 232kB (UM)
064kB 1128kB (M) 1256kB (M) 0512kB 01024kB 02048kB 0*4096kB =
476kB
[876715.534054] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=4096kB

Thanks for any hints

Lukas

–
Lukas Ruf http://www.lpr.ch | Ad Personam
Consecom http://www.consecom.com | Ad Laborem

#11

The modsec devel team is working hard on the new libmodsecurity. You may
just be better off waiting for them to put the finishing touches on that
project. Nginx + modsec 2.9 likely will get no dev attention moving
forward, given that the whole system is being revamped now.

Sent from my iPhone