# Nginx with custom modules crashes in gzip crc32()

**URL:** <https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891>\
**Category:** NGINX\
**Created:** [October 17, 2011, 6:27am UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891 "2011-10-17T06:27:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [October 17, 2011, 6:27am UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891/1 "2011-10-17T06:27:48Z")

</div>

I have nginx with my custom module, that rewrite content on some  
conditions. Without it everything works fine, but after enabling it,  
nginx start to crash approx. every 2 hours ( \> 100 req./sec) coredump  
shows that it crahses in gzip module:

Core was generated by `nginx: worker process '.  
Program terminated with signal 11, Segmentation fault.  
#0 0x000000343ea0286d in crc32 () from /usr/lib64/libz.so.1  
(gdb) bt  
#0 0x000000343ea0286d in crc32 () from /usr/lib64/libz.so.1  
#1 0x0000000000450f3f in ngx\_http\_gzip\_filter\_add\_data (r=0x601f380,  
in=0x29ac280) at src/http/modules/ngx\_http\_gzip\_filter\_module.c:708  
#2 ngx\_http\_gzip\_body\_filter (r=0x601f380, in=0x29ac280) at  
src/http/modules/ngx\_http\_gzip\_filter\_module.c:394  
#3 0x0000000000451ac5 in ngx\_http\_postpone\_filter (r=0x601f380,  
in=0x29ac280) at src/http/ngx\_http\_postpone\_filter\_module.c:82  
#4 0x00000000004521b1 in ngx\_http\_ssi\_body\_filter (r=0x343ea0c9c0,  
in=0x7906442d) at src/http/modules/ngx\_http\_ssi\_filter\_module.c:392  
#5 0x00000000004564b5 in ngx\_http\_charset\_body\_filter (r=0x343ea0c9c0,  
in=0x61c0ff8) at src/http/modules/ngx\_http\_charset\_filter\_module.c:552  
#6 0x0000000000457a5c in ngx\_http\_sub\_body\_filter (r=0x343ea0c9c0,  
in=0x29ac280) at src/http/modules/ngx\_http\_sub\_filter\_module.c:188  
#7 0x0000000000470e8f in ngx\_http\_af\_filter (r=0x601f380, in=0x29ac280)  
at /usr/src/redhat/SOURCES/af-headers/ngx\_af\_headers\_module.c:768  
#8 0x00000000004793e6 in clweb\_c\_body\_filter (r=0x601f380,  
in=0x29ac280) at  
/usr/src/redhat/SOURCES/content-parser-module/ngx\_mod\_content\_parser.c:510  
#9 0x0000000000479d28 in ngx\_http\_gunzip\_body\_filter (r=0x601f380,  
in=0x29ac280) at  
/usr/src/redhat/SOURCES/gunzip/ngx\_http\_gunzip\_filter\_module.c:323  
#10 0x000000000047ca5d in ngx\_subr\_body\_filter (r=0x601f380,  
in=0x1f5f830) at  
/usr/src/redhat/SOURCES/ngx\_subr\_module/ngx\_subr\_module.c:219  
#11 0x000000000047d46d in ngx\_subr\_body\_filter (r=0x601f380,  
in=0x1f5f830) at  
/usr/src/redhat/SOURCES/ngx\_subr\_all\_module/ngx\_subr\_all\_module.c:219  
#12 0x000000000040ba99 in ngx\_output\_chain (ctx=0x5e958c0, in=0x61c0ff8)  
at src/core/ngx\_output\_chain.c:65  
#13 0x000000000043bbf7 in ngx\_http\_copy\_filter (r=0x601f380,  
in=0x1f5f830) at src/http/ngx\_http\_copy\_filter\_module.c:141  
#14 0x000000000044bdd1 in ngx\_http\_range\_body\_filter (r=0x343ea0c9c0,  
in=0x61c0ff8) at src/http/modules/ngx\_http\_range\_filter\_module.c:551  
#15 0x000000000042ed92 in ngx\_http\_output\_filter (r=0x601f380,  
in=0x1f5f830) at src/http/ngx\_http\_core\_module.c:1868  
#16 0x00000000004463fa in ngx\_http\_upstream\_process\_non\_buffered\_request  
(r=0x601f380, do\_write=) at  
src/http/ngx\_http\_upstream.c:2381  
#17 0x00000000004468fc in  
ngx\_http\_upstream\_process\_non\_buffered\_upstream (r=0x601f380,  
u=0x6096858) at src/http/ngx\_http\_upstream.c:2352  
#18 0x00000000004457f6 in ngx\_http\_upstream\_handler (ev=0x148e568) at  
src/http/ngx\_http\_upstream.c:917  
#19 0x00000000004269be in ngx\_epoll\_process\_events (cycle=0x796480,  
timer=, flags=) at  
src/event/modules/ngx\_epoll\_module.c:635  
#20 0x000000000041e3c8 in ngx\_process\_events\_and\_timers (cycle=0x796480)  
at src/event/ngx\_event.c:245  
#21 0x0000000000425203 in ngx\_worker\_process\_cycle (cycle=0x796480,  
data=) at src/os/unix/ngx\_process\_cycle.c:800  
#22 0x0000000000423967 in ngx\_spawn\_process (cycle=0x796480,  
proc=0x425118 \<ngx\_worker\_process\_cycle\>, data=0x0, name=0x4d01ce  
“worker process”, respawn=-3) at src/os/unix/ngx\_process.c:196  
#23 0x000000000042478c in ngx\_start\_worker\_processes (cycle=0x796480,  
n=12, type=-3) at src/os/unix/ngx\_process\_cycle.c:360  
#24 0x0000000000425964 in ngx\_master\_process\_cycle (cycle=0x796480) at  
src/os/unix/ngx\_process\_cycle.c:136  
#25 0x0000000000408dba in main (argc=22, argv=0x795060) at  
src/core/nginx.c:405

dmesg says:  
nginx[20500]: segfault at 61c1000 ip 000000343ea0286d sp  
00007fff348eec78 error 4 in libz.so.1.2.3[343ea00000+14000]

so libz is going to read smth at bad address.  
This is definetly not nginx bug, but it is not simple to debug nginx, so  
I will be happy for any advice, what can cause nginx to crash? May be  
its stack problems?

Server info: Centos linux 5.6, amd64, nginx 1.0.6, zlib-devel-1.2.3-3

Posted at Nginx Forum:

> **[nginx with custom modules crashes in gzip crc32()](https://forum.nginx.org/read.php?2,216804,216804#msg-216804)**
>
> I have nginx with my custom module, that rewrite content on some conditions. Without it everything works fine, but after enabling it, nginx start to crash approx. every 2 hours ( \> 100 req./sec) coredump shows that it crahses in gzip module: Core...

---

<div class="post-metadata">

**Author:** ![artemg](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/45deac/32.png) [@artemg](https://www.ruby-forum.com/u/artemg)\
**Post date:** [October 17, 2011, 10:27am UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891/2 "2011-10-17T10:27:50Z")

</div>

On Mon, Oct 17, 2011 at 12:27:19AM -0400, artemg wrote:

> #1 0x0000000000450f3f in ngx\_http\_gzip\_filter\_add\_data (r=0x601f380,  
> in=0x29ac280) at src/http/modules/ngx\_http\_sub\_filter\_module.c:188  
> /usr/src/redhat/SOURCES/ngx\_subr\_module/ngx\_subr\_module.c:219  
> in=0x1f5f830) at src/http/ngx\_http\_core\_module.c:1868  
> src/event/modules/ngx\_epoll\_module.c:635  
> src/os/unix/ngx\_process\_cycle.c:136  
> its stack problems?
> 
> Server info: Centos linux 5.6, amd64, nginx 1.0.6, zlib-devel-1.2.3-3

It seems your module errorneously overrides memory area that corresponds  
to ctx-\>zstream.next\_in or ctx-\>zstream.avail\_in values:

```
    ctx->crc32 = crc32(ctx->crc32, ctx->zstream.next_in,
                       ctx->zstream.avail_in);

```

–  
Igor S.

---

<div class="post-metadata">

**Author:** ![artemg](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/45deac/32.png) [@artemg](https://www.ruby-forum.com/u/artemg)\
**Post date:** [October 19, 2011, 2:20am UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891/3 "2011-10-19T02:20:54Z")

</div>

Thanks for the answer, but its difficult to understand how can that  
happen. I even compare output of modified (by my module) response and  
unmodified - they are the same, with the only difference that data in  
chains is filled in another way. I.e. input data lengths are 100-200-100  
bytes (.last - .pos) and after my content rewriting module they can be  
0-0-400, or even 0-0-0 chain, and data can be added in next chain. I am  
doing buffering to match some patterns. In fact of this I see in error  
log:

[alert] 31974#0: \*98657 zero size buf in writer t:1 r:1 f:0  
0000000001714FE0 0000000001F8D290-0000000001F8D290 0000000000000000 0-0  
while sending to client

Is that ok, to have zero size bufs, or I need to modify the chain, not  
to pass them forward to other modules?

Posted at Nginx Forum:

> **[Re: nginx with custom modules crashes in gzip crc32()](https://forum.nginx.org/read.php?2,216804,216883#msg-216883)**
>
> I have nginx with my custom module, that rewrite content on some conditions. Without it everything works fine, but after enabling it, nginx start to crash approx. every 2 hours ( \> 100 req./sec) coredump shows that it crahses in gzip module: Core...

---

<div class="post-metadata">

**Author:** ![artemg](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/45deac/32.png) [@artemg](https://www.ruby-forum.com/u/artemg)\
**Post date:** [October 19, 2011, 10:20am UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891/4 "2011-10-19T10:20:20Z")

</div>

agentzh, thanks for the answer, seems the problem is really with zero  
size bufs. I changed code to insert spaces, if buffer is empty and  
everything start to work with gzip disabled (before that nothing worked  
with gzip off), and I think there will be no crashes in gzip now if  
enabled. Now I will create my own chains to pass to downstream filters.

I didn’t try valgrind because usually it consumes a lot more cpu, and  
this is unacceptable on staging machine(there will be high cpu load, or  
I will need to decrease number of requests so I will wait for crash more  
time). But thanks for reminding about it.

Posted at Nginx Forum:

> **[Re: nginx with custom modules crashes in gzip crc32()](https://forum.nginx.org/read.php?2,216804,216893#msg-216893)**
>
> I have nginx with my custom module, that rewrite content on some conditions. Without it everything works fine, but after enabling it, nginx start to crash approx. every 2 hours ( \> 100 req./sec) coredump shows that it crahses in gzip module: Core...

---

<div class="post-metadata">

**Author:** ![artemg](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/45deac/32.png) [@artemg](https://www.ruby-forum.com/u/artemg)\
**Post date:** [October 19, 2011, 5:25am UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891/5 "2011-10-19T05:25:10Z")

</div>

On Wed, Oct 19, 2011 at 8:20 AM, artemg [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> 

Passing zero-size non-special bufs to the downstream output filters is  
surely BAD. You have to fix it 🙂

Also, using tools like valgrind’s memcheck to find memory issues in  
your modules is highly recommended and can often save you huge number  
of hours of debugging 🙂

Regards,  
-agentzh

---

<div class="post-metadata">

**Author:** ![artemg](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/45deac/32.png) [@artemg](https://www.ruby-forum.com/u/artemg)\
**Post date:** [October 20, 2011, 1:53pm UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891/6 "2011-10-20T13:53:23Z")

</div>

On Thu, Oct 20, 2011 at 7:17 PM, artemg [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> By the way passing zero size last\_buf is ok, as I understand? What do  
> you mean by “non-special bufs” ?

A buf with last\_buf set is “special”. Check out the ngx\_buf\_special  
macro definition in nginx core’s src/core/ngx\_buf.h:

```
#define ngx_buf_special(b)
      \
    ((b->flush || b->last_buf || b->sync)
      \
     && !ngx_buf_in_memory(b) && !b->in_file)

```

Regards,  
-agentzh

---

<div class="post-metadata">

**Author:** ![artemg](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/45deac/32.png) [@artemg](https://www.ruby-forum.com/u/artemg)\
**Post date:** [October 20, 2011, 1:17pm UTC](https://www.ruby-forum.com/t/nginx-with-custom-modules-crashes-in-gzip-crc32/212891/7 "2011-10-20T13:17:58Z")

</div>

By the way passing zero size last\_buf is ok, as I understand? What do  
you mean by “non-special bufs” ?

Posted at Nginx Forum:

> **[Re: nginx with custom modules crashes in gzip crc32()](https://forum.nginx.org/read.php?2,216804,216951#msg-216951)**
>
> I have nginx with my custom module, that rewrite content on some conditions. Without it everything works fine, but after enabling it, nginx start to crash approx. every 2 hours ( \> 100 req./sec) coredump shows that it crahses in gzip module: Core...