Has anyone else seen problems with Nginx with 1.3.11? Everything works perfectly fine with same compile options under 1.3.10... It's a pretty vanilla compile as far as modules... proxy, SSI, SSL and status modules. Using SPDY patch (using the correct patch for each version of Nginx) (and the correct SPDY module compile option on 1.3.11)... RIght right 1.3.11 starts being used, we get this stuff in the error log: *** Error in `nginx: worker process': malloc(): memory corruption: 0x00000000009084f0 *** 2013/01/14 14:34:16 [alert] 16599#0: *137 getsockname() failed (9: Bad file descriptor) while SPDY processing, client: 174.70.187.121, server: dpstatic.com, request: "GET /j/tinymce/tiny_mce.js?_v=bba17b4a HTTP/1.1", host: "x.dpstatic.com", referrer: "https://dev.digitalpoint.com/threads/anybody-has-e... 2013/01/14 14:34:16 [alert] 16599#0: *137 getsockname() failed (9: Bad file descriptor) while SPDY processing, client: 174.70.187.121, server: dpstatic.com, request: "GET /j/tinymce/tiny_mce.js?_v=bba17b4a HTTP/1.1", host: "x.dpstatic.com", referrer: "https://dev.digitalpoint.com/threads/anybody-has-e... *** Error in `nginx: worker process': malloc(): memory corruption: 0x000000000088cc00 *** Ended up going back to 1.3.10, and everything back to normal... Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235090#msg-235090
on 2013-01-14 23:50
on 2013-01-15 00:01
On Tuesday 15 January 2013 02:50:14 digitalpoint wrote: > 2013/01/14 14:34:16 [alert] 16599#0: *137 getsockname() failed (9: Bad file > at-answerserp-com-for-yahoo-backlinks.2624660/" *** Error in `nginx: worker > process': malloc(): memory corruption: 0x000000000088cc00 *** > > Ended up going back to 1.3.10, and everything back to normal... > Could you show nginx -V ? wbr, Valentin V. Bartenev -- http://nginx.com/support.html http://nginx.org/en/donation.html
on 2013-01-15 00:05
ya sorry, was just coming back to follow up with that... after I posted that, I was wondering why in the hell I was compiling with SSI module... it's actually WITHOUT SSI. :) This is back on 1.3.10 after I rolled it back after the issues: nginx version: nginx/1.3.10 built by gcc 4.5.1 20101208 [gcc-4_5-branch revision 167585] (SUSE Linux) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/usr/log/ngnix/error.log --http-log-path=/usr/log/ngnix/access.log --with-openssl=/home/software_source/openssl-1.0.1c --with-cc-opt='-I /usr/local/ssl/include' --with-ld-opt='-L /usr/local/ssl/lib' --without-http_proxy_module --without-http_ssi_module --with-http_ssl_module --with-http_stub_status_module The only difference with the compile options when I did it for 1.3.11 was I added "--with-http_spdy_module" to the end since it needs it for 1.3.11+ Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235094#msg-235094
on 2013-01-15 00:16
On Tuesday 15 January 2013 03:05:26 digitalpoint wrote: > --pid-path=/var/run/nginx.pid --error-log-path=/usr/log/ngnix/error.log > --http-log-path=/usr/log/ngnix/access.log > --with-openssl=/home/software_source/openssl-1.0.1c --with-cc-opt='-I > /usr/local/ssl/include' --with-ld-opt='-L /usr/local/ssl/lib' > --without-http_proxy_module --without-http_ssi_module > --with-http_ssl_module --with-http_stub_status_module > > The only difference with the compile options when I did it for 1.3.11 was I > added "--with-http_spdy_module" to the end since it needs it for 1.3.11+ > Ok, thank you. It seems I already figured out where is the problem. Please, try the new patch: http://nginx.org/patches/spdy/patch.spdy-56_1.3.11.txt wbr, Valentin V. Bartenev -- http://nginx.com/support.html http://nginx.org/en/donation.html
on 2013-01-15 00:47
Nope... still same issue. nginx version: nginx/1.3.11 built by gcc 4.5.1 20101208 [gcc-4_5-branch revision 167585] (SUSE Linux) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/usr/log/ngnix/error.log --http-log-path=/usr/log/ngnix/access.log --with-openssl=/home/software_source/openssl-1.0.1c --with-cc-opt='-I /usr/local/ssl/include' --with-ld-opt='-L /usr/local/ssl/lib' --without-http_proxy_module --without-http_ssi_module --with-http_ssl_module --with-http_stub_status_module --with-http_spdy_module ---- End of error.log: *** Error in `nginx: worker process': malloc(): memory corruption: 0x00000000009947f0 *** Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235098#msg-235098
on 2013-01-15 01:07
On Tuesday 15 January 2013 03:46:43 digitalpoint wrote: > --without-http_proxy_module --without-http_ssi_module > --with-http_ssl_module --with-http_stub_status_module > --with-http_spdy_module > > ---- > End of error.log: > > *** Error in `nginx: worker process': malloc(): memory corruption: > 0x00000000009947f0 *** > Thank you for testing. Could you create a debug log for the issue? See this link for instructions: http://nginx.org/en/docs/debugging_log.html wbr, Valentin V. Bartenev -- http://nginx.com/support.html http://nginx.org/en/donation.html
on 2013-01-15 01:22
Here ya go: http://www.shawnhogan.com/error.log.gz Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235100#msg-235100
on 2013-01-15 02:23
On Tuesday 15 January 2013 04:21:43 digitalpoint wrote: > Here ya go: > > http://www.shawnhogan.com/error.log.gz > Thanks a lot. It was really helpful. I believe the problem is fixed now: http://nginx.org/patches/spdy/patch.spdy-57_1.3.11.txt wbr, Valentin V. Bartenev -- http://nginx.com/support.html http://nginx.org/en/donation.html
on 2013-01-15 02:56
Well... the underlying errors went away, but it seems the new SPDY patch broke being able to handle multiple hosts on the same SPDY connection now (it worked under 1.3.10 just fine). For example, we have a SSL cert for both digitalpoint.com and dpstatic.com (dpstatic.com is a cookieless domain for serving static content), so SPDY attempts to use the same connection for multiple hosts. See SPDY session list here: http://f.cl.ly/items/0T1u3g0h0e1A0D1g2N0s/Image%20... With the SPDY patch for 1.3.11, now requests to *.dpstatic.com are *actually* being sent to digitalpoint.com (and getting a file not found). So somehow during a SPDY connection, the host for an individual request is being ignored somewhere along the way. Top browser is Chrome (SPDY connection), bottom browser is Safari (no SPDY support)... the end result is a SPDY connection will yield different results vs the "traditional" SSL connection: http://f.cl.ly/items/3K1Q2N1I3B000c0b0614/Image%20... Again, this worked as expected (ability for SPDY to properly share a connection across multiple hosts) with 1.3.10. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235105#msg-235105
on 2013-01-15 04:18
On Tuesday 15 January 2013 05:55:43 digitalpoint wrote: > > > Again, this worked as expected (ability for SPDY to properly share a > connection across multiple hosts) with 1.3.10. > There is no difference between 1.3.10 and 1.3.11 in terms of SPDY. In fact, 1.3.10 has serious bugs (see: http://nginx.org/en/CHANGES), and you should use 1.3.11 instead. The big difference is between spdy54 and spdy55+ patches. A large part of SPDY implementation was rewritten in spdy55, and also some relevant parts of nginx got new code. One of those changes makes nginx more RFC 6066 compliant. Here is some quotes: 3. Server Name Indication [...] If an application negotiates a server name using an application protocol and then upgrades to TLS, and if a server_name extension is sent, then the extension SHOULD contain the same name that was negotiated in the application protocol. If the server_name is established in the TLS session handshake, the client SHOULD NOT attempt to request a different server name at the application layer. 11.1. Security Considerations for server_name [...] Since it is possible for a client to present a different server_name in the application protocol, application server implementations that rely upon these names being the same MUST check to make sure the client did not present a different name in the application protocol. from http://tools.ietf.org/html/rfc6066 And you will not find in SPDY draft.2 specification any information about the "ability for SPDY to properly share a connection across multiple hosts": http://dev.chromium.org/spdy/spdy-protocol/spdy-pr... Apparently by making TLS SNI in nginx more RFC-compliant I unintentionally broke SPDY. Well, it's safe to use spdy54 with 1.3.11: http://nginx.org/patches/spdy/patch.spdy-54.txt and I recommend you to use it while I will think about a solution. Thanks again for testing. I hope to fix the issue soon. wbr, Valentin V. Bartenev -- http://nginx.com/support.html http://nginx.org/en/donation.html
on 2013-01-15 04:51
Yeah... the problem is that while it might not be part of the SPDY 2 draft to share connections across multiple hosts, Chrome most certainly is doing it (and probably other browsers) as you can see from the previous screenshot. Either way, you guys are doing a crazy awesome job... keep it up. :) Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235107#msg-235107
on 2013-01-27 16:22
On Tuesday 15 January 2013 07:50:30 digitalpoint wrote: > Yeah... the problem is that while it might not be part of the SPDY 2 draft > to share connections across multiple hosts, Chrome most certainly is doing > it (and probably other browsers) as you can see from the previous > screenshot. > > Either way, you guys are doing a crazy awesome job... keep it up. :) > Please, try the new patch: http://nginx.org/patches/spdy/patch.spdy-59_1.3.11.txt The problem should be fixed now. wbr, Valentin V. Bartenev -- http://nginx.com/support.html http://nginx.org/en/donation.html
on 2013-01-27 23:01
So far so good... seems to be working fine. :) Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235624#msg-235624
on 2013-01-28 20:21
Valentin V. Bartenev Wrote: ------------------------------------------------------- > Please, try the new patch: > http://nginx.org/patches/spdy/patch.spdy-59_1.3.11.txt > > The problem should be fixed now. > > wbr, Valentin V. Bartenev Is there a possibility the patch introduced an issue where connections don't expire (like ever)? Our load balancer in front of web server shows 1,511 connections, Nginx is reporting 10,810 connections, and the number of connections as reported by Nginx is just growing and growing and does not even remotely coincide with actual traffic/users/connections. This only seemed to start after the patch. http://f.cl.ly/items/0u2E2U0P3L280l3X033s/Image%20... Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235671#msg-235671
on 2013-01-28 20:35
On Monday 28 January 2013 23:20:27 digitalpoint wrote: > Is there a possibility the patch introduced an issue where connections > don't expire (like ever)? > > Our load balancer in front of web server shows 1,511 connections, Nginx is > reporting 10,810 connections, and the number of connections as reported by > Nginx is just growing and growing and does not even remotely coincide with > actual traffic/users/connections. This only seemed to start after the > patch. Do you mean numbers from stub_status? Then please check the error log. It may also indicate periodically crashing worker processes. wbr, Valentin V. Bartenev -- http://nginx.com/support.html http://nginx.org/en/donation.html
on 2013-01-28 21:33
Valentin V. Bartenev Wrote: ------------------------------------------------------- > > > > actual traffic/users/connections. This only seemed to start after > http://nginx.org/en/donation.html > > _______________________________________________ > http://mailman.nginx.org/mailman/listinfo/nginx That does look like it's the case... 2013/01/28 00:38:19 [alert] 30235#0: worker process 12408 exited on signal 11 2013/01/28 02:00:24 [alert] 30235#0: worker process 15737 exited on signal 11 2013/01/28 02:24:31 [alert] 30235#0: worker process 14897 exited on signal 11 2013/01/28 02:25:19 [alert] 30235#0: worker process 22628 exited on signal 11 2013/01/28 03:03:59 [alert] 30235#0: worker process 23528 exited on signal 11 2013/01/28 03:17:58 [alert] 30235#0: worker process 7916 exited on signal 11 2013/01/28 03:37:03 [alert] 30235#0: worker process 24767 exited on signal 11 2013/01/28 04:02:07 [alert] 30235#0: worker process 23483 exited on signal 11 2013/01/28 04:27:26 [alert] 30235#0: worker process 25164 exited on signal 11 2013/01/28 05:41:34 [alert] 30235#0: worker process 19980 exited on signal 11 2013/01/28 05:47:18 [alert] 30235#0: worker process 26566 exited on signal 11 2013/01/28 08:39:35 [alert] 30235#0: worker process 30184 exited on signal 11 2013/01/28 08:46:40 [alert] 30235#0: worker process 3482 exited on signal 11 2013/01/28 09:42:16 [alert] 30235#0: worker process 27395 exited on signal 11 2013/01/28 10:14:01 [alert] 30235#0: worker process 5439 exited on signal 11 2013/01/28 10:28:36 [alert] 30235#0: worker process 29917 exited on signal 11 2013/01/28 11:17:04 [alert] 30235#0: worker process 6869 exited on signal 11 I guess I should toll back to the old SPDY patch... :) Posted at Nginx Forum: http://forum.nginx.org/read.php?2,235090,235680#msg-235680
Enable email notification
Enable multi-page view
Please log in before posting. Registration is free and takes only a minute.
Existing account
(Switch to SSL-encrypted connection)
NEW: Do you have a Google/GoogleMail or Yahoo account? No registration required!
Log in with Google account | Log in with Yahoo account
Log in with Google account | Log in with Yahoo account
No account? Register here.