# Upstream prematurely closes cnx =\> 502 Bad Gateway to client **URL:** \ **Category:** NGINX\ **Created:** [March 3, 2016, 11:09am UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190 "2016-03-03T11:09:31Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Sadaf\_N](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/858c86/32.png) [@Sadaf\_N](https://www.ruby-forum.com/u/Sadaf_N)\ **Post date:** [March 3, 2016, 11:09am UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/1 "2016-03-03T11:09:31Z")
Nginx’ers, I trying to figure out why I’m randomly are seeing requests having issues with a nginx 1.7.4 when proxying to an upstream pool like: 2016/03/03 10:24:21 [error] 15905#0: \*3252814 upstream prematurely closed connection while reading response header from upstream, client: 10.45.69.25, server: , request: “POST / HTTP/1.1”, upstream: “[http://10.45.69.28:8081/](http://10.45.69.28:8081/)”, host: “mxosCl:8081” 2016/03/03 10:30:02 [error] 15905#0: \*3237632 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.45.69.99, server: , request: “POST / HTTP/1.1”, upstream: “[http://10.45.69.25:8081/](http://10.45.69.25:8081/)”, host: “mxosCl:8081” 2016/03/03 10:30:02 [error] 15905#0: \*3237670 upstream prematurely closed connection while reading response header from upstream, client: 10.45.69.99, server: , request: “POST / HTTP/1.1”, upstream: “[http://10.45.69.28:8081/](http://10.45.69.28:8081/)”, host: “mxosCl:8081” 2016/03/03 10:30:03 [error] 15905#0: \*3237670 upstream prematurely closed connection while reading response header from upstream, client: 10.45.69.99, server: , request: “DELETE / HTTP/1.1”, upstream: “[http://10.45.69.26:8081/](http://10.45.69.26:8081/)”, host: “mxosCl:8081” upstream servers doesn’t log any issues, it seems. clients get a 502 Bad Gateway for such requests. TIA /Steffen ---
**Author:** ![Nginx\_U](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/dec6dc/32.png) [@Nginx\_U](https://www.ruby-forum.com/u/Nginx_U)\ **Post date:** [March 3, 2016, 11:58am UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/2 "2016-03-03T11:58:00Z")
My config btw: user imail; worker\_processes auto; daemon on; master\_process on; error\_log logs/mos\_error.tcp debug\_tcp; error\_log logs/mos\_error.log; pid /opt/imail/nginx/logs/mos\_nginx.pid; worker\_rlimit\_nofile 200000; worker\_rlimit\_core 500M; working\_directory /opt/imail/nginx; events { worker\_connections 25000; use epoll; multi\_accept off; } http { log\_format testlogs '$remote\_addr - $remote\_user [$time\_local] ’ '“$request” $status $body\_bytes\_sent ’ '“$http\_referer” “$upstream\_addr” ’ ‘$request\_time $upstream\_response\_time’; access\_log logs/mos\_access.log; sendfile on; keepalive\_requests 500000; keepalive\_timeout 20s; tcp\_nopush on; tcp\_nodelay on; client\_body\_buffer\_size 128k; client\_body\_temp\_path /dev/shm/mos\_nginx/client\_temp 1 2; open\_file\_cache max=25000 inactive=30s; open\_file\_cache\_valid 60s; open\_file\_cache\_min\_uses 2; open\_file\_cache\_errors on; upstream mosgenericbackend { server mss1:8081; server mss2:8081; server mss3:8081; server mss4:8081; healthcheck\_enabled; healthcheck\_delay 1000; healthcheck\_timeout 1000; healthcheck\_failcount 2; healthcheck\_send ‘GET /mxos/monitor HTTP/1.0’; keepalive 300; } server { listen 8081; keepalive\_timeout 600s; client\_max\_body\_size 0; location /mxos/ { proxy\_http\_version 1.1; proxy\_set\_header Connection “”; proxy\_pass [http://mosgenericbackend](http://mosgenericbackend); proxy\_connect\_timeout 2; more\_set\_headers “Content-Type: application/json”; } location /mxos/health\_status { healthcheck\_status; } proxy\_connect\_timeout 60; proxy\_read\_timeout 30; proxy\_send\_timeout 60; } } My [vendor patched/supplied due to application specific load balancing+health check] nginx version: # /opt/imail/nginx/sbin/nginx -V nginx version: nginx/1.7.4 built by gcc 3.4.6 20060404 (Red Hat 3.4.6-3) TLS SNI support disabled configure arguments: --with-debug –add-module=/bld/current/emailmx90\_git\_blds/emailmx90\_contrib\_shared/contrib/nginx\_1.7.4/source/addons/healthcheck\_nginx\_upstreams-master –add-module=/bld/current/emailmx90\_git\_blds/emailmx90\_contrib\_shared/contrib/nginx\_1.7.4/source/addons/headers-more-nginx-module-0.23 –add-module=/bld/current/emailmx90\_git\_blds/emailmx90\_contrib\_shared/contrib/nginx\_1.7.4/source/addons/yaoweibin-nginx\_tcp\_proxy\_module-f2156ef Posted at Nginx Forum: [https://forum.nginx.org/read.php?2,265031,265032#msg-265032](https://forum.nginx.org/read.php?2,265031,265032#msg-265032) ---
**Author:** ![Nginx\_U](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/dec6dc/32.png) [@Nginx\_U](https://www.ruby-forum.com/u/Nginx_U)\ **Post date:** [March 3, 2016, 12:53pm UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/3 "2016-03-03T12:53:30Z")
The HTTP specification states every request shall receive a response. Your backend closes the connection while nginx is awaiting/reading the headers. ## The problem definitely comes from your backend. You could use tcpdump between nginx and your backend to record what they say to each other. ​Try to correlate logs from nginx with logs from your backend.​ _B. R._ ---
**Author:** ![Nginx\_U](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/dec6dc/32.png) [@Nginx\_U](https://www.ruby-forum.com/u/Nginx_U)\ **Post date:** [March 4, 2016, 8:13am UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/4 "2016-03-04T08:13:39Z")
@B.R. You’re right, seemed my upstream tomcat instances were RESETing cnx as reply something. So far I improved it a lot my altering a http connector keepAliveTimeout value from mistakenly expressed as sec when in fact it should be msec 😉 When heavy load it still occurs but far less frequently, will dig deeper into tomcat trimming/tunning Posted at Nginx Forum: > **[Re: upstream prematurely closes cnx =\> 502 Bad Gateway to client](https://forum.nginx.org/read.php?2,265031,265070#msg-265070)** > > Nginx’ers, I trying to figure out why I'm randomly are seeing requests having issues with a nginx 1.7.4 when proxying to an upstream pool like: 2016/03/03 10:24:21 \[error\] 15905#0: \*3252814 upstream prematurely closed connection while reading... ---
**Author:** ![Nginx\_U](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/dec6dc/32.png) [@Nginx\_U](https://www.ruby-forum.com/u/Nginx_U)\ **Post date:** [March 4, 2016, 11:01am UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/5 "2016-03-04T11:01:19Z")
stefws Wrote: ---
**Author:** ![Nginx\_U](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/dec6dc/32.png) [@Nginx\_U](https://www.ruby-forum.com/u/Nginx_U)\ **Post date:** [March 4, 2016, 11:00am UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/6 "2016-03-04T11:00:44Z")
Seems I’m not alone w/TC issues 😉 Posted at Nginx Forum: > **[Re: upstream prematurely closes cnx =\> 502 Bad Gateway to client](https://forum.nginx.org/read.php?2,265031,265078#msg-265078)** > > Nginx’ers, I trying to figure out why I'm randomly are seeing requests having issues with a nginx 1.7.4 when proxying to an upstream pool like: 2016/03/03 10:24:21 \[error\] 15905#0: \*3252814 upstream prematurely closed connection while reading... ---
**Author:** ![Nginx\_U](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/dec6dc/32.png) [@Nginx\_U](https://www.ruby-forum.com/u/Nginx_U)\ **Post date:** [March 4, 2016, 11:49am UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/7 "2016-03-04T11:49:13Z")
Hi. Am 04-03-2016 11:00, schrieb stefws: > ## stefws Wrote: > > > Seems I’m not alone w/TC issues 😉 > > missed the link: > > [http://permalink.gmane.org/gmane.comp.web.haproxy/26860](http://permalink.gmane.org/gmane.comp.web.haproxy/26860) Well Maybe you have a completely different situation. Is it possible to build a more recent nginx version without the additional modules? Please can you setup the tomcat as described in my answer. [http://permalink.gmane.org/gmane.comp.web.haproxy/26862](http://permalink.gmane.org/gmane.comp.web.haproxy/26862) #### Please can you try to run the connector with debug on. > **[Apache Tomcat 8 (8.0.53) - Logging in Tomcat](https://tomcat.apache.org/tomcat-8.0-doc/logging.html)** I would try to use this. org.apache.catalina.session.level=ALL org.apache.coyote.http11.Http11Protocol.level=ALL Pay attention this will produce a lot entries in the logs and could have some impact to the performance. The standard setup have also some low limits maybe you must increase this limits. > **[Standard Implementation - Apache Tomcat 8 Configuration Reference (8.0.53) - The...](https://tomcat.apache.org/tomcat-8.0-doc/config/http.html#Standard_Implementation)** > > The standard HTTP connectors (BIO, NIO, NIO2 and APR/native) all support the following attributes in addition to the common Connector attributes listed above. #### Maybe I have missed it but which tomcat version do you use? Please also setup nginx with debug option as described here > **[A debugging log](http://nginx.org/en/docs/debugging_log.html)** What’s the error line(s) on the tomcat site? Please can you also post the current ‘conf/server.xml’ with all ‘Connector\*’, thanks. BR Aleks ---
**Author:** ![Nginx\_U](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/dec6dc/32.png) [@Nginx\_U](https://www.ruby-forum.com/u/Nginx_U)\ **Post date:** [March 4, 2016, 2:38pm UTC](https://www.ruby-forum.com/t/upstream-prematurely-closes-cnx-502-bad-gateway-to-client/244190/8 "2016-03-04T14:38:38Z")
I’ve also seen the issue when running plain nginx 1.9.11 sort like builting a new nginx, only the issue is that upstream closes cnxs, not nginx. Since I’ve discovered that our TCs had the Connector’ keepAliveTimeout way low (10 msec), mistakenly thought the units were sec and not as actually msec. Since increasing this a 1000 fold everything seems much better now 🙂 ``` ``` Thanks, will look into your suggestions next time I need to debug TC, right now they are running at high throttle 😉 BTW I’m using Apache Tomcat/7.0.27 tcpdump snippet between nginx(10.45.69.14) and TC(10.45.69.25), where TC RESETs cnx instead of replying: 01:15:49.744283 IP 10.45.69.14.58702 \> 10.45.69.25.8081: Flags [P.], seq 31688:39753, ack 191, win 219, options [nop,nop,TS val 38708511 ecr 32578863], length 8065 /PUT / HTTP/1.1^M Host: mosgenericbackend^M Content-Length: 7811^M Content-Type: application/x-www-form-urlencoded^M User-Agent: Java/1.8.0\_66^M Accept: text/html, image/gif, image/jpeg, \*; q=.2, _/_; q=.2^M ^M 01:15:49.744311 IP 10.45.69.25.8081 \> 10.45.69.14.58702: Flags [R], seq 1376851868, win 0, length 0 01:15:49.744467 IP 10.45.69.14.58702 \> 10.45.69.25.8081: Flags [.], ack 192, win 219, options [nop,nop,TS val 38708512 ecr 32578874], length 0 01:15:49.744480 IP 10.45.69.25.8081 \> 10.45.69.14.58702: Flags [R], seq 1376851869, win 0, length 0 Posted at Nginx Forum: > **[Re: upstream prematurely closes cnx =\> 502 Bad Gateway to client](https://forum.nginx.org/read.php?2,265031,265090#msg-265090)** > > Nginx’ers, I trying to figure out why I'm randomly are seeing requests having issues with a nginx 1.7.4 when proxying to an upstream pool like: 2016/03/03 10:24:21 \[error\] 15905#0: \*3252814 upstream prematurely closed connection while reading...