Nginx double requests for my upstreams?

funkdoobiest · June 5, 2009, 2:10pm

Hi,

I have a strange behavior for my site load balanced with nginx.
I have 1 Loadbalancer with nginx and 3 servers with apache.

sometimes a request is doubled and send to 2 of my apache webservers.
this is a big problem for me, because this is sometimes a requests that
is doing a payement.
I checked this behavior with doing a log of all my payment transactions
and there a some transactions
for the same person double at the same second! (the user can’t do this
at the website, because I send the form which is doing the request with
JS and ask the User if he really wants to pay and so on)

Here are my config files for nginx, please help me:

#ningx.conf
user www-data;
worker_processes 4;

error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;

events {
worker_connections 512;
}

http {

include mime.types;
include proxy.conf;

access_log  logs/access.log  combined;

sendfile        on;
tcp_nopush     on;

keepalive_timeout  0;
#keepalive_timeout  15;

#gzip  on;

include upstream.conf;

include server_www.conf;

}

proxy.conf

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 32 4k;

#upstream.conf
upstream webseite {
ip_hash;
server 10.XXX.XX.1:8081;
server 10.XXX.XX.2:8081;
server 10.XXX.XX.3:8081;
}

#server_www.conf
server {
listen 80;
server_name www.website.com;

# serve static files

location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|html|htm|swf)$ {
root /var/www/www.website.com/;
expires 1d;
}

location / {
  proxy_pass http://webseite;
}

}

Thank you!

Posted at Nginx Forum:

funkdoobiest · June 5, 2009, 3:40pm

Hello!

On Fri, Jun 05, 2009 at 07:59:13AM -0400, funkdoobiest wrote:

Hi,

I have a strange behavior for my site load balanced with nginx.
I have 1 Loadbalancer with nginx and 3 servers with apache.

sometimes a request is doubled and send to 2 of my apache webservers.
this is a big problem for me, because this is sometimes a requests that is doing a payement.
I checked this behavior with doing a log of all my payment transactions and there a some transactions
for the same person double at the same second! (the user can’t do this at the website, because I send the form which is doing the request with JS and ask the User if he really wants to pay and so on)

http://wiki.nginx.org/NginxHttpProxyModule#proxy_next_upstream

Maxim D.

funkdoobiest · June 8, 2009, 2:44pm

Hi Maxim,

Maxim D. Wrote:

with apache.
send the form which is doing the request with JS
and ask the User if he really wants to pay and so
on)

Module ngx_http_proxy_module
ext_upstream

Maxim D.

Thank you for your help. I set proxy_next_upstream to off, but there are
still double requests

Posted at Nginx Forum:

funkdoobiest · June 8, 2009, 2:57pm

I think I noticed this behavior too. The upstream didn’t fail but it
tried multiple in parallel anyway.

On Jun 7, 2009, at 12:35 PM, “funkdoobiest” [email protected]

funkdoobiest · June 9, 2009, 5:53am

Hello,

Maxim D. Wrote:

servers
all

on)
double requests

It doesn’t looks like nginx problem. Try checking
logs (frontend
and backend ones) - probably your javascript code
just does
something wrong.

Maxim D.

I checked it about 100 times. I checked the logs also. It’s not my
script.
The requests are at the same second. If i disable the loadbalancer (in
this case nginx) double request are stopping.

Jan

Posted at Nginx Forum:

funkdoobiest · June 8, 2009, 4:07pm

Hello!

On Sun, Jun 07, 2009 at 03:35:41PM -0400, funkdoobiest wrote:

my payment transactions and there a some
Maxim D.

Thank you for your help. I set proxy_next_upstream to off, but there are still double requests

It doesn’t looks like nginx problem. Try checking logs (frontend
and backend ones) - probably your javascript code just does
something wrong.

Maxim D.

funkdoobiest · June 9, 2009, 7:28am

Hello!

On Mon, Jun 08, 2009 at 06:45:03AM -0400, funkdoobiest wrote:

is

send the form which is doing the request with
Maxim D.

Maxim D.

I checked it about 100 times. I checked the logs also. It’s not my script.
The requests are at the same second. If i disable the loadbalancer (in this case nginx) double request are stopping.

Try the following:

Post here nginx -V output to make it clear which version do you
use.
Post here results of your logs investigations to make it clear
that problem appears somewhere at nginx level.

This should include relevant nginx access logs (expected: one line
with client request; and some grep results to show there is no other
identical lines), relevant backend access logs (doubled request
should be visible here), relevant lines from nginx error_log if any.

Post here your nginx configs to make it clear how requests in
question are processed. There are more than one way to screw
things up, and nginx config does matter.
Try obtaining relevant debug log (some info may be found at
http://wiki.nginx.org/NginxDebugging), post it here.

Maxim D.

funkdoobiest · June 23, 2009, 4:30am

Hi Maxim,

I have checked the nginx-access logs and I found following:

84.72.240.XXX - - [03/Jun/2009:16:38:46 +0200] “POST /premium/paypremium
HTTP/1.1” 499 0 “http://www.host.com/premium/getpremium/” “Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)”
84.72.240.XXX - - [03/Jun/2009:16:38:48 +0200] “POST /premium/paypremium
HTTP/1.1” 200 2230 “http://www.host.com/premium/getpremium/”
“Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)”

What can I do or what do I wrong?

Thank you,
Jan

Posted at Nginx Forum:

funkdoobiest · June 23, 2009, 1:54pm

My nginx-version:

nginx version: nginx/0.7.59
built by gcc 4.3.2 (Debian 4.3.2-1.1)
configure arguments: --with-http_ssl_module --with-http_realip_module
–with-http_addition_module --with-http_sub_module
–with-http_stub_status_module --user=www-data --group=www-data

Posted at Nginx Forum:

funkdoobiest · June 23, 2009, 2:32pm

Hello!

On Mon, Jun 22, 2009 at 10:19:54PM -0400, funkdoobiest wrote:

Hi Maxim,

I have checked the nginx-access logs and I found following:

84.72.240.XXX - - [03/Jun/2009:16:38:46 +0200] “POST /premium/paypremium HTTP/1.1” 499 0 “http://www.host.com/premium/getpremium/” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”
84.72.240.XXX - - [03/Jun/2009:16:38:48 +0200] “POST /premium/paypremium HTTP/1.1” 200 2230 “http://www.host.com/premium/getpremium/” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”

What can I do or what do I wrong?

Error code 499 means that client closed connection before it got
response. In this particular case it looks like duplicate form
submit, e.g. due to second click on submit button.

Maxim D.

funkdoobiest · June 24, 2009, 9:46am

Maxim D. Wrote:

84.72.240.XXX - - [03/Jun/2009:16:38:46 +0200]

Maxim D.
I got this 499 error code for about 10% of all the connections. not just
for form-submits.
It seems that “proxy_ignore_client_abort on;” fix that problem. But why?
Is it ok to ignore client aborts?

Jan

Posted at Nginx Forum:

funkdoobiest · June 24, 2009, 10:08am

Hello!

On Tue, Jun 23, 2009 at 10:33:05AM -0400, funkdoobiest wrote:

following:
5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
button.

Maxim D.

I got this 499 error code for about 10% of all the connections. not just for form-submits.

In general 499 errors are normal, since clients tend to close
browser windows/tabs, cancel downloads or just navigate to another
page e.g. before loading all images etc. High percentage of this
errors may indicate that your site is too slow, but since actual
numbers depend heavily on site usage pattern it’s hard to say what
is “high”.

It seems that “proxy_ignore_client_abort on;” fix that problem. But why? Is it ok to ignore client aborts?

Directive proxy_ignore_client_abort instructs nginx to finish
request processing even if client closed connection. It’s safe
but usually it’s just waste of resources (since backend response
will be thrown away).

Maxim D.

funkdoobiest · July 1, 2009, 10:53am

Hi,

the problem of the doubled requests is still there

Here are the lines from the logs:

nginx-access-log:

93.210.31.XXX - - [28/Jun/2009:14:11:15 +0200] “GET / HTTP/1.1” 200 2480
“-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1;
.NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; .NET CLR
1.1.4322)”
87.242.40.XX - - [28/Jun/2009:14:11:16 +0200] “GET / HTTP/1.1” 200 2480
“DOMAIN.COM - Google Search”
“Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11”
72.30.79.XXX - - [28/Jun/2009:14:11:17 +0200] “GET
/index.php?option=com_content&view=category&id=3&Itemid=13&limitstart=1440
HTTP/1.0” 404 289 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp/3.0;
Why is Slurp crawling my page? | Search for Desktop Help - SLN22600)”
91.17.86.XX - - [28/Jun/2009:14:11:17 +0200] “POST
/werbung/ad/fullsize/subpart/1501 HTTP/1.1” 200 176
“http://www.DOMAIN.COM/live/show/1/” “Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR
1.1.4322)”
217.231.102.XXX - - [28/Jun/2009:14:11:19 +0200] “GET /live/ HTTP/1.1”
200 2959 “http://www.DOMAIN.COM” “Mozilla/4.0 (compatible; MSIE 5.5;
Windows 98; T-Online Internatinal AG)”

213.162.66.XXX - - [28/Jun/2009:14:11:20 +0200] “POST
/premium/paypremium HTTP/1.1” 400 0
“http://www.DOMAIN.COM/premium/getpremium/” “Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center
PC 5.0; .NET CLR 3.0.04506)”

87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] “GET / HTTP/1.1” 200 2480
“DOMAIN.COM - Google Search”
“Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11”
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] “GET /DOMAIN/css/style.css
HTTP/1.1” 200 18139 “http://www.DOMAIN.COM/” “Mozilla/5.0 (Windows; U;
Windows NT 5.1; hu; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11”
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] “GET
/DOMAIN/css/layout.css HTTP/1.1” 200 3798 “http://www.DOMAIN.COM/”
“Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11”
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] “GET
/DOMAIN/css/jScrollPane.css HTTP/1.1” 200 1736 “http://www.DOMAIN.COM/”
“Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11”
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] “GET
/DOMAIN/images/favicon.ico HTTP/1.1” 200 1150 “-” “Mozilla/5.0 (Windows;
U; Windows NT 5.1; hu; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11”
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] “GET
/DOMAIN/javascript/AC_RunActiveContent.js HTTP/1.1” 200 8321
“http://www.DOMAIN.COM/” “Mozilla/5.0 (Windows; U; Windows NT 5.1; hu;
rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11”
62.226.184.XXX - - [28/Jun/2009:14:11:21 +0200] “POST
/werbung/ad/fullsize/subpart/1504 HTTP/1.1” 200 176
“http://www.DOMAIN.COM/live/show/1/” “Mozilla/5.0 (Windows; U; Windows
NT 6.0; de; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11”
89.53.233.XXX - - [28/Jun/2009:14:11:21 +0200] “GET
/mediacenter/overview/25/ HTTP/1.1” 200 2875
“http://www.DOMAIN.COM/mediacenter/list/9/2008/” “Mozilla/4.0
(compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727;
Media Center PC 5.0; .NET CLR 3.0.04506)”
89.53.233.XXX - - [28/Jun/2009:14:11:22 +0200] “GET
/mediacenter/overview/25/DOMAIN/images/favicon.ico HTTP/1.1” 404 529 “-”
“Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET
CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)”

apache-access-log:

10.226.50.67 - - [28/Jun/2009:14:11:20 +0200] “POST /premium/paypremium
HTTP/1.0” 200 2330 “http://www.DOMAIN.com/premium/getpremium/”
“Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET
CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)”
10.226.50.67 - - [28/Jun/2009:14:11:20 +0200] “POST /premium/paypremium
HTTP/1.0” 200 2330 “http://www.DOMAIN.com/premium/getpremium/”
“Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET
CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)”

There is no entry in the error-logs for this requests.
So the problem is not JavaScript or Form-Error. I got double requests
for other requests to.

Thank you,
Jan

Posted at Nginx Forum:

funkdoobiest · July 13, 2010, 1:37pm

I also had the same mistakes of double request
I try to do proxy_ignore_client_abort onï¼Œhttp_499 fewer errors, but
appeared http_400 error.
the double request at the some time problem still was not solved

I have 1 Loadbalancer with nginx and 4 servers with apache.
apache 2.0.59

nginx version: nginx/0.7.65
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
configure arguments: --user=www --group=www --prefix=/usr/local/nginx/
–with-http_stub_status_module --with-http_realip_module

Posted at Nginx Forum:

funkdoobiest · July 1, 2009, 11:58am

Hello!

On Tue, Jun 30, 2009 at 10:54:57AM -0400, funkdoobiest wrote:

Hi,

the problem of the doubled requests is still there

Here are the lines from the logs:

nginx-access-log:

[…]

213.162.66.XXX - - [28/Jun/2009:14:11:20 +0200] “POST /premium/paypremium HTTP/1.1” 400 0 “http://www.DOMAIN.COM/premium/getpremium/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)”

[…]

Here nginx returned 400 (Bad Request). Since number of bytes sent
to client logged as 0 it’s looks like connection has been dropped
by client while sending request. It shouldn’t involve any
upstream work after error was detected though.

There is no entry in the error-logs for this requests.

Some detailed info should be logged at info level. If your config
hasn’t changed from initial post - your logging level currently
set to error which isn’t enough to see such messages.

Could you please try to obtain debug log for such request?

Maxim D.

funkdoobiest · July 14, 2010, 4:16am

I can not find the method is also modified.

Posted at Nginx Forum: