Forum: NGINX Nginx double requests for my upstreams?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
funkdoobiest (Guest)
on 2009-06-05 16:10
(Received via mailing list)
Hi,

I have a strange behavior for my site load balanced with nginx.
I have 1 Loadbalancer with nginx and 3 servers with apache.

sometimes a request  is doubled and send to 2 of my apache webservers.
this is a big problem for me, because this is sometimes a requests that
is doing a payement.
I checked this behavior with doing a log of all my payment transactions
and there a some transactions
for the same person double at the same second! (the user can't do this
at the website, because I send the form which is doing the request with
JS and ask the User if he really wants to pay and so on)

Here are my config files for nginx, please help me:


#ningx.conf
user  www-data;
worker_processes  4;

error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  512;
}

http {

  include       mime.types;
  include       proxy.conf;

    access_log  logs/access.log  combined;

    sendfile        on;
    tcp_nopush     on;

    keepalive_timeout  0;
    #keepalive_timeout  15;

    #gzip  on;


  include   upstream.conf;

  include server_www.conf;




}



# proxy.conf
proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;




#upstream.conf
  upstream webseite {
    ip_hash;
    server 10.XXX.XX.1:8081;
    server 10.XXX.XX.2:8081;
    server 10.XXX.XX.3:8081;
  }



#server_www.conf
server {
    listen 80;
    server_name www.website.com;


    # serve static files
  location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|html|htm|swf)$ {
            root    /var/www/www.website.com/;
            expires 1d;
    }

    location / {
      proxy_pass http://webseite;
    }
}


Thank you!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,2644#msg-2644
Maxim D. (Guest)
on 2009-06-05 17:40
(Received via mailing list)
Hello!

On Fri, Jun 05, 2009 at 07:59:13AM -0400, funkdoobiest wrote:

> Hi,
>
> I have a strange behavior for my site load balanced with nginx.
> I have 1 Loadbalancer with nginx and 3 servers with apache.
>
> sometimes a request  is doubled and send to 2 of my apache webservers.
> this is a big problem for me, because this is sometimes a requests that is doing a 
payement.
> I checked this behavior with doing a log of all my payment transactions and there a some 
transactions
> for the same person double at the same second! (the user can't do this at the website, 
because I send the form which is doing the request with JS and ask the User if he really 
wants to pay and so on)

http://wiki.nginx.org/NginxHttpProxyModule#proxy_n...

Maxim D.
funkdoobiest (Guest)
on 2009-06-08 16:44
(Received via mailing list)
Hi Maxim,

Maxim D. Wrote:
-------------------------------------------------------
> with apache.
> send the form which is doing the request with JS
> and ask the User if he really wants to pay and so
> on)
>
> http://wiki.nginx.org/NginxHttpProxyModule#proxy_n
> ext_upstream
>
> Maxim D.

Thank you for your help. I set proxy_next_upstream to off, but there are
still double requests :-(

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,2689#msg-2689
Michael S. (Guest)
on 2009-06-08 16:57
(Received via mailing list)
I think I noticed this behavior too. The upstream didn't fail but it
tried multiple in parallel anyway.

On Jun 7, 2009, at 12:35 PM, "funkdoobiest" 
<removed_email_address@domain.invalid>
Maxim D. (Guest)
on 2009-06-08 18:07
(Received via mailing list)
Hello!

On Sun, Jun 07, 2009 at 03:35:41PM -0400, funkdoobiest wrote:

> > >
> > my payment transactions and there a some
> > Maxim D.
>
> Thank you for your help. I set proxy_next_upstream to off, but there are still double 
requests :-(

It doesn't looks like nginx problem.  Try checking logs (frontend
and backend ones) - probably your javascript code just does
something wrong.

Maxim D.
funkdoobiest (Guest)
on 2009-06-09 07:53
(Received via mailing list)
Hello,

Maxim D. Wrote:
-------------------------------------------------------
> -----
> servers
> all
> > > on)
> double requests :-(
>
> It doesn't looks like nginx problem.  Try checking
> logs (frontend
> and backend ones) - probably your javascript code
> just does
> something wrong.
>
> Maxim D.

I checked it about 100 times. I checked the logs also. It's not my
script.
The requests are at the same second. If i disable the loadbalancer (in
this case nginx) double request are stopping.

Jan

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,2706#msg-2706
Maxim D. (Guest)
on 2009-06-09 09:28
(Received via mailing list)
Hello!

On Mon, Jun 08, 2009 at 06:45:03AM -0400, funkdoobiest wrote:

> > >
> > > > >
> > is
> > > > send the form which is doing the request with
> > > > Maxim D.
> >
> > Maxim D.
>
> I checked it about 100 times. I checked the logs also. It's not my script.
> The requests are at the same second. If i disable the loadbalancer (in this case nginx) 
double request are stopping.

Try the following:

1. Post here nginx -V output to make it clear which version do you
use.

2. Post here results of your logs investigations to make it clear
that problem appears somewhere at nginx level.

This should include relevant nginx access logs (expected: one line
with client request; and some grep results to show there is no other
identical lines), relevant backend access logs (doubled request
should be visible here), relevant lines from nginx error_log if any.

3. Post here your nginx configs to make it clear how requests in
question are processed.  There are more than one way to screw
things up, and nginx config does matter.

4. Try obtaining relevant debug log (some info may be found at
http://wiki.nginx.org/NginxDebugging), post it here.

Maxim D.
funkdoobiest (Guest)
on 2009-06-23 06:30
(Received via mailing list)
Hi Maxim,

I have checked the nginx-access logs and I found following:

84.72.240.XXX - - [03/Jun/2009:16:38:46 +0200] "POST /premium/paypremium
HTTP/1.1" 499 0 "http://www.host.com/premium/getpremium/" "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
84.72.240.XXX - - [03/Jun/2009:16:38:48 +0200] "POST /premium/paypremium
HTTP/1.1" 200 2230 "http://www.host.com/premium/getpremium/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)"



What can I do or what do I wrong?

Thank you,
Jan

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,3272#msg-3272
funkdoobiest (Guest)
on 2009-06-23 15:54
(Received via mailing list)
My nginx-version:

nginx version: nginx/0.7.59
built by gcc 4.3.2 (Debian 4.3.2-1.1)
configure arguments: --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_stub_status_module --user=www-data --group=www-data

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,3287#msg-3287
Maxim D. (Guest)
on 2009-06-23 16:32
(Received via mailing list)
Hello!

On Mon, Jun 22, 2009 at 10:19:54PM -0400, funkdoobiest wrote:

> Hi Maxim,
>
> I have checked the nginx-access logs and I found following:
>
> 84.72.240.XXX - - [03/Jun/2009:16:38:46 +0200] "POST /premium/paypremium HTTP/1.1" 499 0 
"http://www.host.com/premium/getpremium/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 
5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
> 84.72.240.XXX - - [03/Jun/2009:16:38:48 +0200] "POST /premium/paypremium HTTP/1.1" 200 
2230 "http://www.host.com/premium/getpremium/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 
NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
>
>
>
> What can I do or what do I wrong?

Error code 499 means that *client* closed connection before it got
response.  In this particular case it looks like duplicate form
submit, e.g. due to second click on submit button.

Maxim D.
funkdoobiest (Guest)
on 2009-06-24 11:46
(Received via mailing list)
Maxim D. Wrote:
-------------------------------------------------------
> > 84.72.240.XXX - - [03/Jun/2009:16:38:46 +0200]
> >
> Maxim D.
I got this 499 error code for about 10% of all the connections. not just
for form-submits.
It seems that "proxy_ignore_client_abort on;" fix that problem. But why?
Is it ok to ignore client aborts?

Jan

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,3293#msg-3293
Maxim D. (Guest)
on 2009-06-24 12:08
(Received via mailing list)
Hello!

On Tue, Jun 23, 2009 at 10:33:05AM -0400, funkdoobiest wrote:

> > following:
> > 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
> > button.
> >
> > Maxim D.
>
> I got this 499 error code for about 10% of all the connections. not just for 
form-submits.

In general 499 errors are normal, since clients tend to close
browser windows/tabs, cancel downloads or just navigate to another
page e.g. before loading all images etc.  High percentage of this
errors may indicate that your site is too slow, but since actual
numbers depend heavily on site usage pattern it's hard to say what
is "high".

> It seems that "proxy_ignore_client_abort on;" fix that problem. But why? Is it ok to 
ignore client aborts?

Directive proxy_ignore_client_abort instructs nginx to finish
request processing even if client closed connection.  It's safe
but usually it's just waste of resources (since backend response
will be thrown away).

Maxim D.
funkdoobiest (Guest)
on 2009-07-01 12:53
(Received via mailing list)
Hi,

the problem of the doubled requests is still there :-(

Here are the lines from the logs:

nginx-access-log:


93.210.31.XXX - - [28/Jun/2009:14:11:15 +0200] "GET / HTTP/1.1" 200 2480
"-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1;
.NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; .NET CLR
1.1.4322)"
87.242.40.XX - - [28/Jun/2009:14:11:16 +0200] "GET / HTTP/1.1" 200 2480
"http://www.google.hu/search?q=DOMAIN.COM&ie=utf-8&...
"Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11"
72.30.79.XXX - - [28/Jun/2009:14:11:17 +0200] "GET
/index.php?option=com_content&view=category&id=3&Itemid=13&limitstart=1440
HTTP/1.0" 404 289 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0;
http://help.yahoo.com/help/us/ysearch/slurp)"
91.17.86.XX - - [28/Jun/2009:14:11:17 +0200] "POST
/werbung/ad/fullsize/subpart/1501 HTTP/1.1" 200 176
"http://www.DOMAIN.COM/live/show/1/" "Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR
1.1.4322)"
217.231.102.XXX - - [28/Jun/2009:14:11:19 +0200] "GET /live/ HTTP/1.1"
200 2959 "http://www.DOMAIN.COM" "Mozilla/4.0 (compatible; MSIE 5.5;
Windows 98; T-Online Internatinal AG)"

213.162.66.XXX - - [28/Jun/2009:14:11:20 +0200] "POST
/premium/paypremium HTTP/1.1" 400 0
"http://www.DOMAIN.COM/premium/getpremium/" "Mozilla/4.0 (compatible;
MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center
PC 5.0; .NET CLR 3.0.04506)"

87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] "GET / HTTP/1.1" 200 2480
"http://www.google.hu/search?q=DOMAIN.COM&ie=utf-8&...
"Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11"
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] "GET /DOMAIN/css/style.css
HTTP/1.1" 200 18139 "http://www.DOMAIN.COM/" "Mozilla/5.0 (Windows; U;
Windows NT 5.1; hu; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11"
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] "GET
/DOMAIN/css/layout.css HTTP/1.1" 200 3798 "http://www.DOMAIN.COM/"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11"
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] "GET
/DOMAIN/css/jScrollPane.css HTTP/1.1" 200 1736 "http://www.DOMAIN.COM/"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11"
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] "GET
/DOMAIN/images/favicon.ico HTTP/1.1" 200 1150 "-" "Mozilla/5.0 (Windows;
U; Windows NT 5.1; hu; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11"
87.242.40.XX - - [28/Jun/2009:14:11:21 +0200] "GET
/DOMAIN/javascript/AC_RunActiveContent.js HTTP/1.1" 200 8321
"http://www.DOMAIN.COM/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; hu;
rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11"
62.226.184.XXX - - [28/Jun/2009:14:11:21 +0200] "POST
/werbung/ad/fullsize/subpart/1504 HTTP/1.1" 200 176
"http://www.DOMAIN.COM/live/show/1/" "Mozilla/5.0 (Windows; U; Windows
NT 6.0; de; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11"
89.53.233.XXX - - [28/Jun/2009:14:11:21 +0200] "GET
/mediacenter/overview/25/ HTTP/1.1" 200 2875
"http://www.DOMAIN.COM/mediacenter/list/9/2008/" "Mozilla/4.0
(compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727;
Media Center PC 5.0; .NET CLR 3.0.04506)"
89.53.233.XXX - - [28/Jun/2009:14:11:22 +0200] "GET
/mediacenter/overview/25/DOMAIN/images/favicon.ico HTTP/1.1" 404 529 "-"
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET
CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"



apache-access-log:


10.226.50.67 - - [28/Jun/2009:14:11:20 +0200] "POST /premium/paypremium
HTTP/1.0" 200 2330 "http://www.DOMAIN.com/premium/getpremium/"
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET
CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"
10.226.50.67 - - [28/Jun/2009:14:11:20 +0200] "POST /premium/paypremium
HTTP/1.0" 200 2330 "http://www.DOMAIN.com/premium/getpremium/"
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET
CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"


There is no entry in the error-logs for this requests.
So the problem is not JavaScript or Form-Error. I got double requests
for other requests to.

Thank you,
Jan

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,3559#msg-3559
Maxim D. (Guest)
on 2009-07-01 13:58
(Received via mailing list)
Hello!

On Tue, Jun 30, 2009 at 10:54:57AM -0400, funkdoobiest wrote:

> Hi,
>
> the problem of the doubled requests is still there :-(
>
> Here are the lines from the logs:
>
> nginx-access-log:

[...]

> 213.162.66.XXX - - [28/Jun/2009:14:11:20 +0200] "POST /premium/paypremium HTTP/1.1" 400 
0 "http://www.DOMAIN.COM/premium/getpremium/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows 
NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"

[...]

Here nginx returned 400 (Bad Request).  Since number of bytes sent
to client logged as 0 it's looks like connection has been dropped
by client while sending request.  It shouldn't involve any
upstream work after error was detected though.

> There is no entry in the error-logs for this requests.

Some detailed info should be logged at info level.  If your config
hasn't changed from initial post - your logging level currently
set to error which isn't enough to see such messages.

Could you please try to obtain debug log for such request?

Maxim D.
showrunning (Guest)
on 2010-07-13 15:37
(Received via mailing list)
I also had the same mistakes of double request
I try to do proxy_ignore_client_abort on,http_499 fewer errors, but
appeared http_400 error.
the double request at the some time problem still was not solved

sever info
nginx (gzip)--(static request)-->squid--->apache
|
|
(php request )
|
|
nginx(loadbalance+gzip)
|
apache

I have 1 Loadbalancer with nginx and 4 servers with apache.
apache 2.0.59

nginx version: nginx/0.7.65
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
configure arguments: --user=www --group=www --prefix=/usr/local/nginx/
--with-http_stub_status_module --with-http_realip_module

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,108245#msg-108245
gelanyi1 (Guest)
on 2010-07-14 06:16
(Received via mailing list)
I can not find the method is also modified.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2644,108545#msg-108545
This topic is locked and can not be replied to.