Forum: NGINX Losing POST data on vanilla Ubuntu/nginx/mongrel install

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Aaron Starr (Guest)
on 2009-04-04 01:33
(Received via mailing list)
Hi, all. Sorry in advance for what is hopefully a simple question.

I'm using nginx as a reverse-proxy for a few mongrel instances. Using
AJAX (XMLHttpRequest), I POST a small bit of JSON to the server. The
request is properly proxied to mongrel, but with the POST data
missing!

(I.e., in the rails controller, request.env['RAW_POST_DATA'] and
reqest.raw_post are empty.)

When I take nginx out of the mix, by POSTing directly to mongrel, it
works fine. And, it was working fine before when going through nginx,
but some mysterious change happened in the last couple of days, and
now I usually see the problem.

Anyone have any ideas?


Versions:

mongrel 1.1.5
mongrel cluster 1.0.5
nginx 0.7.9
Ubuntu 8.04.1

Configuration:

user www www;
worker_processes 1;

error_log logs/error.log error;
pid        /var/run/nginx.pid;

events {
  worker_connections 1024;
}

http {
  include mime.types;
  default_type application/octet-stream;
  access_log logs/access.log;

  sendfile on;
  #tcp_nopush on;
  keepalive_timeout 65;
  tcp_nodelay on;

  gzip on;
  gzip_min_length 1100;
  gzip_buffers 4 8k;
  gzip_types text/plain text/javascript application/x-javascript;

  upstream mongrel {
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
    server 127.0.0.1:3002;
    server 127.0.0.1:3003;
    server 127.0.0.1:3004;
  }

  server {
    listen 80;
    server_name wiredquote.com;
    root /opt/apps/benefits/public;
    index index.html;
    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect false;
      if (-f $request_filename) {
        break;
      }
      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }
      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }
      if (!-f $request_filename) {
        proxy_pass http://mongrel;
        break;
      }
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
      root   html;
    }
  }
}
Aaron Starr (Guest)
on 2009-04-04 12:02
(Received via mailing list)
Update on the missing POST data when proxying to mongrel:

This seems crazy, but it seems to me that the problem is related to
having very large, un-rotated nginx log files. I noticed that the
files were getting too large, and removed them (and got logrotate set
up), and now I get my POST data, again.

If anyone could confirm or deny this apparently nonsensical
explanation, that would be appreciated.

Aaron
Maxim D. (Guest)
on 2009-04-04 18:48
(Received via mailing list)
Hello!

On Sat, Apr 04, 2009 at 12:49:13AM -0700, Aaron Starr wrote:

> Update on the missing POST data when proxying to mongrel:
>
> This seems crazy, but it seems to me that the problem is related to
> having very large, un-rotated nginx log files. I noticed that the
> files were getting too large, and removed them (and got logrotate set
> up), and now I get my POST data, again.
>
> If anyone could confirm or deny this apparently nonsensical
> explanation, that would be appreciated.

Probably you have no space left for files in client_body_temp_path
- and as soon as you removed large old logs there are space and
everything is working again.

Guess you have nothing in error_log due to the very same reason -
no space left on device.

Maxim D.
Aaron Starr (Guest)
on 2009-04-05 03:04
(Received via mailing list)
Maxim D.,

Thanks for your response!

I have several G on the logs volume, so for that reason it's probably
not a disk space issue.

Also, though, the log file thing seems to have been a red herring. I
just went and tried the POST, again, having made no changes at all
since last time, and it's not working, again. And the log files are
small.

So, it seems that nginx randomly starts stripping POST data when
proxying to mongrel. I can't be the first person who's seen something
like this! I really don't think I've done anything unusual.

Can anyone think of a reason why POST data might be stripped when
proxying?

Thanks for any ideas!

Aaron
Dave C. (Guest)
on 2009-04-05 10:01
(Received via mailing list)
Hi Aaron,

lsof the nginx worker, it could be that the post data is never
received because nginx is still waiting for the client to transmit it.
hup'ing the worker, closes any outstanding connections and you get a
bit of data through before things logjam.

ngrep the input, see if your ajax worker is sending a Content-Length
header on the POST (nginx cannot currently handle chunked style posts)

Cheers

Dave
Maxim D. (Guest)
on 2009-04-05 15:12
(Received via mailing list)
Hello!

On Sat, Apr 04, 2009 at 03:55:31PM -0700, Aaron Starr wrote:

> small.
>
> So, it seems that nginx randomly starts stripping POST data when
> proxying to mongrel. I can't be the first person who's seen something
> like this! I really don't think I've done anything unusual.
>
> Can anyone think of a reason why POST data might be stripped when proxying?

First of all, try looking into error_log.  If it doesn't help -
obtain debug log and post it here together with config.

Maxim D.
Aaron Starr (Guest)
on 2009-04-06 07:38
(Received via mailing list)
Maxim D. and Dave C.,

Thank you for your help. I've acquired lsof and ngrep and set the
logging level to debug for nginx, and now -- of course -- it seems to
be working perfectly. Aaaagh!

As soon as it quits sending POST data, I'll collect all the
information you suggested, and if I can't find anything untoward, I'll
post it here.

Thanks, again, sincerely.

Aaron
Aaron Starr (Guest)
on 2009-04-09 03:53
(Received via mailing list)
One last follow-up, to let you know what I know about this mystery.

I can reproduce it consistently and it looks like it is probably a
mongrel problem, not nginx at all. For any particular mongrel
instance, if the first request after it has been started is a POST,
then the mongrel/rails combo can not seem to find the post data. If
the mongrel instance is hit with a GET first (or a failing POST), then
all subsequent POSTs work just fine.

Using ngrep, I've verified that nginx is sending the post data to the
mongrel instances even for that first POST, so it's looking unlikely
that it has anything to do with nginx.

Thanks again for your time and help, and I'm off to google "mongrel
missing post data on first request."

Aaron
This topic is locked and can not be replied to.