on 2009-04-04 01:33
on 2009-04-04 12:02
Update on the missing POST data when proxying to mongrel: This seems crazy, but it seems to me that the problem is related to having very large, un-rotated nginx log files. I noticed that the files were getting too large, and removed them (and got logrotate set up), and now I get my POST data, again. If anyone could confirm or deny this apparently nonsensical explanation, that would be appreciated. Aaron
on 2009-04-04 18:48
Hello! On Sat, Apr 04, 2009 at 12:49:13AM -0700, Aaron Starr wrote: > Update on the missing POST data when proxying to mongrel: > > This seems crazy, but it seems to me that the problem is related to > having very large, un-rotated nginx log files. I noticed that the > files were getting too large, and removed them (and got logrotate set > up), and now I get my POST data, again. > > If anyone could confirm or deny this apparently nonsensical > explanation, that would be appreciated. Probably you have no space left for files in client_body_temp_path - and as soon as you removed large old logs there are space and everything is working again. Guess you have nothing in error_log due to the very same reason - no space left on device. Maxim D.
on 2009-04-05 03:04
Maxim D., Thanks for your response! I have several G on the logs volume, so for that reason it's probably not a disk space issue. Also, though, the log file thing seems to have been a red herring. I just went and tried the POST, again, having made no changes at all since last time, and it's not working, again. And the log files are small. So, it seems that nginx randomly starts stripping POST data when proxying to mongrel. I can't be the first person who's seen something like this! I really don't think I've done anything unusual. Can anyone think of a reason why POST data might be stripped when proxying? Thanks for any ideas! Aaron
on 2009-04-05 10:01
Hi Aaron, lsof the nginx worker, it could be that the post data is never received because nginx is still waiting for the client to transmit it. hup'ing the worker, closes any outstanding connections and you get a bit of data through before things logjam. ngrep the input, see if your ajax worker is sending a Content-Length header on the POST (nginx cannot currently handle chunked style posts) Cheers Dave
on 2009-04-05 15:12
Hello! On Sat, Apr 04, 2009 at 03:55:31PM -0700, Aaron Starr wrote: > small. > > So, it seems that nginx randomly starts stripping POST data when > proxying to mongrel. I can't be the first person who's seen something > like this! I really don't think I've done anything unusual. > > Can anyone think of a reason why POST data might be stripped when proxying? First of all, try looking into error_log. If it doesn't help - obtain debug log and post it here together with config. Maxim D.
on 2009-04-06 07:38
Maxim D. and Dave C., Thank you for your help. I've acquired lsof and ngrep and set the logging level to debug for nginx, and now -- of course -- it seems to be working perfectly. Aaaagh! As soon as it quits sending POST data, I'll collect all the information you suggested, and if I can't find anything untoward, I'll post it here. Thanks, again, sincerely. Aaron
on 2009-04-09 03:53
One last follow-up, to let you know what I know about this mystery. I can reproduce it consistently and it looks like it is probably a mongrel problem, not nginx at all. For any particular mongrel instance, if the first request after it has been started is a POST, then the mongrel/rails combo can not seem to find the post data. If the mongrel instance is hit with a GET first (or a failing POST), then all subsequent POSTs work just fine. Using ngrep, I've verified that nginx is sending the post data to the mongrel instances even for that first POST, so it's looking unlikely that it has anything to do with nginx. Thanks again for your time and help, and I'm off to google "mongrel missing post data on first request." Aaron