Hello everybody:
When working with Fast CGI Module, I find that Nginx buffers the whole
request content before send it to the FastCGI Server.
For example, the client uploads a file of 10MB to Nginx, I notice that
the
backend Fast CGI server didn’t receive any request until
all the 10MB data has been upload to the Nginx.
I want to upload a very very large file like a stream and process it in
the
backend FastCGIServer as the request begins, how can I do
this with Nginx + FastCGIModule?
Why the Nginx has to buffer the whole request body before send it to
FastCGI
Server?
I tried to change the following configurations:
fastcgi_buffer_size
fastcgi_buffers
fastcgi_busy_buffers_size
fastcgi_temp_file_write_size
This is really a problem for those who want to use Nginx as a
file-upload server.
Why do you suggest that? This prevents your application worker
processes from being tied up waiting for uploads. If you need
something like upload progress there is a module for that
In fact I work with python, and I find that no matter “Nginx +
FastCGI” or “Nginx + Mod_Wsgi”, the nginx force to buffer the whole
client request body in either memory or file system.
This is really a problem for those who want to use Nginx as a
file-upload server.
If the the raw upload file is to be saved in the local server, Nginx
has good performance.
But I want to compress the upload file and split it and then transfer
to another central storage system,
the file is not to be stored locally.
In fact, the problem is that, if there’s 1000 concurrent requests,
each request has 500MB post body,
It’s to comsume 500G temporary disk space, and it will take a lot of
time to process these data too.
So, I think that if there’s stream processing support, It will be
better under this application.