Synchronous write

Server NGINX
1

Hi.

The mod_wsgi implementation is almost complete.
The only feature that I have to implement is the write callable:

The problem is that the WSGI spec requires the passed-in string to be
completely sent to the client when the write function returns.

Is this possible with Nginx?
It does not matter that the entire process is blocked.

Fortunately not so many WSGI applications use the write callable, and it
is going to be removed from the next revision of WSGI.

Thanks Manlio P.

2

On Tue, Oct 16, 2007 at 12:32:53PM +0200, Manlio P. wrote:

Fortunately not so many WSGI applications use the write callable, and it
is going to be removed from the next revision of WSGI.

In general socket API does not allow to say exactly whether data was
sent completely to a client. The single way is close() with SO_LINGER.

3

Igor S. ha scritto:

Is this possible with Nginx?
It does not matter that the entire process is blocked.

Fortunately not so many WSGI applications use the write callable, and it
is going to be removed from the next revision of WSGI.

In general socket API does not allow to say exactly whether data was
sent completely to a client. The single way is close() with SO_LINGER.

No, the problem is not really whether data was sent completely to a peer
(it is sufficient that the data is in OS buffer)

My problem is how to handle NGX_AGAIN in the write callable, since I
can’t return control to Nginx (and setting the event handlers).

As an example (pseudo code):
while ((rc = ngx_http_output_filter(r, &out) == NGX_AGAIN) {
sleep(…)

But this does not works.

Thanks Manlio P.

4

On Tue, Oct 16, 2007 at 09:33:00PM +0200, Manlio P. wrote:

No, the problem is not really whether data was sent completely to a peer
But this does not works.
This code should be as:

my_handler(ngx_event_t *wev)
{
…

rc = ngx_http_output_filter(...);

if (rc == NGX_AGAIN) {
    r->write_handler = my_handler;
    ngx_handle_write_event(wev)
    add_timer(wev, XXX)
    return;
}
5

Igor S. ha scritto:

The problem is that the WSGI spec requires the passed-in string to be
No, the problem is not really whether data was sent completely to a peer
But this does not works.
r->write_handler = my_handler;
ngx_handle_write_event(wev)
add_timer(wev, XXX)
return;
}

But if control does not return to Nginx, how can my handler be called?

Thanks Manlio P.

6

Igor S. ha scritto:

This code should be as:
add_timer(wev, XXX)
return;
}

But if control does not return to Nginx, how can my handler be called?

“return” should return control to nginx.

The problem is that “return” returns control to the WSGI application and
not to Nginx.

In the current implementation of mod_wsgi I’m raising an exception when
Nginx returns a NGX_AGAIN.

A solution for this problem is to buffer all the data that the WSGI
application want to send using the write callable, but I would like to
know if there are other solutions, since buffering in this way is not
allowed by the WSGI spec.

Thanks Manlio P.

7

On Tue, Oct 16, 2007 at 09:57:15PM +0200, Manlio P. wrote:

if (rc == NGX_AGAIN) {
r->write_handler = my_handler;
ngx_handle_write_event(wev)
add_timer(wev, XXX)
return;
}

But if control does not return to Nginx, how can my handler be called?

“return” should return control to nginx.

8

On Sun, Oct 21, 2007 at 11:57:25AM +0200, Manlio P. wrote:

b->pos = data;

I have done some tests and it seems to work.
No, this completely kill the idea of nginx - a worker will be blocked.

9

Manlio P. ha scritto:

Is this possible with Nginx?
It does not matter that the entire process is blocked.

Igor, do you thing that setting temporarily the socket to blocking mode
can be a solution?

 out.buf = b;
 out.next = NULL;

 b->pos = data;
 b->last = data + len;

 b->memory = 1;
 b->flush = 1;

 ngx_blocking(self->r->connection->fd);
 rc = ngx_http_output_filter(self->r, &out);
 ngx_nonblocking(self->r->connection->fd);

I have done some tests and it seems to work.

Thanks Manlio P.

10

On Mon, Oct 22, 2007 at 12:17:33PM +0200, Manlio P. wrote:

b->last = data + len;

No, this completely kill the idea of nginx - a worker will be blocked.

Unfortunately this is required by the WSGI spec, and some “legacy” WSGI
applications will block the worker, anyway.

I have implemented an alternative version that uses a buffer, but I
would like to know if using this “hack” is “safe”, that is if Nginx will
send the correct data to the client and if after setting again the
socket to nonblocking mode, Nginx will work as usual.

Yes, it will work. However, you should use the following cycle:

ngx_blocking(self->r->connection->fd);

do {
    rc = ngx_http_output_filter(self->r, &out);
} while (rc == NGX_AGAIN);

ngx_nonblocking(self->r->connection->fd);
11

Igor S. ha scritto:

No, this completely kill the idea of nginx - a worker will be blocked.

Unfortunately this is required by the WSGI spec, and some “legacy” WSGI
applications will block the worker, anyway.

I have implemented an alternative version that uses a buffer, but I
would like to know if using this “hack” is “safe”, that is if Nginx will
send the correct data to the client and if after setting again the
socket to nonblocking mode, Nginx will work as usual.

Thanks Manlio P.

12

On Mon, Oct 22, 2007 at 01:11:21PM +0200, Manlio P. wrote:

socket to nonblocking mode, Nginx will work as usual.

That’s unexpected, I have assumed that with socket in blocking mode
ngx_http_output_filter never returns NGX_AGAIN.

No, write() to a socket in blocking mode may send less bytes than you
ask.

13

Igor S. ha scritto:

Yes, it will work. However, you should use the following cycle:

ngx_blocking(self->r->connection->fd);

do {
    rc = ngx_http_output_filter(self->r, &out);
} while (rc == NGX_AGAIN);

ngx_nonblocking(self->r->connection->fd);

That’s unexpected, I have assumed that with socket in blocking mode
ngx_http_output_filter never returns NGX_AGAIN.

Thanks Manlio P.

14

Igor S. ha scritto:

ngx_nonblocking(self->r->connection->fd);

That’s unexpected, I have assumed that with socket in blocking mode
ngx_http_output_filter never returns NGX_AGAIN.

No, write() to a socket in blocking mode may send less bytes than you ask.

Ok, thanks.
I was assuming that NGX_AGAIN is returned only when the socket sets the
EWOULDBLOCK/EAGAIN error.

Manlio P.

15

Igor S. ha scritto:

would like to know if using this “hack” is “safe”, that is if Nginx will
ngx_nonblocking(self->r->connection->fd);

That’s unexpected, I have assumed that with socket in blocking mode
ngx_http_output_filter never returns NGX_AGAIN.

No, write() to a socket in blocking mode may send less bytes than you ask.

But NGX_AGAIN is returned also for EINTR?

I have noted that when testing your “patch” over a slow connection
(qemu) and with a very big buffer, hitting Ctrl-C cause an infinite(?)
loop.

P.S.: even with a very big buffer, ngx_http_output_filter never returns
NGX_AGAIN

Thanks Manlio P.

16

Manlio P. ha scritto:

} while (rc == NGX_AGAIN);
But NGX_AGAIN is returned also for EINTR?

I have noted that when testing your “patch” over a slow connection
(qemu) and with a very big buffer, hitting Ctrl-C cause an infinite(?)
loop.

This happen also if the client closes the connection.

Manlio P.

17

On Mon, Oct 22, 2007 at 09:42:07PM +0200, Manlio P. wrote:

I have implemented an alternative version that uses a buffer, but I

ngx_nonblocking(self->r->connection->fd);

That’s unexpected, I have assumed that with socket in blocking mode
ngx_http_output_filter never returns NGX_AGAIN.

No, write() to a socket in blocking mode may send less bytes than you ask.

But NGX_AGAIN is returned also for EINTR?

No, nginx handles EINTR in write/sendfile opertaion and does not return
NGX_AGAIN.

I have noted that when testing your “patch” over a slow connection
(qemu) and with a very big buffer, hitting Ctrl-C cause an infinite(?) loop.

Could you show debug log of this loop ?

P.S.: even with a very big buffer, ngx_http_output_filter never returns
NGX_AGAIN

Probably because kernel TCP buffer is big enough.

18

Igor S. ha scritto:

In usual processing event->ready is set by event module (kqueue/epoll/etc).

Ok, this solves the problem, thanks!
Now ngx_http_output_filter returns NGX_ERROR when the client closes the
connection.

There is only one minor problem.
When I send a SIGINT signal to nginx during a blocking write operation,
the termination is not “clean”:

[notice] 12546#0: using the “epoll” event method
[notice] 12546#0: nginx/0.5.31
[notice] 12546#0: built by gcc 4.1.2 20061115 (prerelease) (Debian
4.1.1-21)
[notice] 12546#0: OS: Linux 2.6.18-5-686
[notice] 12546#0: sysctl(KERN_RTSIGMAX): 0
[notice] 12546#0: getrlimit(RLIMIT_NOFILE): 1024:1024
[notice] 12546#0: start worker processes
[notice] 12546#0: start worker process 12548
[notice] 12546#0: start worker process 12549
[debug] 12548#0: *1 wsgi_handler
[debug] 12548#0: *1 mod_wsgi: Loading WSGI script
‘/usr/local/nginx/nginx-write.py’
[debug] 12548#0: *1 mod_wsgi: start_response called
[debug] 12548#0: *1 mod_wsgi: sent headers, 0
[debug] 12548#0: *1 mod_wsgi: write called 128000000 0
[notice] 12549#0: signal 2 (SIGINT) received, exiting
[info] 12549#0: epoll_wait() failed (4: Interrupted system call)
[notice] 12549#0: exiting
[notice] 12548#0: signal 2 (SIGINT) received, exiting
[notice] 12546#0: signal 17 (SIGCHLD) received
[notice] 12546#0: worker process 12549 exited with code 0
[notice] 12546#0: signal 2 (SIGINT) received, exiting
[notice] 12546#0: signal 29 (SIGIO) received
[notice] 12546#0: signal 14 (SIGALRM) received
[notice] 12546#0: signal 14 (SIGALRM) received
[notice] 12546#0: signal 14 (SIGALRM) received
[notice] 12546#0: signal 14 (SIGALRM) received
[notice] 12546#0: signal 17 (SIGCHLD) received
[alert] 12546#0: worker process 12548 exited on signal 9
[notice] 12546#0: exit

Not a big problem, however.

Manlio P.

19

On Tue, Oct 23, 2007 at 11:29:41AM +0200, Manlio P. wrote:

} while (rc == NGX_AGAIN);
the termination is not “clean”:
[notice] 12546#0: start worker process 12549
[notice] 12546#0: signal 17 (SIGCHLD) received

Not a big problem, however.

while (rc == NGX_AGAIN && !ngx_quit && !ngx_terminate);

20

On Tue, Oct 23, 2007 at 11:10:37AM +0400, Igor S. wrote:

applications will block the worker, anyway.
rc = ngx_http_output_filter(self->r, &out);
But NGX_AGAIN is returned also for EINTR?

No, nginx handles EINTR in write/sendfile opertaion and does not return
NGX_AGAIN.

I have noted that when testing your “patch” over a slow connection
(qemu) and with a very big buffer, hitting Ctrl-C cause an infinite(?) loop.

Could you show debug log of this loop ?

The loop should be:

do {
   rc = ngx_http_output_filter(self->r, &out);

   self->r->connection->write->ready = 1;

} while (rc == NGX_AGAIN);

In usual processing event->ready is set by event module
(kqueue/epoll/etc).