Help with AIO

Server NGINX
1

Hi

I configured Nginx with *aio *mode *on *and 1 worker process
In benchmarking I saw limit of 1024 concurrent sessions even configured
to
4096 worker connections
When I looked in the source code I could see that io_setup is called
with
fix value of 1024 max events
can anyone please explain me the hard coded limit
Is it OK to change it or it is derived from other limit I am not aware
of
any help will be appreciated

thanks
Hagai

–

Hagai A.
Qwilt | Work: +972-72-2221644| Mobile: +972-54-4895656 |
[email protected][email protected]

2

Hello!

On Tue, Sep 06, 2011 at 01:21:00AM +0300, Hagai A. wrote:

Hi

I configured Nginx with *aio *mode *on *and 1 worker process
In benchmarking I saw limit of 1024 concurrent sessions even configured to
4096 worker connections

Sure it’s not limitation of the benchmarking tool you use? Most
tools out there use select() and hence limited to 1024
concurrent connections.

When I looked in the source code I could see that io_setup is called with
fix value of 1024 max events
can anyone please explain me the hard coded limit
Is it OK to change it or it is derived from other limit I am not aware of
any help will be appreciated

io_setup() maxevents is a limit on number of aio requests allowed
to be submitted simulteneously; if nginx hits this limit, it will
fallback to normal blocking read.

If you are going to change it, make sure to keep total for all
processes lower than /proc/sys/fs/aio-max-nr.

Maxim D.

3

Thanks a lot
I will try it

On Tue, Sep 6, 2011 at 9:33 AM, Maxim D. [email protected] wrote:

nginx mailing list
[email protected]
nginx Info Page

–

Hagai A.
Qwilt | Work: +972-72-2221644| Mobile: +972-54-4895656 |
[email protected][email protected]

4

Hi

When benchmarking AIO read, I reached more than 1024 concurrent Async IO

  1. I saw a weird error log
  2. Nginx did not fallback to regular blocking read

I looked on the source code – src/os/unix/ngx_linux_aio_read.c
line:105
and I think there is a problem in errno handling

// Current Code
//----------------------------------------------------------------------------
*n = io_submit(ngx_aio_ctx, 1, piocb);

if (n == 1) {
    ev->active = 1;
    ev->ready = 0;
    ev->complete = 0;

    return NGX_AGAIN;
}

n = -n;

<---------------------------------------------------------??

if (n == NGX_EAGAIN) {
    return ngx_read_file(file, buf, size, offset);
}

ngx_log_error(NGX_LOG_CRIT, file->log, n,
              "io_submit(\"%V\") failed", &file->name);

if (n == NGX_ENOSYS) {
    ngx_file_aio = 0;
    return ngx_read_file(file, buf, size, offset);
}

return NGX_ERROR;*

// I think it should be
//----------------------------------------------------------------------------
*n = io_submit(ngx_aio_ctx, 1, piocb);

if (n == 1) {
    ev->active = 1;
    ev->ready = 0;
    ev->complete = 0;

    return NGX_AGAIN;
}

if (n == 0)
{
    ngx_log_error(NGX_LOG_CRIT, file->log, 0,
              "io_submit(\"%V\") failed - 0 - I/O request blocks",

&file->name);

    return NGX_ERROR;
}
else
{
    n = errno;
}
    • if (n== NGX_EAGAIN) {
      return ngx_read_file(file, buf, size, offset);
      }

    ngx_log_error(NGX_LOG_CRIT, file->log, n,
    “io_submit("%V") failed”, &file->name);

    if (n== NGX_ENOSYS) {
    ngx_file_aio = 0;
    return ngx_read_file(file, buf, size, offset);
    }

    return NGX_ERROR;*
    //----------------------------------------------------------------------------

your help will be appreciated
Thanks
Hagai

On Tue, Sep 6, 2011 at 9:33 AM, Maxim D. [email protected] wrote:

nginx mailing list
[email protected]
nginx Info Page

–

Hagai A.
Qwilt | Work: +972-72-2221644| Mobile: +972-54-4895656 |
[email protected][email protected]

5

On Thu, Sep 15, 2011 at 04:21:44PM +0200, Hagai A. wrote:

//----------------------------------------------------------------------------
n = -n; <---------------------------------------------------------??
Thank you, this is a bug. I confused this with _syscall(2).

–
Igor S.

6

On Fri, Sep 16, 2011 at 06:10:02PM +0400, Igor S. wrote:

}

Also I’m going to introduce the directive:

events {
worker_aio_events 100;
}

to set how much aio events worker may request to the kernel.
I’m going to set default value to 32 events per worker.
Currenly this value is hardcoded to 1024 and it seems it is too large,
since default /proc/sys/fs/aio-max-nr value is 65536 and this means
that you can run no more than 64 workers.

I think it should be renamed to

events {
worker_aio_requests 100;
}

–
Igor S.

7

On Fri, Sep 16, 2011 at 02:39:07PM +0400, Igor S. wrote:

// Current Code

n = -n;    <---------------------------------------------------------??

Thank you, this is a bug. I confused this with _syscall(2).

Linux AIO fixed in these two commits:

http://trac.nginx.org/nginx/changeset/4130/nginx
http://trac.nginx.org/nginx/changeset/4131/nginx

Also I’m going to introduce the directive:

events {
worker_aio_events 100;
}

to set how much aio events worker may request to the kernel.
I’m going to set default value to 32 events per worker.
Currenly this value is hardcoded to 1024 and it seems it is too large,
since default /proc/sys/fs/aio-max-nr value is 65536 and this means
that you can run no more than 64 workers.

–
Igor S.