Nginx writing to Cephfs

Server NGINX
1

Hello,

I’m having an issue with nginx writing to cephfs. Often I’m getting:

writev() “/home/ceph/temp/44/94/1/0000119444” failed (4: Interrupted
system call) while reading upstream

looking with strace, this happens:

…
write(65, “e\314\366\36\302”…, 65536) = ? ERESTARTSYS (To be
restarted)

It happens after first 4MBs (exactly) are written, subsequent write gets
ERESTARTSYS (sometimes, but more rarely, it fails after first 32 or
64MBs, etc are written). Apparently nginx doesn’t expect this and
doesn’t handle it so it cancels writes and deletes this partial file.
Looking at the code, I saw it doesn’t handle ERESTARTSYS in any
different way compared to other write errors. Shouldn’t it try to write
same data again for a couple of times before finally giving up and
erroring out? Do you have any suggestions on how to resolve this? I’m
using latest stable nginx.

Regards,
Vedran

2

On Thursday 23 July 2015 14:40:58 Vedran Furač wrote:

write(65, “e\314\366\36\302”…, 65536) = ? ERESTARTSYS (To be restarted)

It more looks like a bug in cephfs. writev() should never return
ERESTARTSYS.

wbr, Valentin V. Bartenev

3

Valentin V. Bartenev Wrote:

It more looks like a bug in cephfs. writev() should never return
ERESTARTSYS.

I’ve talked to the ceph people, they say ERESTARTSYS shows up in strace
output but it is handled by the kernel and that writev(2) is interrupted
by
the SIGALRM, which actually appears in the strace output just after
writev
fails.

I also failed to get this error by doing the same this as nginx using
dd, dd
always succeeded so it happens due to combination of nginx and cephfs.

Here’s full strace output (2 examples from 2 differently configured
servers):

http://pastebin.com/wUAAcdT7

http://pastebin.com/wHyWc9U5

Thanks,
Vedran

Posted at Nginx Forum:

4

Hello,

So Ceph devs final reply was:

“ngx_write_fd() is just a write(), which, when interrupted by SIGALRM,
fails with EINTR because SA_RESTART is not set. We can try digging
further, but I think nginx should retry in this case.”

Let me know what do you think.

Thanks,
Vedran

Posted at Nginx Forum:

5

On Thursday 23 July 2015 14:51:58 vedranf wrote:

I also failed to get this error by doing the same this as nginx using dd, dd
always succeeded so it happens due to combination of nginx and cephfs.

Here’s full strace output (2 examples from 2 differently configured
servers):

http://pastebin.com/wUAAcdT7

http://pastebin.com/wHyWc9U5

Do you have timer_resolution configured?

wbr, Valentin V. Bartenev

6

Valentin V. Bartenev Wrote:

interrupted by

servers):

http://pastebin.com/wUAAcdT7

http://pastebin.com/wHyWc9U5

Do you have timer_resolution configured?

Yes, it’s:

timer_resolution 50ms;

Posted at Nginx Forum:

7

On Friday 24 July 2015 09:55:04 vedranf wrote:

output but it is handled by the kernel and that writev(2) is interrupted by

http://pastebin.com/wHyWc9U5

Do you have timer_resolution configured?

Yes, it’s:

timer_resolution 50ms;

This is the root cause of interrupts. Every 50ms it signals nginx
and can interrupt any interruptible syscall (writing to file is
usually not, but it seems different for Cephfs).

You should avoid using timer_resolution, or try this patch:

diff -r be8d8b1dad78 src/os/unix/ngx_files.c
— a/src/os/unix/ngx_files.c Fri Jul 24 17:18:20 2015 +0300
+++ b/src/os/unix/ngx_files.c Fri Jul 24 17:18:37 2015 +0300
@@ -264,6 +264,7 @@ ngx_write_chain_to_file(ngx_file_t *file
u_char *prev;
size_t size;
ssize_t total, n;

  • ngx_err_t err;
    ngx_array_t vec;
    struct iovec *iov, iovs[NGX_IOVS];

@@ -335,10 +336,20 @@ ngx_write_chain_to_file(ngx_file_t *file
file->sys_offset = offset;
}

+eintr:
+
n = writev(file->fd, vec.elts, vec.nelts);

     if (n == -1) {

  •        ngx_log_error(NGX_LOG_CRIT, file->log, ngx_errno,

  •        err = ngx_errno;

  •        if (err == NGX_EINTR) {

  •            ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, err,

  •                           "writev() was interrupted");

  •            goto eintr;

  •        }

  •        ngx_log_error(NGX_LOG_CRIT, file->log, err,
                     "writev() \"%s\" failed", file->name.data);
       return NGX_ERROR;
   }
8

Valentin V. Bartenev Wrote:

always succeeded so it happens due to combination of nginx and

You should avoid using timer_resolution, or try this patch:

Thank you, removing timer_resolution fixed the problem. I’ll look into
the
patch in the next few days.

Regards,
Vedran

Posted at Nginx Forum: