# Nginx writing to Cephfs **URL:** \ **Category:** NGINX\ **Created:** [July 23, 2015, 2:42pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741 "2015-07-23T14:42:27Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Rafa\_F](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/958977/32.png) [@Rafa\_F](https://www.ruby-forum.com/u/Rafa_F)\ **Post date:** [July 23, 2015, 2:42pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/1 "2015-07-23T14:42:27Z")
Hello, I’m having an issue with nginx writing to cephfs. Often I’m getting: writev() “/home/ceph/temp/44/94/1/0000119444” failed (4: Interrupted system call) while reading upstream looking with strace, this happens: … write(65, “e\314\366\36\302”…, 65536) = ? ERESTARTSYS (To be restarted) It happens after first 4MBs (exactly) are written, subsequent write gets ERESTARTSYS (sometimes, but more rarely, it fails after first 32 or 64MBs, etc are written). Apparently nginx doesn’t expect this and doesn’t handle it so it cancels writes and deletes this partial file. Looking at the code, I saw it doesn’t handle ERESTARTSYS in any different way compared to other write errors. Shouldn’t it try to write same data again for a couple of times before finally giving up and erroring out? Do you have any suggestions on how to resolve this? I’m using latest stable nginx. Regards, Vedran ---
**Author:** ![Vedran\_FuraSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/f05b48/32.png) [@Vedran\_FuraSS](https://www.ruby-forum.com/u/Vedran_FuraSS)\ **Post date:** [July 23, 2015, 3:44pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/2 "2015-07-23T15:44:43Z")
On Thursday 23 July 2015 14:40:58 Vedran Furač wrote: > write(65, “e\314\366\36\302”…, 65536) = ? ERESTARTSYS (To be restarted) It more looks like a bug in cephfs. writev() should never return ERESTARTSYS. wbr, Valentin V. Bartenev ---
**Author:** ![Vedran\_FuraSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/f05b48/32.png) [@Vedran\_FuraSS](https://www.ruby-forum.com/u/Vedran_FuraSS)\ **Post date:** [July 23, 2015, 8:52pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/3 "2015-07-23T20:52:35Z")
Valentin V. Bartenev Wrote: > It more looks like a bug in cephfs. writev() should never return > ERESTARTSYS. I’ve talked to the ceph people, they say ERESTARTSYS shows up in strace output but it is handled by the kernel and that writev(2) is interrupted by the SIGALRM, which actually appears in the strace output just after writev fails. I also failed to get this error by doing the same this as nginx using dd, dd always succeeded so it happens due to combination of nginx and cephfs. Here’s full strace output (2 examples from 2 differently configured servers): [http://pastebin.com/wUAAcdT7](http://pastebin.com/wUAAcdT7) [http://pastebin.com/wHyWc9U5](http://pastebin.com/wHyWc9U5) Thanks, Vedran Posted at Nginx Forum: [http://forum.nginx.org/read.php?2,260512,260527#msg-260527](http://forum.nginx.org/read.php?2,260512,260527#msg-260527) ---
**Author:** ![Vedran\_FuraSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/f05b48/32.png) [@Vedran\_FuraSS](https://www.ruby-forum.com/u/Vedran_FuraSS)\ **Post date:** [July 24, 2015, 3:40pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/4 "2015-07-24T15:40:36Z")
Hello, So Ceph devs final reply was: “ngx\_write\_fd() is just a write(), which, when interrupted by SIGALRM, fails with EINTR because SA\_RESTART is not set. We can try digging further, but I think nginx should retry in this case.” Let me know what do you think. Thanks, Vedran Posted at Nginx Forum: > **[Re: Nginx writing to Cephfs](https://forum.nginx.org/read.php?2,260512,260540#msg-260540)** > > Hello, I'm having an issue with nginx writing to cephfs. Often I'm getting: writev() "/home/ceph/temp/44/94/1/0000119444" failed (4: Interrupted system call) while reading upstream looking with strace, this happens: .... write ---
**Author:** ![Vedran\_FuraSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/f05b48/32.png) [@Vedran\_FuraSS](https://www.ruby-forum.com/u/Vedran_FuraSS)\ **Post date:** [July 24, 2015, 3:41pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/5 "2015-07-24T15:41:47Z")
On Thursday 23 July 2015 14:51:58 vedranf wrote: > I also failed to get this error by doing the same this as nginx using dd, dd > always succeeded so it happens due to combination of nginx and cephfs. > > Here’s full strace output (2 examples from 2 differently configured > servers): > > [http://pastebin.com/wUAAcdT7](http://pastebin.com/wUAAcdT7) > > [http://pastebin.com/wHyWc9U5](http://pastebin.com/wHyWc9U5) Do you have timer\_resolution configured? wbr, Valentin V. Bartenev ---
**Author:** ![Vedran\_FuraSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/f05b48/32.png) [@Vedran\_FuraSS](https://www.ruby-forum.com/u/Vedran_FuraSS)\ **Post date:** [July 24, 2015, 3:55pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/6 "2015-07-24T15:55:48Z")
## Valentin V. Bartenev Wrote: > interrupted by > > > servers): > > > > [http://pastebin.com/wUAAcdT7](http://pastebin.com/wUAAcdT7) > > > > [http://pastebin.com/wHyWc9U5](http://pastebin.com/wHyWc9U5) > > Do you have timer\_resolution configured? Yes, it’s: timer\_resolution 50ms; Posted at Nginx Forum: [http://forum.nginx.org/read.php?2,260512,260543#msg-260543](http://forum.nginx.org/read.php?2,260512,260543#msg-260543) ---
**Author:** ![Vedran\_FuraSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/f05b48/32.png) [@Vedran\_FuraSS](https://www.ruby-forum.com/u/Vedran_FuraSS)\ **Post date:** [July 24, 2015, 4:20pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/7 "2015-07-24T16:20:56Z")
On Friday 24 July 2015 09:55:04 vedranf wrote: > > > output but it is handled by the kernel and that writev(2) is interrupted by > > > > > > [http://pastebin.com/wHyWc9U5](http://pastebin.com/wHyWc9U5) > > > > Do you have timer\_resolution configured? > > Yes, it’s: > > timer\_resolution 50ms; This is the root cause of interrupts. Every 50ms it signals nginx and can interrupt any interruptible syscall (writing to file is usually not, but it seems different for Cephfs). You should avoid using timer\_resolution, or try this patch: diff -r be8d8b1dad78 src/os/unix/ngx\_files.c — a/src/os/unix/ngx\_files.c Fri Jul 24 17:18:20 2015 +0300 +++ b/src/os/unix/ngx\_files.c Fri Jul 24 17:18:37 2015 +0300 @@ -264,6 +264,7 @@ ngx\_write\_chain\_to\_file(ngx\_file\_t \*file u\_char \*prev; size\_t size; ssize\_t total, n; - ngx\_err\_t err; ngx\_array\_t vec; struct iovec \*iov, iovs[NGX\_IOVS]; @@ -335,10 +336,20 @@ ngx\_write\_chain\_to\_file(ngx\_file\_t \*file file-\>sys\_offset = offset; } +eintr: + n = writev(file-\>fd, vec.elts, vec.nelts); ``` if (n == -1) { ``` - ``` ngx_log_error(NGX_LOG_CRIT, file->log, ngx_errno, ``` - ``` err = ngx_errno; ``` - - ``` if (err == NGX_EINTR) { ``` - ``` ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, err, ``` - ``` "writev() was interrupted"); ``` - ``` goto eintr; ``` - ``` } ``` - - ``` ngx_log_error(NGX_LOG_CRIT, file->log, err, "writev() \"%s\" failed", file->name.data); return NGX_ERROR; } ``` ---
**Author:** ![Vedran\_FuraSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/f05b48/32.png) [@Vedran\_FuraSS](https://www.ruby-forum.com/u/Vedran_FuraSS)\ **Post date:** [July 24, 2015, 7:29pm UTC](https://www.ruby-forum.com/t/nginx-writing-to-cephfs/242741/8 "2015-07-24T19:29:01Z")
## Valentin V. Bartenev Wrote: > > > > always succeeded so it happens due to combination of nginx and > > You should avoid using timer\_resolution, or try this patch: Thank you, removing timer\_resolution fixed the problem. I’ll look into the patch in the next few days. Regards, Vedran Posted at Nginx Forum: [http://forum.nginx.org/read.php?2,260512,260550#msg-260550](http://forum.nginx.org/read.php?2,260512,260550#msg-260550)