Bug?

i keep getting:

2009/09/24 18:21:09 [crit] 11956#0: *50 pwrite() has written only 266240
of 524288, client: 10.100.0.115, server: contentdata.spacash.com,
request: “POST /store.php HTTP/1.1”, host: “10.100.0.155”

2009/09/24 18:22:19 [crit] 11956#0: *81 pwrite() has written only 225280
of 524288, client: 10.100.0.115, server: contentdata.spacash.com,
request: “POST /store.php HTTP/1.1”, host: “10.100.0.155”

store.php just receives files via POST and then moves them to the final
location via move_uploaded_file()

I tracked the string “pwrite() has written only” within nginx’s source
and got to:

src/os/unix/ngx_files.c where there is:

ssize_t
ngx_write_file(ngx_file_t *file, u_char *buf, size_t size, off_t offset)
{
ssize_t n;

ngx_log_debug4(NGX_LOG_DEBUG_CORE, file->log, 0,
               "write: %d, %p, %uz, %O", file->fd, buf, size,

offset);

#if (NGX_HAVE_PWRITE)

n = pwrite(file->fd, buf, size, offset);

if (n == -1) {
    ngx_log_error(NGX_LOG_CRIT, file->log, ngx_errno, "pwrite()

failed");
return NGX_ERROR;
}

if ((size_t) n != size) {
    ngx_log_error(NGX_LOG_CRIT, file->log, 0,
                  "pwrite() has written only %z of %uz", n, size);
    return NGX_ERROR;
}

#else

if (file->sys_offset != offset) {
    if (lseek(file->fd, offset, SEEK_SET) == -1) {
        ngx_log_error(NGX_LOG_CRIT, file->log, ngx_errno, "lseek()

failed");
return NGX_ERROR;
}

    file->sys_offset = offset;
}

n = write(file->fd, buf, size);
if (n == -1) {
    ngx_log_error(NGX_LOG_CRIT, file->log, ngx_errno, "write()

failed");
return NGX_ERROR;
}

if ((size_t) n != size) {
    ngx_log_error(NGX_LOG_CRIT, file->log, 0,
                  "write() has written only %z of %uz", n, size);
    return NGX_ERROR;
}

file->sys_offset += n;

#endif

file->offset += n;

return n;

}

Googling for pwrite, i found out that pwrite just isn’t to write all of
its data at once, so one has to put it into a loop.

So is this a bug or did i do something terribly wrong?

Thank you for your time.

Alejandro Martínez Lanfranco

On Thu, Sep 24, 2009 at 09:23:25PM +0200, Alejandro Mart??nez wrote:

store.php just receives files via POST and then moves them to the final
ssize_t n;
ngx_log_error(NGX_LOG_CRIT, file->log, ngx_errno, "pwrite()
#else

    return NGX_ERROR;

Googling for pwrite, i found out that pwrite just isn’t to write all of
its data at once, so one has to put it into a loop.

So is this a bug or did i do something terribly wrong?

pwrite() on file system should write all data at once (at least I see it
for the first time). What file system do you use ?

It’s seems the issue occures while writing client body in temporary
file,
I need to add file name in log message.

pwrite() on file system should write all data at once (at least I see it
for the first time). What file system do you use ?

Return Value
value pwrite() was successful. The value returned is the number of
bytes actually written. This number is less than or equal to nbyte.
-1 pwrite() was not successful. The errno global variable is set to
indicate the error.

I don’t see anywhere that the data should be written all at once. If it
was like that, then why whould it return the bytes written?

What file system do you use ?

from mount:

/dev/sda6 on /d1 type ext3 (rw)

Thank you for your time.

Alejandro Martínez Lanfranco

i forgot to mention.

im talking about BIG files.

up to 1Gb.

Files under 400Mg aprox work fine.

On another server it works fine even with the big ones.

On Thu, Sep 24, 2009 at 10:34:38PM +0200, Alejandro Mart??nez wrote:

I don’t see anywhere that the data should be written all at once. If it
was like that, then why whould it return the bytes written?

The partial or short write/pwrite/writev()s are usual for sockets,
pipes,
etc. However, they are not so usual for disk files and may be caused
by some limits: rlimits, qoutes, or free space. I think that retry
will not be successful, but will return errno explainig the cause.

What file system do you use ?

from mount:

/dev/sda6 on /d1 type ext3 (rw)

i forgot to mention.

im talking about BIG files.

up to 1Gb.

Files under 400Mg aprox work fine.

On another server it works fine even with the big ones.

What does “ulimit -f” show on these servers ?

On Fri, Sep 25, 2009 at 09:37:37AM +0400, Igor S. wrote:

indicate the error.

I don’t see anywhere that the data should be written all at once. If it
was like that, then why whould it return the bytes written?

The partial or short write/pwrite/writev()s are usual for sockets, pipes,
etc. However, they are not so usual for disk files and may be caused
by some limits: rlimits, qoutes, or free space. I think that retry
will not be successful, but will return errno explainig the cause.

Could you try the attached patch. It retries short write.

On Fri, Sep 25, 2009 at 10:10:45AM +0400, Igor S. wrote:

bytes actually written. This number is less than or equal to nbyte.

Could you try the attached patch. It retries short write.

Sorry, this patch is wrong. Try the new attached one.

What does “ulimit -f” show on these servers ?

Unlimited in both.

Igor S. wrote:

On Fri, Sep 25, 2009 at 10:10:45AM +0400, Igor S. wrote:

bytes actually written. This number is less than or equal to nbyte.

Could you try the attached patch. It retries short write.

Sorry, this patch is wrong. Try the new attached one.

We are getting there…

2009/09/25 12:54:37 [crit] 18192#0: *35 pwrite()
“/usr/local/nginx/client_body_temp/0000000002” has written only 274432
of 524288, client: 10.100.0.115, server: contentdata.spacash.com,
request: “POST /store.php HTTP/1.1”, host: “10.100.0.155”
2009/09/25 12:54:37 [crit] 18192#0: *35 pwrite()
“/usr/local/nginx/client_body_temp/0000000002” failed (28: No space left
on device), client: 10.100.0.115, server: contentdata.spacash.com,
request: “POST /store.php HTTP/1.1”, host: “10.100.0.155”

No space left on device. I thought setting the upload path at php.ini
would be enough, but it seems nginx first stores the files in that
“client_body_temp” directory and then moved to php’s temporary uploaded
file location.

Is that configurable for nginx? It would be nice if the “copy” could be
avoided, as i have the temporary uploaded file location por php in a
different partition (where i have plenty of space left).

Thank you for the patch, and your time.

Alejandro Martínez Lanfranco

Maxim D. wrote:

Hello!

On Fri, Sep 25, 2009 at 03:03:17PM +0200, Alejandro Martínez wrote:

on device), client: 10.100.0.115, server: contentdata.spacash.com,
request: “POST /store.php HTTP/1.1”, host: “10.100.0.155”

No space left on device. I thought setting the upload path at php.ini
would be enough, but it seems nginx first stores the files in that
“client_body_temp” directory and then moved to php’s temporary uploaded
file location.

Is that configurable for nginx? It would be nice if the “copy” could be
avoided, as i have the temporary uploaded file location por php in a
different partition (where i have plenty of space left).

It doesn’t “move”, but pass it to fastcgi application according
to fastcgi protocol (i.e. writes via socket). If you want to avoid
it you may consider using Valery K.'s upload module, see

NGINX 3rd Party Modules | NGINX

Note that this aproach requires special handling from your code.

Maxim D.

Nice. I’ll check it out.

As right now i have some time constraints, i replaced the
client_body_temp dir with a dynamic link to a partition with enough
space and it seems to work fine. But then i will give that module a try.

Thank you!

Alejandro Martínez Lanfranco

Hello!

On Fri, Sep 25, 2009 at 03:03:17PM +0200, Alejandro Martínez wrote:

on device), client: 10.100.0.115, server: contentdata.spacash.com,
request: “POST /store.php HTTP/1.1”, host: “10.100.0.155”

No space left on device. I thought setting the upload path at php.ini
would be enough, but it seems nginx first stores the files in that
“client_body_temp” directory and then moved to php’s temporary uploaded
file location.

Is that configurable for nginx? It would be nice if the “copy” could be
avoided, as i have the temporary uploaded file location por php in a
different partition (where i have plenty of space left).

It doesn’t “move”, but pass it to fastcgi application according
to fastcgi protocol (i.e. writes via socket). If you want to avoid
it you may consider using Valery K.'s upload module, see

Note that this aproach requires special handling from your code.

Maxim D.

Hello!

On Fri, Sep 25, 2009 at 03:57:31PM +0200, Alejandro Martínez wrote:

would be enough, but it seems nginx first stores the files in that

space and it seems to work fine. But then i will give that module a try.
I don’t really understand what means “replaced … with a dynamic
link” (symbolic link?), but just to make sure you know:

http://wiki.nginx.org/NginxHttpCoreModule#client_body_temp_path

Maxim D.

Maxim D. wrote:

Hello!

On Fri, Sep 25, 2009 at 03:57:31PM +0200, Alejandro Martínez wrote:

would be enough, but it seems nginx first stores the files in that

space and it seems to work fine. But then i will give that module a try.
I don’t really understand what means “replaced … with a dynamic
link” (symbolic link?), but just to make sure you know:

Module ngx_http_core_module

Maxim D.

Hah!

Yes. Stupid me :P. I meant a symbolic link.

But your suggestion, once again, is better.

Thank you again then :slight_smile:

Alejandro Martínez Lanfranco