Forum: NGINX alert: pread() read only

34c7b0659e6361c2bbbdbfa6ca597cc0?d=identicon&s=25 Nikolaos Milas (Guest)
on 2013-11-19 08:40
(Received via mailing list)
Hello,

We are running a Joomla website loading a google map in an iframe (under
NGINX) in the main (home) web page. This is the page mostly visited as
it contains almost real-time data to be viewed by clients.

The system info:

Linux myserver.example.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28
17:19:38 UTC 2013 x86_64
Database Version     5.5.34
Database Collation     utf8_general_ci
PHP Version     5.3.3
Web Server     nginx/1.4.2
WebServer to PHP Interface     fpm-fcgi
Joomla! Version     Joomla! 2.5.8 Stable [ Ember ] 8-November-2012 14:00
GMT
Joomla! Platform Version     Joomla Platform 11.4.0 Stable [ Brian
Kernighan ] 03-Jan-2012 00:00 GMT

The problem is that there is a repeating error of the form (I have
changed real host name and web root path, as well as client IP address):

2013/11/17 12:39:14 [alert] 20709#0: *9059 pread() read only 38605 of
39107 from "/path/to/web/root/HTML/gmap/gmapv3_auto_el.html" while
sending response to client, client: ::ffff:xxx.xxx.241.42, server:
www.example.com, request: "GET /HTML/gmap/gmapv3_auto_el.html HTTP/1.1",
host: "www.example.com"

I found here:

http://translate.google.com/translate?sl=ru&tl=en&...

...that this is probably related to the "open_file_cache" directive, and
in fact I do use (based on advice found on the Internet):

    open_file_cache max=5000 inactive=30s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

Is there anything I could/should do to optimize system operation to
avoid errors?

Should I disable open_file_cache or not? (I do not entirely understand
its implications.)

How should we determine the directive benefits?

Note: The aim is to be able to serve a few thousand requests per sec at
peak, while normal traffic is < 20 reqs/sec.

I appreciate your suggestions.

Regards,
Nick
34c7b0659e6361c2bbbdbfa6ca597cc0?d=identicon&s=25 Nikolaos Milas (Guest)
on 2013-11-19 08:48
(Received via mailing list)
On 19/11/2013 9:39 πμ, Nikolaos Milas wrote:

> The system info:
>
> Linux myserver.example.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug
> 28 17:19:38 UTC 2013 x86_64

I forgot to mention that this is a VPS running CentOS 6.4 as a VM under
KVM. The file system at the VM is ext4 over LVM but I don't know the
actual storage topology of the VM host cluster.

I thought I should add this info as it may be pertinent to the
situation.

Regards,
Nick
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-19 10:19
(Received via mailing list)
On Tue, Nov 19, 2013 at 09:39:44AM +0200, Nikolaos Milas wrote:

Hi there,

> 2013/11/17 12:39:14 [alert] 20709#0: *9059 pread() read only 38605 of
> 39107 from "/path/to/web/root/HTML/gmap/gmapv3_auto_el.html"

> ...that this is probably related to the "open_file_cache" directive, and
> in fact I do use (based on advice found on the Internet):
>
>    open_file_cache max=5000 inactive=30s;

> Is there anything I could/should do to optimize system operation to
> avoid errors?

Don't change files that nginx has in open_file_cache.

> Should I disable open_file_cache or not? (I do not entirely understand
> its implications.)

open_file_cache says "I am happy to occasionally return stale data or
errors, in exchange for the speed-up in normal use".

The errors will happen when a file size is changed in-place -- so "mv
a.html b.html && echo new > a.html" should continue to serve the old
content of a.html; but just "echo new > a.html" will show something odd
if the old length of a.html was not exactly 4.

> How should we determine the directive benefits?

Measure.

Only you can do a test against your hardware. Get some clients (on
remote
machines) and make the requests and examine the responses.

Note that you later say that you are using a virtual machine on a shared
system. That suggests that the actual hardware available to nginx will
vary over time, so any tests will not be valid for any other time.

You also describe how the file system is set up. I can't tell whether
that will necessarily break open_file_cache, even if the files
themselves
are not changing underneath nginx.

If you know that the files are not changing but the pread() errors
persist, then disable open_file_cache to see if it stops the errors,
and then it my be worth investigating whether that filesystem set up is
as stable as nginx expects.

  f
--
Francis Daly        francis@daoine.org
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2013-11-19 10:21
(Received via mailing list)
Hello!

On Tue, Nov 19, 2013 at 09:39:44AM +0200, Nikolaos Milas wrote:

[...]

> I found here:
>
>
http://translate.google.com/translate?sl=ru&tl=en&...
>
> ...that this is probably related to the "open_file_cache" directive,
> and in fact I do use (based on advice found on the Internet):

It's not "related" to the "open_file_cache" directive, but the
"open_file_cache" directive makes the underlying problem more
visible.

The root cause of the messages in question is non-atomic file
update.  Somebody on your system edited the file in question
in-place, instead of re-creating it with a temporary name and then
using "mv" to atomically update the file.

Non-atomic updates create a race: a file which is opened by nginx
(and stat()'ed for nginx to know it's length) suddenly changes.
This can happen in the middle of a response, and results in a
corrupted response - first part of a response is from original
file, and second is from updated one.  If nginx is able to detect
the problem due to file size mismatch - it logs the message in
question.

The only correct solution is to update files atomically, i.e.,
create a new file and then rename to a desired name.

However, the "open_file_cache" directive makes the race window
bigger by keeping files open for a long time.  Switching it off is
a good idea if you can't eliminate non-atomic updates for some
reason.

[...]

> Should I disable open_file_cache or not? (I do not entirely
> understand its implications.)
>
> How should we determine the directive benefits?
>
> Note: The aim is to be able to serve a few thousand requests per sec
> at peak, while normal traffic is < 20 reqs/sec.

I don't think that open_file_cache results in a measurable
difference in your case.  I would recommend disabling it unless
you have good reasons to enable it, just to simplify maintenance.

--
Maxim Dounin
http://nginx.org/en/donation.html
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-11-19 10:38
(Received via mailing list)
On Tue, Nov 19, 2013 at 09:19:19AM +0000, Francis Daly wrote:
> On Tue, Nov 19, 2013 at 09:39:44AM +0200, Nikolaos Milas wrote:

Hi there,

Oh, and following Maxim's mail...

> The errors will happen when a file size is changed in-place -- so "mv
> a.html b.html && echo new > a.html" should continue to serve the old
> content of a.html; but just "echo new > a.html" will show something odd
> if the old length of a.html was not exactly 4.

...for later requests, that started after the file was changed. For a
request ongoing while the file is changed, odd things can be shown even
if the file size stays the same.

  f
--
Francis Daly        francis@daoine.org
A9ea944f3af26e37f02fd08468552edb?d=identicon&s=25 Patrick Lists (Guest)
on 2013-11-19 14:27
(Received via mailing list)
On 11/19/2013 08:39 AM, Nikolaos Milas wrote:
> Database Version     5.5.34
> Database Collation     utf8_general_ci
> PHP Version     5.3.3
> Web Server     nginx/1.4.2
> WebServer to PHP Interface     fpm-fcgi
> Joomla! Version     Joomla! 2.5.8 Stable [ Ember ] 8-November-2012 14:00
> GMT

Looking at your Joomla version, if this is an Internet facing server, it
will be 0wned once they have probed your box. You might want to update
your Joomla install asap to the latest version (currently 2.5.16) to
plug all the root exploits.

Regards,
Patrick
34c7b0659e6361c2bbbdbfa6ca597cc0?d=identicon&s=25 Nikolaos Milas (Guest)
on 2013-11-19 15:37
(Received via mailing list)
On 19/11/2013 11:21 πμ, Maxim Dounin wrote:

> I don't think that open_file_cache results in a measurable
> difference in your case.  I would recommend disabling it unless
> you have good reasons to enable it, just to simplify maintenance.

Thank you all for your suggestions.

It seems that disabling open_file_cache, stops these errors.

Thanks again,
Nick
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.