Alert: pread() read only

Server NGINX
#1

Hello,

We are running a Joomla website loading a google map in an iframe (under
NGINX) in the main (home) web page. This is the page mostly visited as
it contains almost real-time data to be viewed by clients.

The system info:

Linux myserver.example.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28
17:19:38 UTC 2013 x86_64
Database Version 5.5.34
Database Collation utf8_general_ci
PHP Version 5.3.3
Web Server nginx/1.4.2
WebServer to PHP Interface fpm-fcgi
Joomla! Version Joomla! 2.5.8 Stable [ Ember ] 8-November-2012 14:00
GMT
Joomla! Platform Version Joomla Platform 11.4.0 Stable [ Brian
Kernighan ] 03-Jan-2012 00:00 GMT

The problem is that there is a repeating error of the form (I have
changed real host name and web root path, as well as client IP address):

2013/11/17 12:39:14 [alert] 20709#0: *9059 pread() read only 38605 of
39107 from “/path/to/web/root/HTML/gmap/gmapv3_auto_el.html” while
sending response to client, client: ::ffff:xxx.xxx.241.42, server:
www.example.com, request: “GET /HTML/gmap/gmapv3_auto_el.html HTTP/1.1”,
host: “www.example.com”

I found here:

http://translate.google.com/translate?sl=ru&tl=en&js=n&prev=_t&hl=pt-PT&ie=UTF-8&u=http%3A%2F%2Fforum.nginx.org%2Fread.php%3F21%2C9856&act=url

…that this is probably related to the “open_file_cache” directive, and
in fact I do use (based on advice found on the Internet):

open_file_cache max=5000 inactive=30s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

Is there anything I could/should do to optimize system operation to
avoid errors?

Should I disable open_file_cache or not? (I do not entirely understand
its implications.)

How should we determine the directive benefits?

Note: The aim is to be able to serve a few thousand requests per sec at
peak, while normal traffic is < 20 reqs/sec.

I appreciate your suggestions.

Regards,
Nick

#2

On 19/11/2013 9:39 πμ, Nikolaos M. wrote:

The system info:

Linux myserver.example.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug
28 17:19:38 UTC 2013 x86_64

I forgot to mention that this is a VPS running CentOS 6.4 as a VM under
KVM. The file system at the VM is ext4 over LVM but I don’t know the
actual storage topology of the VM host cluster.

I thought I should add this info as it may be pertinent to the
situation.

Regards,
Nick

#3

On Tue, Nov 19, 2013 at 09:39:44AM +0200, Nikolaos M. wrote:

Hi there,

2013/11/17 12:39:14 [alert] 20709#0: *9059 pread() read only 38605 of
39107 from “/path/to/web/root/HTML/gmap/gmapv3_auto_el.html”

…that this is probably related to the “open_file_cache” directive, and
in fact I do use (based on advice found on the Internet):

open_file_cache max=5000 inactive=30s;

Is there anything I could/should do to optimize system operation to
avoid errors?

Don’t change files that nginx has in open_file_cache.

Should I disable open_file_cache or not? (I do not entirely understand
its implications.)

open_file_cache says “I am happy to occasionally return stale data or
errors, in exchange for the speed-up in normal use”.

The errors will happen when a file size is changed in-place – so “mv
a.html b.html && echo new > a.html” should continue to serve the old
content of a.html; but just “echo new > a.html” will show something odd
if the old length of a.html was not exactly 4.

How should we determine the directive benefits?

Measure.

Only you can do a test against your hardware. Get some clients (on
remote
machines) and make the requests and examine the responses.

Note that you later say that you are using a virtual machine on a shared
system. That suggests that the actual hardware available to nginx will
vary over time, so any tests will not be valid for any other time.

You also describe how the file system is set up. I can’t tell whether
that will necessarily break open_file_cache, even if the files
themselves
are not changing underneath nginx.

If you know that the files are not changing but the pread() errors
persist, then disable open_file_cache to see if it stops the errors,
and then it my be worth investigating whether that filesystem set up is
as stable as nginx expects.

f

Francis D. [email protected]

#4

On Tue, Nov 19, 2013 at 09:19:19AM +0000, Francis D. wrote:

On Tue, Nov 19, 2013 at 09:39:44AM +0200, Nikolaos M. wrote:

Hi there,

Oh, and following Maxim’s mail…

The errors will happen when a file size is changed in-place – so “mv
a.html b.html && echo new > a.html” should continue to serve the old
content of a.html; but just “echo new > a.html” will show something odd
if the old length of a.html was not exactly 4.

…for later requests, that started after the file was changed. For a
request ongoing while the file is changed, odd things can be shown even
if the file size stays the same.

f

Francis D. [email protected]

#5

On 11/19/2013 08:39 AM, Nikolaos M. wrote:

Database Version 5.5.34
Database Collation utf8_general_ci
PHP Version 5.3.3
Web Server nginx/1.4.2
WebServer to PHP Interface fpm-fcgi
Joomla! Version Joomla! 2.5.8 Stable [ Ember ] 8-November-2012 14:00
GMT

Looking at your Joomla version, if this is an Internet facing server, it
will be 0wned once they have probed your box. You might want to update
your Joomla install asap to the latest version (currently 2.5.16) to
plug all the root exploits.

Regards,
Patrick

#6

On 19/11/2013 11:21 πμ, Maxim D. wrote:

I don’t think that open_file_cache results in a measurable
difference in your case. I would recommend disabling it unless
you have good reasons to enable it, just to simplify maintenance.

Thank you all for your suggestions.

It seems that disabling open_file_cache, stops these errors.

Thanks again,
Nick

#7

Hello!

On Tue, Nov 19, 2013 at 09:39:44AM +0200, Nikolaos M. wrote:

[…]

I found here:

http://translate.google.com/translate?sl=ru&tl=en&js=n&prev=_t&hl=pt-PT&ie=UTF-8&u=http%3A%2F%2Fforum.nginx.org%2Fread.php%3F21%2C9856&act=url

…that this is probably related to the “open_file_cache” directive,
and in fact I do use (based on advice found on the Internet):

It’s not “related” to the “open_file_cache” directive, but the
“open_file_cache” directive makes the underlying problem more
visible.

The root cause of the messages in question is non-atomic file
update. Somebody on your system edited the file in question
in-place, instead of re-creating it with a temporary name and then
using “mv” to atomically update the file.

Non-atomic updates create a race: a file which is opened by nginx
(and stat()'ed for nginx to know it’s length) suddenly changes.
This can happen in the middle of a response, and results in a
corrupted response - first part of a response is from original
file, and second is from updated one. If nginx is able to detect
the problem due to file size mismatch - it logs the message in
question.

The only correct solution is to update files atomically, i.e.,
create a new file and then rename to a desired name.

However, the “open_file_cache” directive makes the race window
bigger by keeping files open for a long time. Switching it off is
a good idea if you can’t eliminate non-atomic updates for some
reason.

[…]

Should I disable open_file_cache or not? (I do not entirely
understand its implications.)

How should we determine the directive benefits?

Note: The aim is to be able to serve a few thousand requests per sec
at peak, while normal traffic is < 20 reqs/sec.

I don’t think that open_file_cache results in a measurable
difference in your case. I would recommend disabling it unless
you have good reasons to enable it, just to simplify maintenance.

–
Maxim D.
http://nginx.org/en/donation.html