have the same setup on two machines, identical configurations but the
newer machines reports the following a few times an hour:
2008/05/07 16:36:28 [error] 5969#0: *128011 memcached sent invalid
trailer while sending to client, client: 66.60.241.3, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
My google skills failed me. Is there a solution to this?
On Wed, May 07, 2008 at 04:44:02PM -0400, Sean A. wrote:
have the same setup on two machines, identical configurations but the
newer machines reports the following a few times an hour:2008/05/07 16:36:28 [error] 5969#0: *128011 memcached sent invalid
trailer while sending to client, client: 66.60.241.3, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”My google skills failed me. Is there a solution to this?
This may be nginx error or memcached error.
Both nginxes go to the same memcached ?
Is it reproduceable ? I.e. if you will request “/s-Pcpx300BT24p/lp” just
after you see error you see it again ?
On May 13, 2008, at 2:14 AM, Igor S. wrote:
My google skills failed me. Is there a solution to this?
This may be nginx error or memcached error.
Both nginxes go to the same memcached ?
Is it reproduceable ? I.e. if you will request “/s-Pcpx300BT24p/lp”
just
after you see error you see it again ?
different memcache installations. same version of memcache.
if i request, i dont get the error. so it has to be something related
in some fashion
to the client.
but i dont see anything special in any fashion about them… the 7
that came in while i was writing this:
151.213.86.255 - - [13/May/2008:16:24:41 -0400] “GET /s-Pcpx300BT24j/
lp HTTP/1.0” 200 24516 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows
NT 5.1; FunWebProducts)”
75.90.4.28 - - [13/May/2008:16:22:56 -0400] “GET /s-Pcpx300BT24p/lp
HTTP/1.0” 200 24516 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1)”
64.70.15.217 - - [13/May/2008:16:22:33 -0400] “GET /s-Pcpx300BT24p/lp
HTTP/1.1” 200 24516 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.1.4322)”
24.128.107.217 - - [13/May/2008:16:27:40 -0400] “GET /s-Pcpx300BT30K/
lp HTTP/1.0” 200 24516 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows
NT 5.1; SV1; .NET CLR 1.1.4322)”
72.231.196.6 - - [13/May/2008:16:28:52 -0400] “GET /s-Pcpx300BT24p/lp
HTTP/1.0” 200 24516 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; FunWebProducts)”
72.82.225.115 - - [13/May/2008:16:29:24 -0400] “GET /s-Pcpx300BT24p/lp
HTTP/1.0” 200 24516 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; FunWebProducts)”
72.88.124.28 - - [13/May/2008:16:30:52 -0400] “GET /s-Pcol300BT24j/lp
HTTP/1.1” 200 24516 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1)”
and the error logs:
2008/05/13 16:22:33 [error] 18511#0: *471 memcached sent invalid
trailer while sending to client, client: 64.70.15.217, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.1”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
2008/05/13 16:22:56 [error] 18511#0: *580 memcached sent invalid
trailer while sending to client, client: 75.90.4.28, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
2008/05/13 16:24:40 [error] 18511#0: *1022 memcached sent invalid
trailer while sending to client, client: 151.213.86.255, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24j/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
2008/05/13 16:27:40 [error] 18508#0: *2141 memcached sent invalid
trailer while sending to client, client: 24.128.107.217, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT30K/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
2008/05/13 16:28:52 [error] 18513#0: *2541 memcached sent invalid
trailer while sending to client, client: 72.231.196.6, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
2008/05/13 16:29:23 [error] 18511#0: *2738 memcached sent invalid
trailer while sending to client, client: 72.82.225.115, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
ō2008/05/13 16:30:52 [error] 18509#0: *3198 memcached sent invalid
trailer while sending to client, client: 72.88.124.28, server:
micro.thepinkpatch.com, request: “GET /s-Pcol300BT24j/lp HTTP/1.1”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
they are all coming from XP machines but other than that… not much
else i can see from it.
On Tue, May 13, 2008 at 04:34:32PM -0400, Sean A. wrote:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.0”,
lp HTTP/1.0" 200 24516 “-” "Mozilla/4.0 (compatible; MSIE 7.0; Windows
72.231.196.6 - - [13/May/2008:16:28:52 -0400] "GET /s-Pcpx300BT24p/lpmicro.thepinkpatch.com, request: “GET /s-Pcpx300BT24j/lp HTTP/1.0”,
trailer while sending to client, client: 72.82.225.115, server:
micro.thepinkpatch.com, request: “GET /s-Pcpx300BT24p/lp HTTP/1.0”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”
—??2008/05/13 16:30:52 [error] 18509#0: *3198 memcached sent invalid
trailer while sending to client, client: 72.88.124.28, server:
micro.thepinkpatch.com, request: “GET /s-Pcol300BT24j/lp HTTP/1.1”,
upstream: “memcached://127.0.0.1:11211”, host: “micro.thepinkpatch.com”they are all coming from XP machines but other than that… not much
else i can see from it.
It seems you have small number of request/s. Could you enable debug
logging
for several minutes until the bug will appear ?
It seems you have small number of request/s. Could you enable debug
logging
for several minutes until the bug will appear ?
have a small number of requests related to this problem, have a large
number overall.
i’m not familiar with debug logging, how do i enable that?
can i enable only on the error log ( as its small but the access logs
are about 250 megs
a day ).
On Wed, May 14, 2008 at 09:41:20AM -0400, Sean A. wrote:
a day ).
The log will be in error_log and it will be large. VERY LARGE.
So it should be set per some addresses only or if the load is low (as
in your case) for short time.
You need to build nginx using
./configure --with-debug …
Then you should enable debug log level in nginx.conf:
error_log /path/to/log debug;
Otherwise you may set debug per some addresses only:
events {
debug_connection 192.168.1.0/24;
debug_connection 192.168.10.32;
…
}
Disabled debug has very low CPU overhead.
On Thu, May 15, 2008 at 01:40:34PM -0400, Sean A. wrote:
number overall.
You need to build nginx using
debug_connection 192.168.1.0/24;
so this is going to end up lurking as i cant create the situation
anymore.
The “memcached sent invalid trailer” message means that after reading
specified number of bytes, nginx espects to see memcached trailer
“\r\nEND\r\n”. For some reason, nginx did not see the trailer, it may be
nginx bug, or software that put the item to memcached (I do not know
if it’s possible to put in memcached item with wrong length).
On May 14, 2008, at 3:26 PM, Igor S. wrote:
can i enable only on the error log ( as its small but the access logs
}
Disabled debug has very low CPU overhead.
damn. the advertiser running the banners that caused the problem
finally fixed an outstanding issue on their end
and it fixed the problem on our end. how the two were related is a
little beyond me but apparently they were
so this is going to end up lurking as i cant create the situation
anymore.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs