Md5 collision

Server NGINX
1

Hi,
Looking for help with a high number of md5 collisions with proxy
caching.
It’s my understanding that we shouldn’t be getting any, but we seem to
be
getting quite a few.

-V
nginx version: nginx/1.2.3
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
configure arguments: --prefix=/opt/nginx --with-http_stub_status_module
–with-pcre=…/pcre-8.31 --without-http_access_module
–without-http_auth_basic_module --without-http_autoindex_module
–without-http_browser_module --without-http_charset_module
–without-http_fastcgi_module --without-http_geo_module
–without-http_memcached_module --without-http_referer_module
–without-http_scgi_module --without-http_split_clients_module
–without-http_ssi_module --without-http_upstream_ip_hash_module
–without-http_userid_module --without-http_uwsgi_module
–add-module=…/agentzh-headers-more-nginx-module-278ba7d
–conf-path=nginx.conf --error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log --with-debug

I believe this might be relevant as well:

checking for system md library … not found
checking for system md5 library … not found
checking for OpenSSL md5 crypto library … found
checking for zlib library … found
creating objs/Makefile

Configuration summary

  • using PCRE library: …/pcre-8.31
  • OpenSSL library is not used
  • md5: using system crypto library
  • sha1 library is not used
  • using system zlib library

We are using the default cace key, though I’ve tried a few variations.
We
are doing one “weird” thing…The upstream server might re-request a
slightly modified version from Nginx in the hopes that this modified
version
is in the cache. If it isn’t, this modified request goes back to the
upstream, which then returns a file (which then goes back to the
upstream to
fulfill the original request). At most, this is 2 loops… NGINX →
Upstream → NGINX → Upstream but the URLs wil be different between
the
first set and the second one. Not sure that this is a problem, but since
it’s the only odd thing, I figured I’d mention it.

Cheers,
Karl

Posted at Nginx Forum:

2

Hello!

On Thu, Sep 13, 2012 at 07:07:09AM -0400, karlseguin wrote:

Hi,
Looking for help with a high number of md5 collisions with proxy caching.
It’s my understanding that we shouldn’t be getting any, but we seem to be
getting quite a few.

What makes you think that there are md5 collisions? Do you see
nginx complaining in error log?

-V
nginx version: nginx/1.2.3
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)

Uhm, am I right assuming it’s on ARM?

–conf-path=nginx.conf --error-log-path=/var/log/nginx/error.log

Configuration summary

  • using PCRE library: …/pcre-8.31
  • OpenSSL library is not used
  • md5: using system crypto library
  • sha1 library is not used
  • using system zlib library

So nginx is using system crypto library (aka OpenSSL’s crypto
library) and it seems to do wrong thing. Could you please try to
force nginx’s own md5 implementation with

./configure --with-cc-opt=“-DNGX_HAVE_MD5=0” …

to see if it helps?

You may also want to produce debug log, see
Debugging | NGINX for details.

Maxim D.

3

Hello!

On Thu, Sep 13, 2012 at 11:21:27AM -0400, karlseguin wrote:

Getting OpenSSL to handle md5 didn’t change anything

As copilation long as

(wasn’t able to get it
to compile with your specific flag, but I figured this was what you had
intended?)

With the flag in question (–with-cc-opt=“-DNGX_HAVE_MD5=0”) nginx
should compile with internal md5 implementation. If it doesn’t -
you may want to provide more details.

2012/09/13 15:15:16 [debug] 27367#0: *3441 add cleanup: 000000000196CEC0
2012/09/13 15:15:16 [debug] 27367#0: *3441 http file cache fd: 24
2012/09/13 15:15:16 [debug] 27367#0: *3441 read: 24, 000000000196D198, 285,
0
2012/09/13 15:15:16 [crit] 27367#0: *3441 cache file
“/mnt/tmp/nginx/cache/90/6d/2b/422c25423170bac1c4e655ed9c2b6d90” has md5
collision, client: 127.0.0.1, server: atlas-x.example.com, request: “GET
/uploads/channel/thumbnail/5927/e74e97624ef652bc7a8419c02f8a7f92ae71d127.jpg
HTTP/1.1”, host: “127.0.0.1”
2012/09/13 15:15:16 [debug] 27367#0: *3441 http upstream cache: -5

Calculated md5 looks fine.

Do you see the same problem if you re-request the resource in
question?

If yes - what’s in the cache file? Intresting part is first bytes
and the KEY string. Note that first bytes are binary, please use
hd/xxd to show them.

If no - how the cache was populated? Note that cache files aren’t
portable and you can’t move the cache from one host to another.

Maxim D.

4

Yes, re-requesting the same image will cause the same error (can’t say
with
100% certainty that it’ll always do it though).

sudo hd /mnt/tmp/nginx/cache/5f/85/1b/56cc2e05fb85eb0a806444f1811b855f
00000000 ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01
|…JFIF…|

Naively, there doesn’t seem to be any special header bytes?

Relevant config is:

upstream origin_magick {
server 127.0.0.1:4000;
keepalive 50;
}

proxy_cache_path /mnt/tmp/nginx/cache levels=2:2:2 keys_zone=main:200m
max_size=100G;
proxy_cache_use_stale updating;
proxy_http_version 1.1;
proxy_cache_valid any 1m;
more_clear_headers ‘X-amz-’ 'x-amz-’ ‘X-Content-Digest’
'X-Rack-Cache;

location / {
  proxy_cache_bypass $http_nocache;
  proxy_cache main;
  proxy_pass http://origin_magick;
  more_set_headers -s '404 403' 'Cache-Control: public,

max-age=120’;
}

Posted at Nginx Forum:

5

Hello!

On Thu, Sep 13, 2012 at 07:43:16PM -0400, karlseguin wrote:

Yes, re-requesting the same image will cause the same error (can’t say with
100% certainty that it’ll always do it though).

sudo hd /mnt/tmp/nginx/cache/5f/85/1b/56cc2e05fb85eb0a806444f1811b855f
00000000 ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01
|…JFIF…|

Naively, there doesn’t seem to be any special header bytes?

This looks like a completely corrupted cache file, without any
headers at all.

proxy_http_version 1.1;
proxy_cache_valid any 1m;
more_clear_headers ‘X-amz-’ 'x-amz-’ ‘X-Content-Digest’ 'X-Rack-Cache;

location / {
  proxy_cache_bypass $http_nocache;
  proxy_cache main;
  proxy_pass http://origin_magick;
  more_set_headers -s '404 403' 'Cache-Control: public, max-age=120';
}

Please try removing the cache file in question (move it away or
just delete) and re-request it twice (once to cache the response,
and once to see the error) to see if problem persists. Please
also make sure you are able to reproduce the problem without any
3rd party modules/patches (and preferably with latest nginx
1.3.6).

If the problem persists, please follow
Debugging | NGINX and provide minimal full config
you are able to reproduce the probem with, and full debug log
(since file removal).

Maxim D.

6

Getting OpenSSL to handle md5 didn’t change anything (wasn’t able to get
it
to compile with your specific flag, but I figured this was what you had
intended?)

Here’s a sample from the debug log:

2012/09/13 15:15:16 [debug] 27367#0: *3441 http cache key:
“http://origin_magick”
2012/09/13 15:15:16 [debug] 27367#0: *3441 http cache key:
“/uploads/channel/thumbnail/5927/e74e97624ef652bc7a8419c02f8a7f92ae71d127.jpg”
2012/09/13 15:15:16 [debug] 27367#0: *3441 add cleanup: 000000000196CE68
2012/09/13 15:15:16 [debug] 27367#0: *3441 http file cache exists: 0 e:1
2012/09/13 15:15:16 [debug] 27367#0: *3441 cache file:
“/mnt/tmp/nginx/cache/90/6d/2b/422c25423170bac1c4e655ed9c2b6d90”
2012/09/13 15:15:16 [debug] 27367#0: *3441 add cleanup: 000000000196CEC0
2012/09/13 15:15:16 [debug] 27367#0: *3441 http file cache fd: 24
2012/09/13 15:15:16 [debug] 27367#0: *3441 read: 24, 000000000196D198,
285,
0
2012/09/13 15:15:16 [crit] 27367#0: *3441 cache file
“/mnt/tmp/nginx/cache/90/6d/2b/422c25423170bac1c4e655ed9c2b6d90” has md5
collision, client: 127.0.0.1, server: atlas-x.example.com, request: “GET
/uploads/channel/thumbnail/5927/e74e97624ef652bc7a8419c02f8a7f92ae71d127.jpg
HTTP/1.1”, host: “127.0.0.1”
2012/09/13 15:15:16 [debug] 27367#0: *3441 http upstream cache: -5

Posted at Nginx Forum: