# Md5 collision

**URL:** <https://www.ruby-forum.com/t/md5-collision/222873>\
**Category:** NGINX\
**Created:** [September 13, 2012, 1:07pm UTC](https://www.ruby-forum.com/t/md5-collision/222873 "2012-09-13T13:07:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![aris](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/258eb7/32.png) [@aris](https://www.ruby-forum.com/u/aris)\
**Post date:** [September 13, 2012, 1:07pm UTC](https://www.ruby-forum.com/t/md5-collision/222873/1 "2012-09-13T13:07:50Z")

</div>

Hi,  
Looking for help with a high number of md5 collisions with proxy  
caching.  
It’s my understanding that we shouldn’t be getting any, but we seem to  
be  
getting quite a few.

-V  
nginx version: nginx/1.2.3  
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)  
configure arguments: --prefix=/opt/nginx --with-http\_stub\_status\_module  
–with-pcre=…/pcre-8.31 --without-http\_access\_module  
–without-http\_auth\_basic\_module --without-http\_autoindex\_module  
–without-http\_browser\_module --without-http\_charset\_module  
–without-http\_fastcgi\_module --without-http\_geo\_module  
–without-http\_memcached\_module --without-http\_referer\_module  
–without-http\_scgi\_module --without-http\_split\_clients\_module  
–without-http\_ssi\_module --without-http\_upstream\_ip\_hash\_module  
–without-http\_userid\_module --without-http\_uwsgi\_module  
–add-module=…/agentzh-headers-more-nginx-module-278ba7d  
–conf-path=nginx.conf --error-log-path=/var/log/nginx/error.log  
–http-log-path=/var/log/nginx/access.log --with-debug

I believe this might be relevant as well:

checking for system md library … not found  
checking for system md5 library … not found  
checking for OpenSSL md5 crypto library … found  
checking for zlib library … found  
creating objs/Makefile

Configuration summary

- using PCRE library: …/pcre-8.31
- OpenSSL library is not used
- md5: using system crypto library
- sha1 library is not used
- using system zlib library

We are using the default cace key, though I’ve tried a few variations.  
We  
are doing one “weird” thing…The upstream server might re-request a  
slightly modified version from Nginx in the hopes that this modified  
version  
is in the cache. If it isn’t, this modified request goes back to the  
upstream, which then returns a file (which then goes back to the  
upstream to  
fulfill the original request). At most, this is 2 loops… NGINX →   
Upstream → NGINX → Upstream but the URLs wil be different between  
the  
first set and the second one. Not sure that this is a problem, but since  
it’s the only odd thing, I figured I’d mention it.

Cheers,  
Karl

Posted at Nginx Forum:

> **[md5 collision](https://forum.nginx.org/read.php?2,230686,230686#msg-230686)**
>
> Hi, Looking for help with a high number of md5 collisions with proxy caching. It's my understanding that we shouldn't be getting any, but we seem to be getting quite a few. -V nginx version: nginx/1.2.3 built by gcc 4.6.3 (Ubuntu/Linaro...

---

<div class="post-metadata">

**Author:** ![karlseguin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b3f665/32.png) [@karlseguin](https://www.ruby-forum.com/u/karlseguin)\
**Post date:** [September 13, 2012, 2:49pm UTC](https://www.ruby-forum.com/t/md5-collision/222873/2 "2012-09-13T14:49:04Z")

</div>

Hello!

On Thu, Sep 13, 2012 at 07:07:09AM -0400, karlseguin wrote:

> Hi,  
> Looking for help with a high number of md5 collisions with proxy caching.  
> It’s my understanding that we shouldn’t be getting any, but we seem to be  
> getting quite a few.

What makes you think that there are md5 collisions? Do you see  
nginx complaining in error log?

> -V  
> nginx version: nginx/1.2.3  
> built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)

Uhm, am I right assuming it’s on ARM?

> –conf-path=nginx.conf --error-log-path=/var/log/nginx/error.log
> 
> Configuration summary
> 
> - using PCRE library: …/pcre-8.31
> - OpenSSL library is not used
> - md5: using system crypto library
> - sha1 library is not used
> - using system zlib library

So nginx is using system crypto library (aka OpenSSL’s crypto  
library) and it seems to do wrong thing. Could you please try to  
force nginx’s own md5 implementation with

./configure --with-cc-opt=“-DNGX\_HAVE\_MD5=0” …

to see if it helps?

You may also want to produce debug log, see  
[Debugging | NGINX](http://wiki.nginx.org/Debugging) for details.

Maxim D.

---

<div class="post-metadata">

**Author:** ![karlseguin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b3f665/32.png) [@karlseguin](https://www.ruby-forum.com/u/karlseguin)\
**Post date:** [September 13, 2012, 10:15pm UTC](https://www.ruby-forum.com/t/md5-collision/222873/3 "2012-09-13T22:15:10Z")

</div>

Hello!

On Thu, Sep 13, 2012 at 11:21:27AM -0400, karlseguin wrote:

> Getting OpenSSL to handle md5 didn’t change anything

As copilation long as

> (wasn’t able to get it  
> to compile with your specific flag, but I figured this was what you had  
> intended?)

With the flag in question (–with-cc-opt=“-DNGX\_HAVE\_MD5=0”) nginx  
should compile with internal md5 implementation. If it doesn’t -  
you may want to provide more details.

> 2012/09/13 15:15:16 [debug] 27367#0: \*3441 add cleanup: 000000000196CEC0  
> 2012/09/13 15:15:16 [debug] 27367#0: \*3441 http file cache fd: 24  
> 2012/09/13 15:15:16 [debug] 27367#0: \*3441 read: 24, 000000000196D198, 285,  
> 0  
> 2012/09/13 15:15:16 [crit] 27367#0: \*3441 cache file  
> “/mnt/tmp/nginx/cache/90/6d/2b/422c25423170bac1c4e655ed9c2b6d90” has md5  
> collision, client: 127.0.0.1, server: [atlas-x.example.com](http://atlas-x.example.com), request: “GET  
> /uploads/channel/thumbnail/5927/e74e97624ef652bc7a8419c02f8a7f92ae71d127.jpg  
> HTTP/1.1”, host: “127.0.0.1”  
> 2012/09/13 15:15:16 [debug] 27367#0: \*3441 http upstream cache: -5

Calculated md5 looks fine.

Do you see the same problem if you re-request the resource in  
question?

If yes - what’s in the cache file? Intresting part is first bytes  
and the KEY string. Note that first bytes are binary, please use  
hd/xxd to show them.

If no - how the cache was populated? Note that cache files aren’t  
portable and you can’t move the cache from one host to another.

Maxim D.

---

<div class="post-metadata">

**Author:** ![karlseguin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b3f665/32.png) [@karlseguin](https://www.ruby-forum.com/u/karlseguin)\
**Post date:** [September 14, 2012, 1:43am UTC](https://www.ruby-forum.com/t/md5-collision/222873/4 "2012-09-14T01:43:49Z")

</div>

Yes, re-requesting the same image will cause the same error (can’t say  
with  
100% certainty that it’ll always do it though).

sudo hd /mnt/tmp/nginx/cache/5f/85/1b/56cc2e05fb85eb0a806444f1811b855f  
00000000 ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01  
|…JFIF…|

Naively, there doesn’t seem to be any special header bytes?

Relevant config is:

upstream origin\_magick {  
server 127.0.0.1:4000;  
keepalive 50;  
}

proxy\_cache\_path /mnt/tmp/nginx/cache levels=2:2:2 keys\_zone=main:200m  
max\_size=100G;  
proxy\_cache\_use\_stale updating;  
proxy\_http\_version 1.1;  
proxy\_cache\_valid any 1m;  
more\_clear\_headers ‘X-amz-_’ 'x-amz-_’ ‘X-Content-Digest’  
'X-Rack-Cache;

```
location / {
  proxy_cache_bypass $http_nocache;
  proxy_cache main;
  proxy_pass http://origin_magick;
  more_set_headers -s '404 403' 'Cache-Control: public, 

```

max-age=120’;  
}

Posted at Nginx Forum:

> **[Re: md5 collision](https://forum.nginx.org/read.php?2,230686,230714#msg-230714)**
>
> Hi, Looking for help with a high number of md5 collisions with proxy caching. It's my understanding that we shouldn't be getting any, but we seem to be getting quite a few. -V nginx version: nginx/1.2.3 built by gcc 4.6.3 (Ubuntu/Linaro...

---

<div class="post-metadata">

**Author:** ![karlseguin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b3f665/32.png) [@karlseguin](https://www.ruby-forum.com/u/karlseguin)\
**Post date:** [September 14, 2012, 1:15pm UTC](https://www.ruby-forum.com/t/md5-collision/222873/5 "2012-09-14T13:15:40Z")

</div>

Hello!

On Thu, Sep 13, 2012 at 07:43:16PM -0400, karlseguin wrote:

> Yes, re-requesting the same image will cause the same error (can’t say with  
> 100% certainty that it’ll always do it though).
> 
> sudo hd /mnt/tmp/nginx/cache/5f/85/1b/56cc2e05fb85eb0a806444f1811b855f  
> 00000000 ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01  
> |…JFIF…|
> 
> Naively, there doesn’t seem to be any special header bytes?

This looks like a completely corrupted cache file, without any  
headers at all.

> proxy\_http\_version 1.1;  
> proxy\_cache\_valid any 1m;  
> more\_clear\_headers ‘X-amz-_’ 'x-amz-_’ ‘X-Content-Digest’ 'X-Rack-Cache;
> 
> ```
> location / {
> proxy_cache_bypass $http_nocache;
> proxy_cache main;
> proxy_pass http://origin_magick;
> more_set_headers -s '404 403' 'Cache-Control: public, max-age=120';
> }
> 
> ```

Please try removing the cache file in question (move it away or  
just delete) and re-request it twice (once to cache the response,  
and once to see the error) to see if problem persists. Please  
also make sure you are able to reproduce the problem without any  
3rd party modules/patches (and preferably with latest nginx  
1.3.6).

If the problem persists, please follow  
[Debugging | NGINX](http://wiki.nginx.org/Debugging) and provide minimal full config  
you are able to reproduce the probem with, and full debug log  
(since file removal).

Maxim D.

---

<div class="post-metadata">

**Author:** ![karlseguin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b3f665/32.png) [@karlseguin](https://www.ruby-forum.com/u/karlseguin)\
**Post date:** [September 13, 2012, 5:21pm UTC](https://www.ruby-forum.com/t/md5-collision/222873/6 "2012-09-13T17:21:48Z")

</div>

Getting OpenSSL to handle md5 didn’t change anything (wasn’t able to get  
it  
to compile with your specific flag, but I figured this was what you had  
intended?)

Here’s a sample from the debug log:

2012/09/13 15:15:16 [debug] 27367#0: \*3441 http cache key:  
“http://origin\_magick”  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 http cache key:  
“/uploads/channel/thumbnail/5927/e74e97624ef652bc7a8419c02f8a7f92ae71d127.jpg”  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 add cleanup: 000000000196CE68  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 http file cache exists: 0 e:1  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 cache file:  
“/mnt/tmp/nginx/cache/90/6d/2b/422c25423170bac1c4e655ed9c2b6d90”  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 add cleanup: 000000000196CEC0  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 http file cache fd: 24  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 read: 24, 000000000196D198,  
285,  
0  
2012/09/13 15:15:16 [crit] 27367#0: \*3441 cache file  
“/mnt/tmp/nginx/cache/90/6d/2b/422c25423170bac1c4e655ed9c2b6d90” has md5  
collision, client: 127.0.0.1, server: [atlas-x.example.com](http://atlas-x.example.com), request: “GET  
/uploads/channel/thumbnail/5927/e74e97624ef652bc7a8419c02f8a7f92ae71d127.jpg  
HTTP/1.1”, host: “127.0.0.1”  
2012/09/13 15:15:16 [debug] 27367#0: \*3441 http upstream cache: -5

Posted at Nginx Forum:

> **[Re: md5 collision](https://forum.nginx.org/read.php?2,230686,230703#msg-230703)**
>
> Hi, Looking for help with a high number of md5 collisions with proxy caching. It's my understanding that we shouldn't be getting any, but we seem to be getting quite a few. -V nginx version: nginx/1.2.3 built by gcc 4.6.3 (Ubuntu/Linaro...