# Avoiding ext3's 4K entry limit when caching

**URL:** <https://www.ruby-forum.com/t/avoiding-ext3s-4k-entry-limit-when-caching/48022>\
**Category:** Rails\
**Created:** [November 16, 2005, 2:18am UTC](https://www.ruby-forum.com/t/avoiding-ext3s-4k-entry-limit-when-caching/48022 "2005-11-16T02:18:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cool\_screen\_name900](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c68b51/32.png) [@cool\_screen\_name900](https://www.ruby-forum.com/u/cool_screen_name900)\
**Post date:** [November 16, 2005, 2:18am UTC](https://www.ruby-forum.com/t/avoiding-ext3s-4k-entry-limit-when-caching/48022/1 "2005-11-16T02:18:05Z")

</div>

I’ve just started checking out caching. I have  
thousands of items with URLs like ‘/item/view/ID’. I  
see caching puts them in public/item/view/ID.html. I’m  
using Linux’s ext3 filesystem and I’ve run into  
problems before with caching and ext3’s 4K entries per  
directory limit. How can I avoid this?

thanks  
csn

* * *

Yahoo! FareChase: Search multiple travel sites in one click.  
[http://farechase.yahoo.com](http://farechase.yahoo.com)

---

<div class="post-metadata">

**Author:** ![cool\_screen\_name900](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c68b51/32.png) [@cool\_screen\_name900](https://www.ruby-forum.com/u/cool_screen_name900)\
**Post date:** [November 16, 2005, 2:27am UTC](https://www.ruby-forum.com/t/avoiding-ext3s-4k-entry-limit-when-caching/48022/2 "2005-11-16T02:27:08Z")

</div>

Wow, this is an excellent question.

ext3’s performance with super large directories can actually be pretty  
decent with dir\_index (it’s depressingly bad without it). You could  
alter ext3 itself to allow more entries, but honestly, with that many  
files in a directory you probably spend more time seeking the directory  
for the file than you would spend generating the page dynamically. You  
could harvest the oldest files in the directory every few minutes, or  
whatever seems appropriate, with a scheduled job.

It’d be very interesting to see how other large sites have handled this.  
I’d push for either the cron, or rethinking what you cache (ie, don’t  
cache whole pages, just cache parts that repeat). With more than a few  
hundred files in a directory you’ll loose alot of performance no matter  
what filesystem you use.

Looking forward to hearing from some more experienced deployment folk,

-Matt B

---

<div class="post-metadata">

**Author:** ![cool\_screen\_name900](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c68b51/32.png) [@cool\_screen\_name900](https://www.ruby-forum.com/u/cool_screen_name900)\
**Post date:** [November 16, 2005, 8:00pm UTC](https://www.ruby-forum.com/t/avoiding-ext3s-4k-entry-limit-when-caching/48022/3 "2005-11-16T20:00:44Z")

</div>

Matthew B. wrote:

> Wow, this is an excellent question.
> 
> ext3’s performance with super large directories can actually be pretty  
> decent with dir\_index (it’s depressingly bad without it). You could  
> alter ext3 itself to allow more entries, but honestly, with that many  
> files in a directory you probably spend more time seeking the directory  
> for the file than you would spend generating the page dynamically. You  
> could harvest the oldest files in the directory every few minutes, or  
> whatever seems appropriate, with a scheduled job.

Matthew,

What are the performance characteristics of ext3 filesystems with and  
without  
dir\_index for small directories up to large ones?

How many files do you need in a directory before dir\_index is worth it?

Right now all my filesystems do not have dir\_index enabled, so it would  
require  
some downtime to enable it.

Regards,  
Blair

---

<div class="post-metadata">

**Author:** ![cool\_screen\_name900](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c68b51/32.png) [@cool\_screen\_name900](https://www.ruby-forum.com/u/cool_screen_name900)\
**Post date:** [November 17, 2005, 6:03am UTC](https://www.ruby-forum.com/t/avoiding-ext3s-4k-entry-limit-when-caching/48022/4 "2005-11-17T06:03:59Z")

</div>

On Wed, 2005-11-16 at 11:00 -0800, Blair Z. wrote:

> Matthew,
> 
> What are the performance characteristics of ext3 filesystems with and without  
> dir\_index for small directories up to large ones?

Not sure on any benchmarks. I wouldn’t say “stunning” is a bad word to  
use. Basically, instead of using lists for files it uses B-Trees, which  
are the same tech that make reiserfs directories so damn fast.

dir\_index -  
Use hashed b-trees to speed up lookups in large directories.

> How many files do you need in a directory before dir\_index is worth it?

I don’t know. But if you have 4000 I’d say that’s a good place to  
start 🙂

> Right now all my filesystems do not have dir\_index enabled, so it would require  
> some downtime to enable it.

Yeah, that’s the crappy part. In theory, you can enable it with  
tune2fs, but in practice I’ve only gotten it with mke2fs. My tools when  
implementing it were mostly from older debian distro though, this was  
unheard of stuff when they were written.

gl! Let us know how you fare.

-Matthew B.