# Finding the first file in a dir, fast, for message queue **URL:** \ **Category:** Ruby\ **Created:** [January 8, 2008, 1:24pm UTC](https://www.ruby-forum.com/t/finding-the-first-file-in-a-dir-fast-for-message-queue/125347 "2008-01-08T13:24:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![alexmic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/977dab/32.png) [@alexmic](https://www.ruby-forum.com/u/alexmic)\ **Post date:** [January 8, 2008, 1:24pm UTC](https://www.ruby-forum.com/t/finding-the-first-file-in-a-dir-fast-for-message-queue/125347/1 "2008-01-08T13:24:23Z")
I have a file based msg queue, and msgs are stored five folders down. When a request for the next msg is received, I need to grab the first file I can find, as fast as I can. I’ve tried Dir.glob, and only selecting the first file. This however is an awful way of doing it as it loads every file into memory, before selecting the first. A bit better is Find.find, which finds files incrementally. However, this still takes about 0.005 seconds (I presume since it’s also ‘finding’ directories’). Is there a faster way to do this? ---
**Author:** ![alexmic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/977dab/32.png) [@alexmic](https://www.ruby-forum.com/u/alexmic)\ **Post date:** [January 8, 2008, 2:04pm UTC](https://www.ruby-forum.com/t/finding-the-first-file-in-a-dir-fast-for-message-queue/125347/2 "2008-01-08T14:04:58Z")
2008/1/8, Alex M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > this still takes about 0.005 seconds (I presume since it’s also > ‘finding’ directories’). > > Is there a faster way to do this? You find 5ms when accessing the file system long? I’d say that’s pretty fast considering what you do (recursive search). I doubt you will get much improvement as long as you always access the file system for your search. If you know the change frequency of files then you could store file system contents in memory and only update every n seconds / minutes or whatever or have a background thread that continuously updates your in memory representation. Kind regards robert ---
**Author:** ![alexmic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/977dab/32.png) [@alexmic](https://www.ruby-forum.com/u/alexmic)\ **Post date:** [January 8, 2008, 2:18pm UTC](https://www.ruby-forum.com/t/finding-the-first-file-in-a-dir-fast-for-message-queue/125347/3 "2008-01-08T14:18:32Z")
> You find 5ms when accessing the file system long? I’d say that’s > pretty fast considering what you do (recursive search). I doubt you > will get much improvement as long as you always access the file system > for your search. If you know the change frequency of files then you > could store file system contents in memory and only update every n > seconds / minutes or whatever or have a background thread that > continuously updates your in memory representation. Well, it’s long compared to generating folder names, a guid, and writing files. It means that I can publish about 1000 msgs per second on to my queue, but only pull of 243 msgs per second. ---
**Author:** ![alexmic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/977dab/32.png) [@alexmic](https://www.ruby-forum.com/u/alexmic)\ **Post date:** [January 8, 2008, 2:24pm UTC](https://www.ruby-forum.com/t/finding-the-first-file-in-a-dir-fast-for-message-queue/125347/4 "2008-01-08T14:24:20Z")
2008/1/8, Alex M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > files. It means that I can publish about 1000 msgs per second on to my > queue, but only pull of 243 msgs per second. Here’s another option: find all files and put them in a queue. Only redo search when the queue is empty. This might pay off over all. robert ---
**Author:** ![alexmic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/977dab/32.png) [@alexmic](https://www.ruby-forum.com/u/alexmic)\ **Post date:** [January 8, 2008, 3:25pm UTC](https://www.ruby-forum.com/t/finding-the-first-file-in-a-dir-fast-for-message-queue/125347/5 "2008-01-08T15:25:03Z")
Robert K. wrote: > 2008/1/8, Alex M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > > > files. It means that I can publish about 1000 msgs per second on to my > > queue, but only pull of 243 msgs per second. > > Here’s another option: find all files and put them in a queue. Only > redo search when the queue is empty. This might pay off over all. > > robert Yes, that’s what I’ve done. Message polling is now the same speed as publishing (so the overall speed is about 10000 msg per second). If you interested, here’s the queue: [http://code.google.com/p/sparrow](http://code.google.com/p/sparrow)