# Fast searching of large files

**URL:** <https://www.ruby-forum.com/t/fast-searching-of-large-files/190944>\
**Category:** Ruby\
**Created:** [July 1, 2010, 11:46am UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944 "2010-07-01T11:46:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sclarke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b19c9b/32.png) [@sclarke](https://www.ruby-forum.com/u/sclarke)\
**Post date:** [July 1, 2010, 11:46am UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944/1 "2010-07-01T11:46:49Z")

</div>

Hey all,

Could anyone advise me on a fast way to search a single, but very large  
file (1Gb) quickly for a string of text? Also, is there a library to  
identify the file offset this string was found within the file?

Thanks

---

<div class="post-metadata">

**Author:** ![sclarke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b19c9b/32.png) [@sclarke](https://www.ruby-forum.com/u/sclarke)\
**Post date:** [July 1, 2010, 12:41pm UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944/2 "2010-07-01T12:41:02Z")

</div>

On Thu, Jul 1, 2010 at 6:47 PM, Stuart C.  
[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Hey all,
> 
> Could anyone advise me on a fast way to search a single, but very large  
> file (1Gb) quickly for a string of text? Also, is there a library to  
> identify the file offset this string was found within the file?

You can use IO#grep like this:  
File.open(‘qimo-2.0-desktop.iso’, ‘r:BINARY’){|io|  
io.grep(/apiKey/){|m| p io.pos =\> m } }

The pos is the position the match ended, so just substract the string  
length.  
The above example was a file with 700mb, took around 40s the first  
time, 2s subsequently, so disk I/O is the limiting factor in terms of  
speed (as usual).  
Oh, and also don’t use binary encoding if you are dealing with another  
one 😉

---

<div class="post-metadata">

**Author:** ![sclarke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b19c9b/32.png) [@sclarke](https://www.ruby-forum.com/u/sclarke)\
**Post date:** [July 1, 2010, 1:03pm UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944/3 "2010-07-01T13:03:52Z")

</div>

2010/7/1 Michael F. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> io.grep(/apiKey/){|m| p io.pos =\> m } }
> 
> The pos is the position the match ended, so just substract the string length.  
> The above example was a file with 700mb, took around 40s the first  
> time, 2s subsequently, so disk I/O is the limiting factor in terms of  
> speed (as usual).

If you only need to know whether the string occurs in the file you can  
do

found = File.foreach(“foo”).any? {|line| /apiKey/ =~ line}

This will stop searching as soon as the sequence is found.

“fgrep -l foo” is likely faster.

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![sclarke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b19c9b/32.png) [@sclarke](https://www.ruby-forum.com/u/sclarke)\
**Post date:** [July 1, 2010, 1:58pm UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944/4 "2010-07-01T13:58:47Z")

</div>

Thanks.

This seems to be pretty much the best logic for me, however it takes a  
good 20 minutes to scan a 2Gb file.

Any ideas?

Thanks

Michael F. wrote:

> On Thu, Jul 1, 2010 at 6:47 PM, Stuart C.  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > Hey all,
> > 
> > Could anyone advise me on a fast way to search a single, but very large  
> > file (1Gb) quickly for a string of text? Also, is there a library to  
> > identify the file offset this string was found within the file?
> 
> You can use IO#grep like this:  
> File.open(‘qimo-2.0-desktop.iso’, ‘r:BINARY’){|io|  
> io.grep(/apiKey/){|m| p io.pos =\> m } }
> 
> The pos is the position the match ended, so just substract the string  
> length.  
> The above example was a file with 700mb, took around 40s the first  
> time, 2s subsequently, so disk I/O is the limiting factor in terms of  
> speed (as usual).  
> Oh, and also don’t use binary encoding if you are dealing with another  
> one 😉

---

<div class="post-metadata">

**Author:** ![sclarke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b19c9b/32.png) [@sclarke](https://www.ruby-forum.com/u/sclarke)\
**Post date:** [July 1, 2010, 9:29pm UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944/5 "2010-07-01T21:29:43Z")

</div>

On Thu, Jul 1, 2010 at 7:03 AM, Robert K.  
[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > The pos is the position the match ended, so just substract the string length.  
> > The above example was a file with 700mb, took around 40s the first  
> > time, 2s subsequently, so disk I/O is the limiting factor in terms of  
> > speed (as usual).
> 
> If you only need to know whether the string occurs in the file you can do  
> found = File.foreach(“foo”).any? {|line| /apiKey/ =~ line}  
> This will stop searching as soon as the sequence is found.
> 
> “fgrep -l foo” is likely faster.

irb\> `fgrep -l waters /usr/share/dict/words`.size \> 0  
=\> true  
irb\> `fgrep -l watershed /usr/share/dict/words`.size \> 0  
=\> true  
irb\> `fgrep -l watershedz /usr/share/dict/words`.size \> 0  
=\> false

irb\> `fgrep -ob waters /usr/share/dict/words`.split.map{|s|  
s.split(‘:’).first}  
=\> [“153088”, “153102”, “204143”, “234643”, “472357”, “856441”,  
“913606”, “913613”, “913623”, “913635”, “913646”, “913656”, “913668”,  
“913679”, “913690”, “913703”]  
irb\> `fgrep -ob watershed /usr/share/dict/words`.split.map{|s|  
s.split(‘:’).first}  
=\> [“913613”, “913623”, “913635”]  
irb\> `fgrep -ob watershedz /usr/share/dict/words`.split.map{|s|  
s.split(‘:’).first}  
=\> []

---

<div class="post-metadata">

**Author:** ![sclarke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b19c9b/32.png) [@sclarke](https://www.ruby-forum.com/u/sclarke)\
**Post date:** [July 1, 2010, 9:33pm UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944/6 "2010-07-01T21:33:00Z")

</div>

Stuart C. wrote:

> Hey all,
> 
> Could anyone advise me on a fast way to search a single, but very large  
> file (1Gb) quickly for a string of text? Also, is there a library to  
> identify the file offset this string was found within the file?

a fast way is to do it in C 🙂

Here are a few other helpers, though:

1.9 has faster regexes  
boost regexes: [GitHub - michaeledgar/ruby-boost-regex: Wraps Boost::Regex in a Ruby binding](http://github.com/michaeledgar/ruby-boost-regex) (you  
could probably optimize it more than it currently is, as well…)

Rubinius also might help.

Also make sure to open your file in binary mode if you’re on 1.9. That  
reads much faster. If that’s an option, anyway.  
GL.  
-rp

---

<div class="post-metadata">

**Author:** ![sclarke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b19c9b/32.png) [@sclarke](https://www.ruby-forum.com/u/sclarke)\
**Post date:** [July 1, 2010, 7:03pm UTC](https://www.ruby-forum.com/t/fast-searching-of-large-files/190944/7 "2010-07-01T19:03:56Z")

</div>

Michael F. wrote:

> io.grep(/apiKey/){|m| p io.pos =\> m } }
> 
> The pos is the position the match ended

Actually, pos will be the position of the end of the line on which the  
match was found, because #grep works line by line.