# Fastest way to parse millions of file

**URL:** <https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054>\
**Category:** Ruby\
**Created:** [October 22, 2013, 11:18am UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054 "2013-10-22T11:18:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![addis\_a](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c4cdca/32.png) [@addis\_a](https://www.ruby-forum.com/u/addis_a)\
**Post date:** [October 22, 2013, 11:18am UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054/1 "2013-10-22T11:18:36Z")

</div>

how i can read file faster. I have millions of file my motto is to read  
all files and extract particular patter and then create a hash with  
those extracted value. What i did is open the file and then reading each  
line.it is taking to much time.

---

<div class="post-metadata">

**Author:** ![asmita](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a87d85/32.png) [@asmita](https://www.ruby-forum.com/u/asmita)\
**Post date:** [October 22, 2013, 12:25pm UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054/2 "2013-10-22T12:25:38Z")

</div>

Is this a one off request or a general purpose search over your million  
files?

If it is a general purpose request then I would say feed them all into a  
search engine (ferret, lucene, sphyinx etc) and then search from that.

Other people will be able to tell you the best gem / package to use.

Also it might just be that some other tool (grep for example) might be  
better suited for the find the data. What sort of pattern are you  
looking  
for?

---

<div class="post-metadata">

**Author:** ![asmita](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a87d85/32.png) [@asmita](https://www.ruby-forum.com/u/asmita)\
**Post date:** [October 22, 2013, 12:41pm UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054/3 "2013-10-22T12:41:32Z")

</div>

I am searching for all variable names which are set by reading files.

Eg.  
ifdef(`SUM’)

I am not getting what exactly you mean by one off request and general  
purpose searching. But i can’t feed them in search engine due to privacy  
issue.

---

<div class="post-metadata">

**Author:** ![asmita](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a87d85/32.png) [@asmita](https://www.ruby-forum.com/u/asmita)\
**Post date:** [October 22, 2013, 1:00pm UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054/4 "2013-10-22T13:00:09Z")

</div>

On Tue, Oct 22, 2013 at 12:41 PM, Asmita C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> I am searching for all variable names which are set by reading files.

If you just need the names then “egrep -o” is probably a more  
efficient choice. You won’t get a Ruby Hash from that though.

> Eg.  
> ifdef(`SUM’)
> 
> I am not getting what exactly you mean by one off request and general  
> purpose searching.

The opposite of “one off” is a repeated task.

> But i can’t feed them in search engine due to privacy  
> issue.

But you can use a text indexing engine like Lucene or others.  
Although the effort pays off only if you need to do this repeatedly.

Cheers

robert

---

<div class="post-metadata">

**Author:** ![asmita](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a87d85/32.png) [@asmita](https://www.ruby-forum.com/u/asmita)\
**Post date:** [October 22, 2013, 1:03pm UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054/5 "2013-10-22T13:03:05Z")

</div>

On Tue, Oct 22, 2013 at 12:41 PM, Asmita C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> I am searching for all variable names which are set by reading files.
> 
> Eg.  
> ifdef(`SUM’)
> 
> I am not getting what exactly you mean by one off request and general  
> purpose searching. But i can’t feed them in search engine due to privacy  
> issue.

What Peter meant is that if this is something that you want to  
repeatedly use, such as today searching for this pattern, tomorrow for  
a different pattern and so on, using this functionality many times,  
then you might be better off using a search engine or database. Also  
you can use a search engine locally without any privacy issues.

If it’s just a one off operation, you probably won’t need that  
complexity.

Jesus.

---

<div class="post-metadata">

**Author:** ![asmita](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a87d85/32.png) [@asmita](https://www.ruby-forum.com/u/asmita)\
**Post date:** [October 23, 2013, 6:52am UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054/6 "2013-10-23T06:52:43Z")

</div>

Thanks all.  
I got it. It is one off operation. and I did it by grep.  
Thanks again. 🙂

---

<div class="post-metadata">

**Author:** ![asmita](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a87d85/32.png) [@asmita](https://www.ruby-forum.com/u/asmita)\
**Post date:** [October 22, 2013, 1:02pm UTC](https://www.ruby-forum.com/t/fastest-way-to-parse-millions-of-file/237054/7 "2013-10-22T13:02:30Z")

</div>

It is possible to set up your own search engine on a desktop pc. It will  
index all your data and store it locally on your pc. If the machine with  
a  
million files on it is considered safe then building your own personal  
search engine on the same machine should not be a privacy issue.

Having said that grep could still be the tool for you

$ grep 'ifdef._SUM’ files/_

Will search all the files that have the string ‘ifdef’ followed by ‘SUM’  
on  
the same line. Grep is quite fast.
