# Which is faster: searching in a file or in a database **URL:** \ **Category:** Rails\ **Created:** [December 26, 2007, 5:04pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215 "2007-12-26T17:04:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 26, 2007, 5:04pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/1 "2007-12-26T17:04:25Z")
Hello, I have ten CVS file with about 20000 rows in each. If I want to search in these files for example get every row that has the word “apple” i a column, which is faster: If I read all these ten files into the database and then making search in the database or If I search directly in the files for this inforamtion? Thank you! Please motivate your answer and include a code to make the search. Best regards, Mark ---
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 26, 2007, 5:35pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/2 "2007-12-26T17:35:21Z")
On 26 Dec 2007, at 16:04, Mark T. wrote: > or > > If I search directly in the files for this inforamtion? > > Thank you! Please motivate your answer and include a code to make the > search. Assuming you aren’t processing a fresh set of csv files, I’d be very surprised if parsing the csv files in ruby each time wasn’t slower than sticking it in the database. It’s what databases are supposed to do. That’s just my hunch though, and I’m afraid I’ve got enough of my own work to do before doing yours 🙂 Fred ---
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 26, 2007, 5:53pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/3 "2007-12-26T17:53:11Z")
Frederick C. wrote: > On 26 Dec 2007, at 16:04, Mark T. wrote: > > > or > > > > If I search directly in the files for this inforamtion? > > > > Thank you! Please motivate your answer and include a code to make the > > search. > > Assuming you aren’t processing a fresh set of csv files, I’d be very > surprised if parsing the csv files in ruby each time wasn’t slower > than sticking it in the database. It’s what databases are supposed to > do. That’s just my hunch though, and I’m afraid I’ve got enough of my > own work to do before doing yours 🙂 > > Fred Thank you for your answer. Just another quick question. I am using fasterCSV to read the file and a column looks like: This is the first line This is the “second” line This is the third line I get an error on the second line from fasterCSV since it includes " char. Can you help me how I can avvoid this problem? Thanks, Mark ---
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 26, 2007, 9:49pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/4 "2007-12-26T21:49:37Z")
You have to do the validation and have a valid CSV file to parse it correctly. Correct the file and try again. On Dec 26, 2007 11:20 AM, Mark T. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Exactly I get this error: > > “Unclosed quoted field on line 21” – [http://www.rubyplus.org/](http://www.rubyplus.org/) Free Ruby and Rails Screencasts ---
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 27, 2007, 2:30pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/5 "2007-12-27T14:30:17Z")
Bala P. wrote: > You have to do the validation and have a valid CSV file to parse it > correctly. Correct the file and try again. > On Dec 26, 2007 11:20 AM, Mark T. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > wrote: > > > Exactly I get this error: > > > > “Unclosed quoted field on line 21” > > – > [http://www.rubyplus.org/](http://www.rubyplus.org/) > Free Ruby and Rails Screencasts Well, I have a correct CSV file, but with some small changes, for example I didn´t use " for the strings. I have this column: This is the first line This is the “second” line This is the third line How can I read this information to my database? Is there any command to replace the " char in the whole file with nothing? ---
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 26, 2007, 8:20pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/6 "2007-12-26T20:20:45Z")
Exactly I get this error: “Unclosed quoted field on line 21” ---
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 27, 2007, 5:01pm UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/7 "2007-12-27T17:01:26Z")
On 12/27/07, Mark T. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > How can I read this information to my database? Is there any command to > replace the " char in the whole file with nothing? In bash, using a temp file, it’d be something like: cp foo.csv foo.csv.tmp sed -e ‘s/"//g’ foo.csv.tmp \>foo.csv rm foo.csv.tmp – Greg D. [http://destiney.com/](http://destiney.com/) ---
**Author:** ![markthomas](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/90db22/32.png) [@markthomas](https://www.ruby-forum.com/u/markthomas)\ **Post date:** [December 28, 2007, 4:05am UTC](https://www.ruby-forum.com/t/which-is-faster-searching-in-a-file-or-in-a-database/124215/8 "2007-12-28T04:05:23Z")
On Dec 27, 5:30 am, Mark T. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > How can I read this information to my database? Is there any command to > replace the " char in the whole file with nothing? There’s always a text editor. 🙂 Seriously, using the IO class to 1) read each line of the input file, 2) use gsub to remove the quotes, and 3) write each line of the file, sounds to me like a useful exercise. ///ark