# Sync list of files in DB vs. list of files in folder **URL:** \ **Category:** Rails\ **Created:** [August 19, 2009, 9:52pm UTC](https://www.ruby-forum.com/t/sync-list-of-files-in-db-vs-list-of-files-in-folder/174037 "2009-08-19T21:52:43Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![trevoke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ee7513/32.png) [@trevoke](https://www.ruby-forum.com/u/trevoke)\ **Post date:** [August 19, 2009, 9:52pm UTC](https://www.ruby-forum.com/t/sync-list-of-files-in-db-vs-list-of-files-in-folder/174037/1 "2009-08-19T21:52:43Z")
This may be the wrong question to ask, but I’ll start here anyway because this could probably use some optimization. The code underneath gets a list of all the files in all the folders specified, then deletes the database entries that aren’t in that list, updates the entries in both the db and the list, and creates the entries that aren’t in the db. Is there a better way to do this? all\_reports is of the form: { “a\_path” =\> [“report1”, “report2”, “report3”], “another\_path” =\> […] … } list = [] all\_reports.each\_value { |reps| list.concat reps } Report.all.each do |rep| rep.destroy if !list.include? rep.filename end all\_reports.each do |path, reports| Dir.chdir path reports.each do |rep| report = Report.find\_by\_filename rep if report.nil? # If it doesn’t exist, create it generate(rep, path) else # If it does exist, update the timestamps. report.absolute\_age = Time.now - File.ctime(rep) report.age = get\_age report.absolute\_age # Magic. aging = modalities.find\_by\_name(report.modality).aging rescue DefaultAging report.old = (report.absolute\_age.to\_hours \> aging) # Yes, I did create a to\_hours report.save end end end 0 ---
**Author:** ![trevoke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ee7513/32.png) [@trevoke](https://www.ruby-forum.com/u/trevoke)\ **Post date:** [August 19, 2009, 11:43pm UTC](https://www.ruby-forum.com/t/sync-list-of-files-in-db-vs-list-of-files-in-folder/174037/2 "2009-08-19T23:43:46Z")
On Aug 19, 3:52 pm, Aldric G. \ wrote: > This may be the wrong question to ask, but I’ll start here anyway > because this could probably use some optimization. > The code underneath gets a list of all the files in all the folders > specified, then deletes the database entries that aren’t in that list, > updates the entries in both the db and the list, and creates the entries > that aren’t in the db. Question: why do you need to duplicate the filesystem’s directory information in the DB? > Is there a better way to do this? > […] Almost certainly. Notice that you’re using find\_by\_filename each time through the loop. That’s a pretty good indication that something’s wrong; in general, database queries do not belong inside loops. I’ll try to come up with something better, but first I’d like to know what you’re trying to achieve here. ## Best, Marnen Laibow-Koser [http://www.marnen.org](http://www.marnen.org) removed\_email\_address@domain.invalid ---
**Author:** ![trevoke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ee7513/32.png) [@trevoke](https://www.ruby-forum.com/u/trevoke)\ **Post date:** [August 20, 2009, 2:29pm UTC](https://www.ruby-forum.com/t/sync-list-of-files-in-db-vs-list-of-files-in-folder/174037/3 "2009-08-20T14:29:28Z")
Marnen Laibow-Koser wrote: > Question: why do you need to duplicate the filesystem’s directory > information in the DB? The filesystem’s directory information has medical reports in Word document format. The naming convention is: “last name, first name{, middle initial, title}, date of study, procedure” I use ‘antiword’ to parse the document and get the information I want (win32OLE is way, way, WAY too slow). The proofreaders need to know the date of study and how long the exam’s been in the directory - which can’t easily be determined from just looking at the directory (it’s awkward). My first version of the page literally just built the table with the information to be displayed on-the-fly, every time it was refreshed. I figured this may be an issue if too many people attempted to do it at once, so I stuck it in the database instead - now the background job does the checking of the directory and the parsing of the files. Additionally, I’m in fact checking a couple of directories and offering them as options in a menu in my view, again, to make their life easier * * * Of course, and this is the issue which brings me here in the first place, when I click on one of those options after a couple of sweeps happened, the rails log overflows with that kind of lines: e[4;35;1mReport Load (0.0ms)e[0m e[0mSELECT \* FROM “reports” WHERE (“reports”.“filename” = ‘SomeReportName.doc’) LIMIT 1e[0m e[4;36;1mReport Update (0.0ms)e[0m e[0;1mUPDATE “reports” SET “updated\_at” = ‘2009-08-19 13:51:22’, “absolute\_age” = 31393 WHERE “id” = 927e[0m Which is what bothered me and what bothers you 🙂 ---
**Author:** ![trevoke](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ee7513/32.png) [@trevoke](https://www.ruby-forum.com/u/trevoke)\ **Post date:** [August 21, 2009, 5:37pm UTC](https://www.ruby-forum.com/t/sync-list-of-files-in-db-vs-list-of-files-in-folder/174037/4 "2009-08-21T17:37:08Z")
Okay, I finally got a clear look at the log, and it is indeed how it seems - all the transactions to the database are queued until I actually look at the database. The DB I am using is SQLite3. How do I force ActiveRecord / SQLite / Rails to commit / process the transaction so it happens in a better type of real-time than that?