# Importing many 'dirty' .csv records..what approach is best?

**URL:** <https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991>\
**Category:** Rails\
**Created:** [April 27, 2008, 6:55pm UTC](https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991 "2008-04-27T18:55:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [April 27, 2008, 6:55pm UTC](https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991/1 "2008-04-27T18:55:56Z")

</div>

I am tasked with importing many .csv files into an application that  
has a ‘scoped’ database that services many distinct customers. I’m  
sure it comes as no surprise that many of these will be damaged in  
some way with spurious data 🙂  
Initially, I’m using FasterCSV in a dbmigrate file to become more  
familiar with this task.

Here is an example of how I’m approaching this;

class ImportAssets \< ActiveRecord::Migration  
def self.up  
n=0  
d=0  
FasterCSV.foreach(“D:/Rails/assets/convert/assets.csv”) do |row|  
c = Asset.new  
c.asset = row[0] # 0  
c.description = row[1] # 1  
c.location = row[2] # 2  
if c.save  
n += 1  
else  
d = Badboy.new  
d.converttable = ‘Asset’  
d.keystring = row[0]  
d.message = ‘Asset Record Failed to convert’  
d += 1  
d.recno = d  
d.save  
end  
end  
end  
end

I find that regardless of the errors in the incoming .csv file, I’m  
unable to get my logic to branch and save the errors to a ‘badboys’  
exception table.  
Firstly, does anyone in our group have to deal with this type of  
situation…do you use a controller approach of some sort or do you  
load the details for each new customers’ data by adding one more  
dbmigrate file?  
Why am I not able to create a ‘badboy’ record when my incoming record  
doesn’t save?  
I am grateful for any links or suggestions.  
Thank you,  
Kathleen

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [April 27, 2008, 7:01pm UTC](https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991/2 "2008-04-27T19:01:19Z")

</div>

On 27 Apr 2008, at 17:54, removed\_email\_address@domain.invalid wrote:

> class ImportAssets \< ActiveRecord::Migration  
> else  
> d = Badboy.new  
> d.converttable = ‘Asset’  
> d.keystring = row[0]  
> d.message = ‘Asset Record Failed to convert’  
> d += 1  
> There’s something fishy here probably partly as a result of using  
> nondescript variable names like n,c,d  
> You’re using d both for some sort of counter and also for your new  
> instance of Badboy

Fred

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [April 27, 2008, 10:51pm UTC](https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991/3 "2008-04-27T22:51:56Z")

</div>

Fred,  
Thanks for the observation. I made that change you mentioned but it  
has no bearing on the issue as the program logic never got down to the  
Badboy area.  
Still hoping for some suggestion?  
Thanks,  
Kathleen

On Apr 27, 11:00 am, Frederick C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [April 28, 2008, 1:13am UTC](https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991/4 "2008-04-28T01:13:39Z")

</div>

On Sun, 2008-04-27 at 09:54 -0700, removed\_email\_address@domain.invalid wrote:

> def self.up  
> d = Badboy.new
> 
> I find that regardless of the errors in the incoming .csv file, I’m  
> unable to get my logic to branch and save the errors to a ‘badboys’  
> exception table.  
> Firstly, does anyone in our group have to deal with this type of  
> situation…do you use a controller approach of some sort or do you  
> load the details for each new customers’ data by adding one more  
> dbmigrate file?  
> Why am I not able to create a ‘badboy’ record when my incoming record  
> doesn’t save?  
> I am grateful for any links or suggestions.

* * *

shouldn’t you be using begin/rescue to capture errors?

Craig

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [April 27, 2008, 11:40pm UTC](https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991/5 "2008-04-27T23:40:13Z")

</div>

The code looks ok, (with the exception mentioned by Fred)  
Are there any validations on Badboy that may prevent saving? (only thing  
I could imagine that could go wrong here). Main suspect is “keystring”,  
which gets data, that just made Asset fail.  
I guess, that there must be some validations on Assets, since you expect  
them to fail on save.

Otherwise I would debug the thing (or use the log) and see, where it  
ends up after the Asset save. Does it run the Badboy part at all and  
fails saving or does it never reach it?

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [April 28, 2008, 11:16am UTC](https://www.ruby-forum.com/t/importing-many-dirty-csv-records-what-approach-is-best/135991/6 "2008-04-28T11:16:56Z")

</div>

On 27 Apr 2008, at 21:50, removed\_email\_address@domain.invalid wrote:

> Fred,  
> Thanks for the observation. I made that change you mentioned but it  
> has no bearing on the issue as the program logic never got down to the  
> Badboy area.  
> Still hoping for some suggestion?  
> Thanks,  
> Kathleen  
> Does c.save ever return false (ie do you have validations on it?)?  
> Have you tried stepping through this in the debugger to see what’s up?

Fred