Finding duplicate records before creating using FasterCSV

Ruby
1

Hello all,
I have a user that will be importing csv files occasionally. I have,
with a great amount of help (thanks James), set up an app. that will
read a csv and save to database.
I need to check for duplicate records before writing to database.

I’m thinking about putting a conditional to check for the duplicates via
column name.

  • In controller -

Irb.transaction do
FasterCSV.parse(file,
:headers => true,
:header_converters => lambda { |h| h.tr(" ",
“").delete("^a-zA-Z0-9”)},
:converters => :all ) do |row|
# I’m thinking conditional looping thru the Model
here. #
Irb.create!( {“reconciled” => 0,
“non_related” => 0
}.merge(row.to_hash))
rowcount += 1
end
end

Thank you for any help.

JohnM

2

On Feb 4, 2010, at 1:15 PM, John M. wrote:

I’m thinking about putting a conditional to check for the duplicates via
column name.

Why not just add a validates_uniqueness_of() validation the the Irb
model, so it will block the save of duplicate records?

James Edward G. II

3

Why not just add a validates_uniqueness_of() validation the the Irb
model, so it will block the save of duplicate records?

I never thought of that.

I’ll give it a try.

thanks again.

john

4

The validation works as it should but it stops the process of importing
the file. I would have to clean the csv of duplicate records before the
import.

john

5

James Edward G. II wrote:

On Feb 4, 2010, at 1:15 PM, John M. wrote:

I’m thinking about putting a conditional to check for the duplicates via
column name.

Why not just add a validates_uniqueness_of() validation the the Irb
model, so it will block the save of duplicate records?

And follow that up with a unique index in the DB. Validations are
insufficient by themselves.

James Edward G. II

Best,
–Â
Marnen Laibow-Koser
http://www.marnen.org
[email protected]

6

And follow that up with a unique index in the DB. Validations are
insufficient by themselves.

The index is very unique.
For example,

YY-XXXXabc

“YY” = year without century,
“XXXX” is an incremental integer and “abc” could be a 1,2 or 3 letter
combination.

John

7

Great, that did it.

Thanks again, James.

John

8

On Feb 4, 2010, at 1:36 PM, John M. wrote:

The validation works as it should but it stops the process of importing
the file. I would have to clean the csv of duplicate records before the
import.

Switch create!() to create(). All fixed. :wink:

James Edward G. II

9

Not to move away from this discussion but what’s the difference between
“create” and “create!” ?
I looked a little and couldn’t find much.

John

10

John M. wrote:

And follow that up with a unique index in the DB. Validations are
insufficient by themselves.

The index is very unique.

Something either is or isn’t unique. There’s no “very”. :slight_smile:

For example,

YY-XXXXabc

“YY” = year without century,
“XXXX” is an incremental integer and “abc” could be a 1,2 or 3 letter
combination.

I think you missed my point, or I missed a joke. “Unique index” is a
technical DB term – look it up if you’re not familiar with the concept.

John

Best,
–Â
Marnen Laibow-Koser
http://www.marnen.org
[email protected]

11

John M. wrote:

Not to move away from this discussion but what’s the difference between
“create” and “create!” ?
I looked a little and couldn’t find much.

This is explained clearly in the AR docs.

John

Best,
–Â
Marnen Laibow-Koser
http://www.marnen.org
[email protected]

12

On Feb 4, 2010, at 2:08 PM, John M. wrote:

Not to move away from this discussion but what’s the difference between
“create” and “create!” ?

create() fails silently, returning the unsaved database object with
errors if it could not be written to the database for some reason.
create!() raises an error under the same circumstances.

James Edward G. II