I’m writing some code that loads data into a database on a
routine basis (dump from legacy system daily). I’ve got a
decent amount of data, and am loading it now using ActiveRecord.
I’m finding that it’s taking quite a while longer to process
than I had hoped. To see if it’s my ruby, or ActiveRecord
that is the cause I’d like to write an implementation that
loads the data using the underlying ActiveRecord connection
rather than ActiveRecord objects.
I come from a Perl DBI and JDBC world, and would like to use
prepared statements. So I’ve gotten the underlying Mysql
connection (using ActiveRecord::Base.connection.raw_connection).
When I have code like
stmt = conn.prepare(%Q/
INSERT INTO sometable (t1,t2,t3,t4)
VALUES (?,?,?,?)
/)
bind_params = [1,2,3,4]
stmt.execute(bind_params)
I get an error on the execute statement. It claims I need
to send it 4 parameters. I’d like to send it an array
containing the paramaters because I build the prepared
statement based on the format of my data file. The
bind_params is then an array which is the result of a split
on a line in a data file.
So what’s the best way to handle this? Should I begin to
use Ruby’s DBI rather than just steeling the underlying
ActiveRecord connections?
ActiveRecord connection rather than ActiveRecord objects.
/)
bind_params = [1,2,3,4]
stmt.execute(bind_params)
You probably just need to change the line above to
stmt.execute(*bind_params)
I get an error on the execute statement. It claims I need to send it 4
parameters. I’d like to send it an array containing the paramaters
because I build the prepared statement based on the format of my data
file. The bind_params is then an array which is the result of a split
on a line in a data file.
So what’s the best way to handle this? Should I begin to use Ruby’s DBI
rather than just steeling the underlying ActiveRecord connections?
I come from a Perl DBI and JDBC world, and would like to use
prepared statements. So I’ve gotten the underlying Mysql
connection (using ActiveRecord::Base.connection.raw_connection).
My guess is that prepared statements are indeed the source of your
performance problems. IIRC, ActiveRecord does not cache prepared
statements by default (and if there’s an option for it, I do not know
of it) so you’re essentially calling prepare once for each INSERT! For
most Rails applications this is a space for improvement but not a show
stopper. For importing loads of data, it’s simply unacceptable.
I get an error on the execute statement. It claims I need
to send it 4 parameters.
Unlike in perl (which I’m assuming you’re used to from the symptoms
here), arrays and lists are not quite the same thing in Ruby. When
you call stmt.execute(bind_params), you are not passing a list of four
parameters to execute (as you might expect), but just one parameter
that is an array. Fortunately, Ruby does provide a mechanism for
“splatting” an array into a list of parameters: