Hi, I’m trying to run a set of migration which brings data from a
legacy project. The legacy project is on production and have lots of
data. So, during the time the migration keep allocating and allocating
memory until it grabs it all. I’m doing pagination to bring the data
step by step. Beside, what is very wear is that the memory never goes
down between migration, for example, I do:
rake db:migrate
== 2 InitialMoveLegacyUsersToUsers: migrating
== 2 InitialMoveLegacyUsersToUsers: migrated (83.3700s)
== 3 AddPasswordResetCodeToUsers: migrating
– add_column(:users, :password_reset_code, :string)
-> 0.0877s
== 3 AddPasswordResetCodeToUsers: migrated (0.0883s)
== 4 AddOpenIdAuthenticationTables: migrating
– create_table(:open_id_authentication_associations, {:force=>true})
-> 0.0054s
– create_table(:open_id_authentication_nonces, {:force=>true})
-> 0.0044s
== 4 AddOpenIdAuthenticationTables: migrated (0.0105s)
And watching the behavior of the memory between migrations it never
free memory, always keep growing.
So, it seems to me like a memory leak, but dunno if I’m doing
something wrong.
Is there a way to call the garbage collector to free the memory
between migrations??
I’ll appreciate your help.
Rafael Chacon wrote:
rake db:migrate
== 2 InitialMoveLegacyUsersToUsers: migrating
== 2 InitialMoveLegacyUsersToUsers: migrated (83.3700s)
== 3 AddPasswordResetCodeToUsers: migrating
– add_column(:users, :password_reset_code, :string)
-> 0.0877s
== 3 AddPasswordResetCodeToUsers: migrated (0.0883s)
== 4 AddOpenIdAuthenticationTables: migrating
– create_table(:open_id_authentication_associations, {:force=>true})
-> 0.0054s
– create_table(:open_id_authentication_nonces, {:force=>true})
-> 0.0044s
== 4 AddOpenIdAuthenticationTables: migrated (0.0105s)
And watching the behavior of the memory between migrations it never
free memory, always keep growing.
You probably need to show me what your migrations look like.
You must be loading a load of database rows into ruby objects?
Maybe you’re setting them to a variable that remains in scope?
Don’t know.
If you do rake db:migrate VERSION=2
and then seperately do rake db:migrate VERSION=4
does it isolate the problem to the first migration?
The most conlficting migration looks like this:
class MoveLegacySugarEntries < ActiveRecord::Migration
class Entries < ActiveRecord::Base;end
class Entry::SugarEntry < Entry;end
def self.up
Legacy::SugarReading.each do |le|
u = User.find_by_login(le.user.login)
unless u== nil
TzTime.zone = u.tz
Entry::SugarEntry.create!(
:registered_at => DateTime.parse("#{le.date}
#{le.time.hour}:#{le.time.min}"),
:comment => le.comment,
:value => le.reading,
:user => u)
end
end
end
Where the active record each is implemented like this:
class <<ActiveRecord::Base
def each(limit=1000)
rows = find(:all, :conditions => [“id > ?”, 0], :limit => limit)
while rows.any?
rows.each { |record| yield record }
rows = find(:all, :conditions => [“id > ?”,
rows.last.id], :limit => limit)
end
self
end
end
So, is getting 1000 rows at a time… Enough to don’t exploit the
memory.
Which variable might not be getting out of scope??
Anyway, how would you think possible to get garbage memory from
previous migrations??
For example, if I ran my migration from 0 when I got to this migration
I already have 1.5 gigas memory occupied, isn’t kind of wear???
Thanks for your help
On Jun 28, 3:37 am, Matthew R. Jacobs <rails-mailing-l…@andreas-