How much "records" can development env stands?!

[adedip]

I didn’t know how to title this question sorry

I wrote this application using a MySQL DB and Solr Server (updated via
acts_as_solr plugin)

I need to load tons of documents to test it out, the app is a
newspaper article collection repository that I upload via SGML corpus.

Now the strange is…that after loading about 30K articles…the app
started to act weird: it does not load anymore or it does after a lot
of minutes…yesterday I’ve kept loading…it reached the 56K
articles…but now is stuck again…

In testing this on a different machine that I use from remote… any
clue?

(the solr server is perfectly reachable…)

thx,
Andrea

[adedip]

Quoting adedip [email protected]:

of minutes…yesterday I’ve kept loading…it reached the 56K
articles…but now is stuck again…

In testing this on a different machine that I use from remote… any
clue?

(the solr server is perfectly reachable…)

I have a development system with over 5 million rows in one table. The
string
field is typically 5-50 characters, the rest is a half dozen integers of
different sizes. The production system has over 6 million rows.

That said, ActiveRecord is convenient and memory greedy. For that afore
mentioned table, I just use straight SQL (inserts and selects can run to
thousands of rows). For the other tables, I continue to use
ActiveRecord.

HTH,
Jeffrey

[adedip]

Jeffrey L. Taylor wrote:

Quoting adedip [email protected]:

of minutes…yesterday I’ve kept loading…it reached the 56K
articles…but now is stuck again…

In testing this on a different machine that I use from remote… any
clue?

(the solr server is perfectly reachable…)

I have a development system with over 5 million rows in one table. The
string
field is typically 5-50 characters, the rest is a half dozen integers of
different sizes. The production system has over 6 million rows.

That said, ActiveRecord is convenient and memory greedy. For that afore
mentioned table, I just use straight SQL (inserts and selects can run to
thousands of rows).

Have you tried ar-extensions? This should let you do bulk queries
properly.

For the other tables, I continue to use
ActiveRecord.

HTH,
Jeffrey

Best,
–Â
Marnen Laibow-Koser
http://www.marnen.org
[email protected]