Forum: Ruby on Rails Dynamically import csv files

Announcement (2017-05-07): is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see and for other Rails- und Ruby-related community platforms.
2e841320f01b170e0910ffefcc5fae27?d=identicon&s=25 Lee (Guest)
on 2009-03-31 02:27
(Received via mailing list)
I'm hoping somebody can give me some direction on how to approach this
problem. We have various projects with data acquisition in the field.
These projects produce time series data of variables such as
temperature, and power usage. I'm using javascript to plot the data.
Currently I have a form in which the user would submit the project
name, the type of data (60 minute, 1 minute, hourly, etc.) and the
names of the variables requested. This generates an ajax request to
find and parse a csv file to get the variables and send them back to
the browser to plot.

The csv files are updated daily. New projects are created often. Each
project has several types of data with many variables and many rows.

I'd like to have the data all in a database, possibly ActiveRecord,
taking advantage of validations and such. I've thought of two
approaches so far:

- Creating a unique new table for every project and for every type of
data (e.g. data_project1_60minute). This results in creating new
classes for every table as well, which doesn't seem optimal. There is
also the possibility of using the "set_table_name" feature of
activerecord, but I wonder if this is the best way.

- Creating one massive table for every type of data (e.g.
data_60minute, with column names project1_var1, project1_var2,
project2_var1, etc.). Then adding columns whenever a new project is
added. This also doesn't seem great since one data's 60minute has time
differences between the others. There is also event driven data which
collects at high frequency for short periods of time, which also
doesn't fit this model.

Really it seems like I want some of the features of ActiveRecord, but
I don't really want the object relational mapping, and it gets in the
way by requiring me to make a bunch of new classes.

Thanks for any advice that someone can give.
054ea2f04b5592b91f8223796cc53979?d=identicon&s=25 Brendon Whateley (brendon)
on 2009-04-04 14:07
(Received via mailing list)

You could also create two tables, one which describes a particular
projects column use and another which stores the actual data columns.
The first metadata table would have one row per project and the data
in the second would identify which project they are associated with.
This way you just create a new row for a new project and never change
the table structures.  The validations can be driven by the metadata
table specifications.

This topic is locked and can not be replied to.