Dynamically import csv files

I’m hoping somebody can give me some direction on how to approach this
problem. We have various projects with data acquisition in the field.
These projects produce time series data of variables such as
temperature, and power usage. I’m using javascript to plot the data.
Currently I have a form in which the user would submit the project
name, the type of data (60 minute, 1 minute, hourly, etc.) and the
names of the variables requested. This generates an ajax request to
find and parse a csv file to get the variables and send them back to
the browser to plot.

The csv files are updated daily. New projects are created often. Each
project has several types of data with many variables and many rows.

I’d like to have the data all in a database, possibly ActiveRecord,
taking advantage of validations and such. I’ve thought of two
approaches so far:

  • Creating a unique new table for every project and for every type of
    data (e.g. data_project1_60minute). This results in creating new
    classes for every table as well, which doesn’t seem optimal. There is
    also the possibility of using the “set_table_name” feature of
    activerecord, but I wonder if this is the best way.

  • Creating one massive table for every type of data (e.g.
    data_60minute, with column names project1_var1, project1_var2,
    project2_var1, etc.). Then adding columns whenever a new project is
    added. This also doesn’t seem great since one data’s 60minute has time
    differences between the others. There is also event driven data which
    collects at high frequency for short periods of time, which also
    doesn’t fit this model.

Really it seems like I want some of the features of ActiveRecord, but
I don’t really want the object relational mapping, and it gets in the
way by requiring me to make a bunch of new classes.

Thanks for any advice that someone can give.


You could also create two tables, one which describes a particular
projects column use and another which stores the actual data columns.
The first metadata table would have one row per project and the data
in the second would identify which project they are associated with.
This way you just create a new row for a new project and never change
the table structures. The validations can be driven by the metadata
table specifications.


This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs