Appending to a CSV file

Bil_K · July 12, 2006, 4:36pm

Just got a call from a telecommuting colleague
who’s using Ruby to automate a bunch of CEV[1]
CFD[2] simulations.

She says that she can’t figure out how to /append/
to a CSV file using Ruby’s standard library.

Does anyone have the magic recipe, or should
she be using FasterCSV instead?

Thanks,

Bil

[1] CEV = Crew Exploration Vehicle
[2] CFD = Computational Fluid Dynamics

Bil_K · July 12, 2006, 7:28pm

Bil K. wrote:
…

She says that she can’t figure out how to /append/
to a CSV file using Ruby’s standard library.
…

Wouldn’t it just be a matter of opening the file in append mode?

cheers
Chris

Bil_K · July 12, 2006, 8:54pm

ChrisH wrote:

Bil K. wrote:
…

She says that she can’t figure out how to /append/
to a CSV file using Ruby’s standard library.
…

Wouldn’t it just be a matter of opening the file in append mode?

cheers
Chris

Ok, note to self “read docs before commenting”

it seems CSV only supports ‘r’,‘rb’,‘w’,‘wb’ for open.
FasterCSV delegates to an IO object and supports all IO’s modes.
So it would be the winner

Cheers

Bil_K · July 12, 2006, 9:39pm

ChrisH wrote:

Bil K. wrote:
…

She says that she can’t figure out how to /append/
to a CSV file using Ruby’s standard library.
…

Wouldn’t it just be a matter of opening the file in append mode?

It would, but unfortunately CSV doesn’t support ‘a’
or ‘a+’, viz, lib/ruby/1.8/csv.rb,

  83   def CSV.open(path, mode, fs = nil, rs = nil, &block)
  84     if mode == 'r' or mode == 'rb'
  85       open_reader(path, mode, fs, rs, &block)
  86     elsif mode == 'w' or mode == 'wb'
  87       open_writer(path, mode, fs, rs, &block)
  88     else
  89       raise ArgumentError.new("'mode' must be 'r', 'rb', 'w',

or ‘wb’")
90 end
91 end

Later,

Bil_K · July 12, 2006, 9:26pm

On Jul 12, 2006, at 1:55 PM, Chris H. wrote:

it seems CSV only supports ‘r’,‘rb’,‘w’,‘wb’ for open.
FasterCSV delegates to an IO object and supports all IO’s modes.
So it would be the winner

Interestingly, this design choice in FasterCSV has been questioned in
the past. I just keep liking it more and more though as issues like
this arise.

James Edward G. II

Bil_K · July 12, 2006, 9:40pm

James G. wrote:

On Jul 12, 2006, at 1:55 PM, Chris H. wrote:

it seems CSV only supports ‘r’,‘rb’,‘w’,‘wb’ for open.
FasterCSV delegates to an IO object and supports all IO’s modes.
So it would be the winner

Interestingly, this design choice in FasterCSV has been questioned in
the past. I just keep liking it more and more though as issues like
this arise.

James Edward G. II

It does seem a bit odd that CSV limits the modes.

On the other hand, the desired result can be achieved by creating a
new file, copy existing data, add the new data, delete (or rename to be
safe)
the original and rename the new file to the old file name.

I think supporting the append mode is easier all round…

cheers
Chris

Bil_K · July 13, 2006, 4:12am

Bil K. wrote:

Wouldn’t it just be a matter of opening the file in append mode?

It would, but unfortunately CSV doesn’t support ‘a’
or ‘a+’, viz, lib/ruby/1.8/csv.rb,

Well, darn.

All I can think of offhand is 1) write the new stuff to a new file
and then 2) use fileutils to append the new file onto the old one.

Disclaimers:

I’m not sure fileutils has a good way to do that, but
if it doesn’t, there’s always system(“cat newfile >>oldfile”).
Obviously be sure the old file is closed before trying to
so anything funny with it.

Hal

Bil_K · July 12, 2006, 11:52pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Bil K. wrote:

Just got a call from a telecommuting colleague
who’s using Ruby to automate a bunch of CEV[1]
CFD[2] simulations.

She says that she can’t figure out how to /append/
to a CSV file using Ruby’s standard library.

Does anyone have the magic recipe, or should
she be using FasterCSV instead?

Please use CSV::Writer. An instance of CSV is not an IO in contrast to
a FasterCSV.

CSV::Writer.generate(file_to_append) do |csv|
csv << [1, 2, 3]
end

Regards,
// NaHi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (Cygwin)

iQEVAwUBRLVubx9L2jg5EEGlAQKfNwf/SJvTZox2avqI+HMjUqHt/lkVK2/oIvHQ
96ZjcxXRUGXwIX/e93RGkczJVeygcF7XnpNfDHWJmSInDjhVdjz0SJqsrODMLtBI
1kDhWwqSCCGXfxBAr+JH/QYdC1Bf0pfO15vdWo0eY9k3zlQ/UKLT5uEicPzvnPlD
hEYIhn0rZnfb3GR7ZTUMECMlnL1zg+wQ7Y4S/r3sW685QXHCSC3iGclrd0ngDIz0
hJOcUvg1Idp9YgfZ+DA98YICIiG+o9iZM1fnk0PUhyncGMLHjIMYlqeeqXdgMX+G
UGpB2Mf+pZBxV7QeYC3M+qAXL6yHMoqF06udVAYyeGYGh7BJ80zamg==
=X1oC
-----END PGP SIGNATURE-----

Bil_K · July 13, 2006, 3:14pm

I’m not very familiar with the CSV lib, but if the heart of the matter
is appending to a CSV file, you hardly need a full-on CSV lib for
something like that.

File.open(my_csv) {|f| f.puts new_record} # think it oughta be as
simple as that
File.open(my_csv) {|f| f.puts *new_records} # puts can take a list, too

Bil_K · July 13, 2006, 3:10pm

Sorry about my last post sort of skipping over the whole matter of the
write mode… you’re smarter than me, though, I’m sure you’ll be fine.
;-p

Bil_K · July 13, 2006, 5:13pm

Bil K. wrote:
…

She says that she can’t figure out how to /append/
to a CSV file using Ruby’s standard library.
…

Not sure if you read ruby-forum (http://www.ruby-forum.com/)
but a response ther indicates that you can do this via the CSV::Writer:

File.open(‘csvout.csv’, ‘a’){ |outfile|
CSV::Writer.generate(outfile) do |csv|
csv << [‘c1’, nil, ‘’, ‘"’, “\r\n”, ‘c2’]
end
}

Cheers
Chris

Bil_K · July 14, 2006, 3:35am

I realize that I am jumping late into this thread. I do not understand
why the responses to “How to append to a csv file” are so complex. Why
do you need a csv module to create, edit, and/or add records to a csv
file.

As a newbie, I have written Ruby scripts that created csv file and
appended records to existing csv files without having the need for a
csv module. To append a record, I simply open the file in ‘a’ mode
(have not needed the a+ mode yet), write the record to the file and
then close file.

I apolized if I am out of term and stuck my nose in the wrong place.

Bil_K · July 13, 2006, 3:52pm

I haven’t used the CSV module before either, but looking into it this
test script I came up with parses a CSV source file, writing it to a
new array. Then you can append to that new array and write the results
back to the original CSV file. It’s not pretty and might not be the
most Rubyish way of getting the job done but it worked for me using
Ruby 1.8.2 on Win32. Try it out and let me know…

require ‘csv’

test.csv consists of three comma delimited fields, something like:

1,“Greg”,“Admin”

2,“Joe”,“User”

3,“Jim”,“Admin”

new_array = []
temp_array = []

reader = CSV.open(‘test.csv’, ‘r’) do |row|
(0…row.nitems).each do |item|
temp_array.push(row[item].data)
end
new_array << temp_array
temp_array = []
end

new_items = [4,“Jane”,“User”]
new_array << new_items

CSV.open(‘test.csv’, ‘w’) do |writer|
new_array.each do |row|
writer << row
end
end

Bil_K · July 14, 2006, 3:48am

I can’t speak for the OP, but I am assuming that they were using the
CSV module because there might have been some complications in
processing the data source. And perhaps some of the CSV module’s
methods helped streamline things so they didn’t have to reinvent the
wheel.

No matter if you are using the CSV module or rolling your own way of
processing a data source file, you would have to parse through the
existing data in some block form and then append to the end of the
stream/array/whatever to accomplish what the OP was looking to do.

If you post a code snippet of your way to accomplishing this without
using the CSV module I would think that the basic concept and workflow
of the script would be similar…

Bil_K · July 16, 2006, 3:11am

gregarican wrote:

If you post a code snippet of your way to accomplishing this without
using the CSV module I would think that the basic concept and workflow
of the script would be similar…

As I said I am a newby to Ruby but not to programming in general.

Now to me a csv file is simply a flatfile database stored in a text
file. The only specifications are that fields have a separator
(normally a comma) and the records have a separator (normally a
newline). In addition neither of these separators can appear within a
field.

So to me, I only need to read a record and write a record. Once a
record is read, how I slice or dice the record does not require a csv
module that provides unique functions to process the record. Remember a
csv record is a string, period. There are no other structural
requirements.

Now it is up to me to verify the sanity of the records and believe me
no module can be written to cover all possiblities since the records
are free-form and can have any format what so ever.

Usually, I open the csv file, read in the records into an array, close
the file and then process the records as appropriately. Should I need
to write those records back I either re-open the csv file in write mode
(which clobbers the original file) or open a newfile in write mode.
Write out each record. If I want to keep the original file as a back
up, I can rename it and give the new file the old name. If my program
generates new records that need to be added to an existing file, I open
the file in append mode and writeout the record(s).

I normally do not use csv files as a permanent database. They usually
are “pass-through” files…an external program may write it outputs in a
csv file which my program processes and/or my program may store its
outputs in a csv file for another program to use.