FasterCSV.foreach loop

baiki · September 13, 2009, 9:27pm

Hello community,

This is the way my CSV file looks like:

id,name,valid_time,date
1,baiki,5:00,9/10/2009
1,baiki,1:45,9/14/2009
2,lee,6:00,9/8/2009
3,fatou,9:15,8/14/2008
3,fatou,10:15,8/13/2008
2,lee,6:00,9/7/2009
2,lee,6:55,9/6/2009
2,lee,6:00,9/8/2009
1,baiki,5:00,9/11/2009
1,baiki,4:00,9/12/2009
1,baiki,2:00,9/13/2009
3,fatou,12:15,8/13/2008
3,fatou,12:15,8/18/2008

…and this is my Ruby, FasterCSV (FCSV) approach to find unique id’s
out of my CSV file and afterwards read all records from those employees,
do some calculations and then generate a ODF file (using
sandros-odf-report) per employee.

But I ran into a problem - ok, I admit it: SEVERAL PROBLEMS :-).

#!/usr/bin/ruby

require ‘rubygems’
require ‘odf-report’
require ‘fastercsv’

version = ‘0.01’
puts “\nGorilla CSV Convert Version
#{version}\n================================”
if ARGV[0].to_s.empty?
puts “ERROR: No file name given. Please call me this way:\nExample: $
ruby do_payment_slips.rb july_2009.csv\n\n”
exit
else
puts “File to load: #{ARGV[0]}\n\n”
input_file = ARGV[0]
end

employee_id = []
FasterCSV.foreach(input_file, :headers => true) do |row|
employee_id.concat(row[0].to_a).uniq!
end

until employee_id.empty?
FasterCSV.foreach(input_file, :headers => true) do |row|
if row[0] == employee_id[0]
puts “value: #{row}”
next
end
employee_id.shift
end
end

Inside the if-loop, I try to print the current value of row, so, it
should be:
5 records of id 1 and
4 record of id 2 and
4 records of id 3

But I get just the records of id 1, 1, 3.

What am I missing?

Thanks for any help welcome

Baiki

baiki · September 13, 2009, 10:10pm

On Sep 13, 2009, at 2:28 PM, Dot Baiki wrote:

Hello community,

Hello.

2,lee,6:00,9/8/2009
1,baiki,5:00,9/11/2009
1,baiki,4:00,9/12/2009
1,baiki,2:00,9/13/2009
3,fatou,12:15,8/13/2008
3,fatou,12:15,8/18/2008

Is this file large? It looks like you may eventually just want all of
the data in it. If that’s the case, and it’s not too big to suck the
whole thing into memory, it may be easier to work with that way. For
example:

data = FCSV.table(input_file)
p data[:id].uniq # these are the uniq ID’s

But I ran into a problem - ok, I admit it: SEVERAL PROBLEMS :-).

until employee_id.empty?
FasterCSV.foreach(input_file, :headers => true) do |row|
if row[0] == employee_id[0]
puts “value: #{row}”
next
end
employee_id.shift
end
end

But I get just the records of id 1, 1, 3.

What am I missing?

You remove an ID from employee_id after reading each line of your CSV
data. I think you meant to do something more like this:

employee_id.each do |id|
FCSV.foreach(input_file, :headers => true) do |row|
puts “value: #{row}” if row[“id”] == id
end
end

Hope that helps.

James Edward G. II

baiki · September 16, 2009, 11:38am

James Edward G. II wrote:

On Sep 13, 2009, at 2:28 PM, Dot Baiki wrote:

Hello.

Hi there again!

How are things going over there? Hope fine

Is this file large? It looks like you may eventually just want all of
the data in it. If that’s the case, and it’s not too big to suck the
whole thing into memory, it may be easier to work with that way. For
example:

Well, I expect that this CSV file contain about 12-15 columns and aprox.
36k rows. I followed your suggestion.

You remove an ID from employee_id after reading each line of your CSV
data. I think you meant to do something more like this:

employee_id.each do |id|
FCSV.foreach(input_file, :headers => true) do |row|
puts “value: #{row}” if row[“id”] == id
end
end
Amazing. Lovely solution. Sweet The only thing I had to do was to
add “.to_s”…here:
puts “value: #{row}” if row[“id”] == id.to_s

Hope that helps.

Indeed it did. Big thank you! Now I have to sort them and send them
trough odf-report and soon soon I will see something working. We’ll keep
in touch.

James Edward G. II

dot baiki

baiki · September 16, 2009, 7:11pm

Good day

Thanks to Mr. James Edward G. II, I understand more and more

After trying to create an array of arrays, I got stuck again. It seems I
can’t properly create such an array. This is my mistake:

data = FCSV.table(input_file)
employee_id = data[:id].uniq

employee_id.each do |id|
employee_record = []
FCSV.foreach(input_file, :headers => true) do |row|
if row[“id”].to_i == id
employee_record << [row]

create_odf_file(employee_record)

end

end
break #just for testing once
end

I saw that FasterCSV.foreach creates an array/hash/something?
#<FasterCSV::Table mode:col_or_row row_count:14>

My array looks like this, but I can’t access certain fields:
[[#<FasterCSV::Row “id”:“1” “name”:“baiki” “valid_time”:“5:00”
“date”:“9/10/2009”>], [#<FasterCSV::Row “id”:“1” “name”:“baiki”
“valid_time”:“1:45” “date”:“9/14/2009”>], [#<FasterCSV::Row “id”:“1”
“name”:“baiki” “valid_time”:“5:00” “date”:“9/11/2009”>],
[#<FasterCSV::Row “id”:“1” “name”:“baiki” “valid_time”:“4:00”
“date”:“9/12/2009”>], [#<FasterCSV::Row “id”:“1” “name”:“baiki”
“valid_time”:“2:00” “date”:“9/13/2009”>]]

If I want an array which contains all records from one employee, should
I use FasterCSV’s filter class method?

Or, how else can I pass this information to a class (ie. odf-report) in
order to create the ODF output?

I’ll keep trying. BTW: If your once in our country, let me know

Greetings
dot baiki

baiki · September 16, 2009, 8:52pm

Hello again

Thanks again for your advice. This is what I have now:

csv_data = FCSV.table(input_file)
employees_id = csv_data[:id].uniq

employees_id.each do |id|
puts “ID: #{id}”
employees_records = csv_data.select { |row| row[“id”] == id }
puts employees_records
break # just for now
end

The thing is, no output. Empty. Right now I really don’t get it. I am
using old software? That’s what I use:

ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]
fastercsv (1.5.0, 1.2.3) ** maybe I should deinstall one??? **

This one doesn’t give me any output as well.

csv_data = FCSV.table(input_file)
employees_id = csv_data[:id].uniq
employees_records = csv_data.select { |row| row[“id”] == 1 }
p employees_records

Hmmm. Hope you can give me another clue!

Regards,
Baiki

baiki · September 16, 2009, 9:06pm

Oh, I see:

csv_data = FCSV.table(input_file)
employees_id = csv_data[:id].uniq

employees_id.each do |id|
puts “ID: #{id}”

employees_records = csv_data.select { |row| row[:id] == id }
puts employees_records
break # just for now
end

:id did the trick. Will try more.

Thanks
Baiki

baiki · September 16, 2009, 8:04pm

On Sep 16, 2009, at 12:11 PM, Dot Baiki wrote:

Good day

Thanks to Mr. James Edward G. II, I understand more and more

I’m glad to hear it.

After trying to create an array of arrays, I got stuck again. It
seems I
can’t properly create such an array. This is my mistake:

data = FCSV.table(input_file)

When you execute the above line you read all of the data into memory.
You would not need to do any more reading, like making calls to
FCSV.foreach().

I saw that FasterCSV.foreach creates an array/hash/something?
#<FasterCSV::Table mode:col_or_row row_count:14>

The data variable above holds a FasterCSV::Table which is a container
for some FasterCSV::Row objects. They represent your data.

If I want an array which contains all records from one employee,
should
I use FasterCSV’s filter class method?

You could grab the records for one employee using code like:

employee1s_records = data.select { |row| row[“id”] == 1 }

Hope that helps.

James Edward G. II

baiki · September 16, 2009, 9:31pm

I guess I need your help once again.

I need “employees_records” in another class (want to create ODF files
per employee). Should I try to convert this variable into a hash or
should I try to access certain fields within “employees_records” using
one of your techniques? How can I access one cell within
“employees_records” or how can I change it to a proper hash (I really do
prefer arrays)

Thank you.
Baiki

baiki · September 16, 2009, 9:42pm

On Sep 16, 2009, at 1:53 PM, Dot Baiki wrote:

end

The thing is, no output. Empty.

Sorry, I messed up. My fault.

Try replacing:

row[“id”]

with:

row[:id]

I bet that fixes you up.

James Edward G. II

baiki · September 16, 2009, 9:48pm

On Sep 16, 2009, at 2:31 PM, Dot Baiki wrote:

I guess I need your help once again.

I need “employees_records” in another class (want to create ODF files
per employee). Should I try to convert this variable into a hash or
should I try to access certain fields within “employees_records” using
one of your techniques? How can I access one cell within
“employees_records” or how can I change it to a proper hash (I
really do
prefer arrays)

How’s this?

employees_records = csv_data.select { |row| row[:id] == 1 }.
map { |row| row.to_hash }

James Edward G. II

baiki · September 17, 2009, 2:24am

James Edward G. II wrote:

How’s this?

employees_records = csv_data.select { |row| row[:id] == 1 }.
map { |row| row.to_hash }
Good evening,

This was just brilliant! Finally some progress (=hope) Will be back
tomorrow.

Thanks so much.
Baiki

baiki · September 17, 2009, 11:40am

Good day again

I came across this article
(http://groups.google.com/group/ruby-reports/browse_thread/thread/7cba5390d1aa3798)
where was written that sorting tables or rows should not be possible? Is
that a fact? Can’t imagine. How could I sort such a table. Is this a
Ruby function or from FCSV?

Regards,
Baiki

baiki · January 11, 2011, 2:50am

Dear Mr. James Edward G. II

I am sorry for late reply. I just wanted to say: it did not work out,
but because of the odf-report part.

Thanks for all the help!

Regards

baiki · September 17, 2009, 2:25pm

On Sep 17, 2009, at 4:40 AM, Dot Baiki wrote:

I came across this article
(http://groups.google.com/group/ruby-reports/browse_thread/thread/7cba5390d1aa3798
)
where was written that sorting tables or rows should not be
possible? Is
that a fact? Can’t imagine. How could I sort such a table. Is this a
Ruby function or from FCSV?

That article just says that FasterCSV doesn’t provide a method to sort
a table and reorder it in memory. You can always create a second
table and reorder elements as you move them over, reorder elements as
you write the results out to disk, etc.

James Edward G. II

baiki · January 11, 2011, 3:17am

Hello again,

I know it’s not your problem/software but maybe you see the mistake I
made here:

report = ODFReport::Report.new(‘gorilla.odt’) do |r|
r.add_field :employees_name, employees_name
r.add_field :employees_id, employees_records[0][:id]
r.add_field :employees_month_of_payment, employees_month_of_payment
i = -1
r.add_table(‘table_time_details’, employees_records) do |t|
i += 1
t.add_column(:employees_date, employees_records[i][:date])
end

r.add_field :employees_monthly_total, employees_monthly_total
r.add_field :software_name, software_name
r.add_field :version, version

end

Seems like this line is the problem: t.add_column(:employees_date,
employees_records[i][:date])

However, thanks anyway

Baiki

baiki · January 11, 2011, 4:16am

On Jan 10, 2011, at 8:17 PM, Dot Baiki wrote:

r.add_table(‘table_time_details’, employees_records) do |t|
employees_records[i][:date])
Sorry, I don’t know what ODFReport is.

James Edward G. II