Forum: Ruby trouble reading a set of files recursively

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
(Guest)
on 2009-04-20 18:51
(Received via mailing list)
Hello,
I am doing a batch run, where I have several .csv files with the same
name in different sub-directories. At the end of the batch run, I want
to cat all the individual files together, and load them into the
database.

Here is the code:
Dir["**/store_need*"].each do |path|
      single_file_nm = path.to_s
      single_file = File.open("#{single_file_nm}", "r")
        single_file.each do |line|
          store_need_file << line
          puts line
        end
      single_file.close
end

I am recursively looking for the store_need.csv files, and then
reading them into a single store_need_file. The problem is that this
code is not reading the file at all. I can see that the path is
defined correctly.

Any help is appreciated. I am running this on windows XP.

Kaushik
Don W. (Guest)
on 2009-04-20 19:04
removed_email_address@domain.invalid wrote:
>
> Here is the code:
> Dir["**/store_need*"].each do |path|
>       single_file_nm = path.to_s
>       single_file = File.open("#{single_file_nm}", "r")
>         single_file.each do |line|
>           store_need_file << line
>           puts line
>         end
>       single_file.close
> end
>

I just tried your code on a Linux box, and it ran as it should.  I don't
do Windows, but at least this tells you to focus on OS specific issues.
Could it have something to do with drive letters or slashes instead of
backslashes?
Robert K. (Guest)
on 2009-04-20 19:33
(Received via mailing list)
2009/4/20 removed_email_address@domain.invalid 
<removed_email_address@domain.invalid>:
> I am doing a batch run, where I have several .csv files with the same
> name in different sub-directories. At the end of the batch run, I want
> to cat all the individual files together, and load them into the
> database.
>
> Here is the code:
> Dir["**/store_need*"].each do |path|
>                        single_file_nm = path.to_s

Why do you convert a string to a string and then do string
interpolation in the next line?

>                        single_file = File.open("#{single_file_nm}", "r")

The string interpolation is superfluous plus you should rather use the
block form of File.open.

>                                single_file.each do |line|
>                                        store_need_file << line

Where do you open store_need_file?

>                                        puts line
>                                end
>                        single_file.close
> end
>
> I am recursively looking for the store_need.csv files, and then
> reading them into a single store_need_file. The problem is that this
> code is not reading the file at all. I can see that the path is
> defined correctly.

This is a shorter variant if your individual CSV files are small:

File.open "load.csv", "w" do |io|
  Dir["**/store_need*"].each do |path|
    # note: puts instead of write to get the terminating newline
    io.puts(File.read(path))
  end
end

If your individual files are large I would probably not bother to
combine them.  If you still want to do it then I would do something
like this

# untested
NL = "\r\n".freeze
BL = 1024

File.open "load.csv", "wb" do |fout|
  buffer = ""

  Dir["**/store_need*"].each do |path|
    File.open path, "rb" do |fin|
      while fin.read(BL, buffer)
        fout.write(buffer)
      end
    end

    # only needed if files are not terminated properly:
    # fout.write(NL)
  end
end

Cheers

robert
This topic is locked and can not be replied to.