Forum: Ruby trouble reading a set of files recursively

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
9f72aa5ee72bd13a8b443fbf3797944c?d=identicon&s=25 katari@gmail.com (Guest)
on 2009-04-20 16:51
(Received via mailing list)
Hello,
I am doing a batch run, where I have several .csv files with the same
name in different sub-directories. At the end of the batch run, I want
to cat all the individual files together, and load them into the
database.

Here is the code:
Dir["**/store_need*"].each do |path|
      single_file_nm = path.to_s
      single_file = File.open("#{single_file_nm}", "r")
        single_file.each do |line|
          store_need_file << line
          puts line
        end
      single_file.close
end

I am recursively looking for the store_need.csv files, and then
reading them into a single store_need_file. The problem is that this
code is not reading the file at all. I can see that the path is
defined correctly.

Any help is appreciated. I am running this on windows XP.

Kaushik
4a5ad89e5a16e46dc7155fa8c6cee83c?d=identicon&s=25 Don Wood (tinnidril)
on 2009-04-20 17:04
katari@gmail.com wrote:
>
> Here is the code:
> Dir["**/store_need*"].each do |path|
>       single_file_nm = path.to_s
>       single_file = File.open("#{single_file_nm}", "r")
>         single_file.each do |line|
>           store_need_file << line
>           puts line
>         end
>       single_file.close
> end
>

I just tried your code on a Linux box, and it ran as it should.  I don't
do Windows, but at least this tells you to focus on OS specific issues.
Could it have something to do with drive letters or slashes instead of
backslashes?
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2009-04-20 17:33
(Received via mailing list)
2009/4/20 katari@gmail.com <katari@gmail.com>:
> I am doing a batch run, where I have several .csv files with the same
> name in different sub-directories. At the end of the batch run, I want
> to cat all the individual files together, and load them into the
> database.
>
> Here is the code:
> Dir["**/store_need*"].each do |path|
>                        single_file_nm = path.to_s

Why do you convert a string to a string and then do string
interpolation in the next line?

>                        single_file = File.open("#{single_file_nm}", "r")

The string interpolation is superfluous plus you should rather use the
block form of File.open.

>                                single_file.each do |line|
>                                        store_need_file << line

Where do you open store_need_file?

>                                        puts line
>                                end
>                        single_file.close
> end
>
> I am recursively looking for the store_need.csv files, and then
> reading them into a single store_need_file. The problem is that this
> code is not reading the file at all. I can see that the path is
> defined correctly.

This is a shorter variant if your individual CSV files are small:

File.open "load.csv", "w" do |io|
  Dir["**/store_need*"].each do |path|
    # note: puts instead of write to get the terminating newline
    io.puts(File.read(path))
  end
end

If your individual files are large I would probably not bother to
combine them.  If you still want to do it then I would do something
like this

# untested
NL = "\r\n".freeze
BL = 1024

File.open "load.csv", "wb" do |fout|
  buffer = ""

  Dir["**/store_need*"].each do |path|
    File.open path, "rb" do |fin|
      while fin.read(BL, buffer)
        fout.write(buffer)
      end
    end

    # only needed if files are not terminated properly:
    # fout.write(NL)
  end
end

Cheers

robert
This topic is locked and can not be replied to.