How to edit a file without loading it all into memory

Ruby
1

I need to change the texture paths in maya files. The lines are easy to
find with grep, and I’m able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

Thanks.

2

newer versions of sed can replace text in place in files.

there’s also perl’s

perl -pi -e ‘s/to change this/to change that/’ filename

3

On 09.11.2007 21:51, Jonathan D. wrote:

I need to change the texture paths in maya files. The lines are easy to
find with grep, and I’m able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

As you said: do it line by line. This is the usual approach to this
problem. You might want to have a look at Ruby’s command line options.
Basically you can do something like this for an inplace edit:

ruby -ni.bak -e ‘your replacement code’

Kind regards

robert

4

Jonathan D. wrote:

The lines are easy to
find with grep, and I’m able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

File.foreach(“data.txt”) do |line|
#look for something in line
end

5

Err.

Well only one of these textures is open at once right?
If this box is actually capable of running maya well, it’s sure to
have more than a few hundred megs of free ram (when maya isn’t running
of course ;).

My guess is that you’ll get a hefty performance hit for parsing the
files line by line. The box has memory to spare, so why not just use
it?

Of course if there was a way to memap the file under ruby that would
probably be the best…but I dunno how to do that.

–Kyle

6

7stud – wrote:

Jonathan D. wrote:

The lines are easy to
find with grep, and I’m able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

File.foreach(“data.txt”) do |line|
#look for something in line
end

Here is what I originally wrote, and what I just tried. Am I missing
something obvious? The first one duplicates the lines, and the second
one does nothing. (doesn’t save the changes)

def change_texture_paths
File.open(Dir.glob(ENV[‘HOME’] + “/*.ma”).first, “r+”) do |file|
file.grep( /setAttr “.fileTextureName” -type “string”/ ) do |line|
texture_file_path = line[52…-2]
texture_file_name = texture_file_path.split("/").last
line = " setAttr “.fileTextureName” -type “string”
“” + ENV[‘HOME’] + “/include/” + texture_file_name + “\n”
file.print line
end
end

File.foreach(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |line|
  if (line =~ /setAttr ".fileTextureName" -type "string"/ )
    texture_file_path = line[52..-2]
    texture_file_name = texture_file_path.split("\/").last
    line = "        setAttr \".fileTextureName\" -type \"string\"

“” + ENV[‘HOME’] + “/include/” + texture_file_name + “\n”
file.print line
end
end
end

7

Jonathan D. wrote:

7stud – wrote:

Jonathan D. wrote:

The lines are easy to
find with grep, and I’m able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

File.foreach(“data.txt”) do |line|
#look for something in line
end

Here is what I originally wrote, and what I just tried. Am I missing
something obvious? The first one duplicates the lines, and the second
one does nothing. (doesn’t save the changes)

  1. Are you under the impression that every method in ruby has the same
    parameters?

  2. When you see an error message, do you just ignore it?

8

mem map. not memap…but you get the idea.