# How to edit a file without loading it all into memory

**URL:** <https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745>\
**Category:** Ruby\
**Created:** [November 9, 2007, 9:51pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745 "2007-11-09T21:51:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 9, 2007, 9:51pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/1 "2007-11-09T21:51:31Z")

</div>

I need to change the texture paths in maya files. The lines are easy to  
find with grep, and I’m able to create the new line easily enough, but  
how can I do this without holding the entire file into memory? The  
files can be hundreds of megabytes.

Thanks.

---

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 9, 2007, 10:04pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/2 "2007-11-09T22:04:43Z")

</div>

newer versions of sed can replace text in place in files.

there’s also perl’s

perl -pi -e ‘s/to change this/to change that/’ filename

---

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 9, 2007, 10:10pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/3 "2007-11-09T22:10:34Z")

</div>

On 09.11.2007 21:51, Jonathan D. wrote:

> I need to change the texture paths in maya files. The lines are easy to  
> find with grep, and I’m able to create the new line easily enough, but  
> how can I do this without holding the entire file into memory? The  
> files can be hundreds of megabytes.

As you said: do it line by line. This is the usual approach to this  
problem. You might want to have a look at Ruby’s command line options.  
Basically you can do something like this for an inplace edit:

ruby -ni.bak -e ‘your replacement code’

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 9, 2007, 10:10pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/4 "2007-11-09T22:10:43Z")

</div>

Jonathan D. wrote:

> The lines are easy to  
> find with grep, and I’m able to create the new line easily enough, but  
> how can I do this without holding the entire file into memory? The  
> files can be hundreds of megabytes.

File.foreach(“data.txt”) do |line|  
#look for something in line  
end

---

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 9, 2007, 10:40pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/5 "2007-11-09T22:40:23Z")

</div>

Err.

Well only one of these textures is open at once right?  
If this box is actually capable of running maya well, it’s sure to  
have more than a few hundred megs of free ram (when maya isn’t running  
of course ;).

My guess is that you’ll get a hefty performance hit for parsing the  
files line by line. The box has memory to spare, so why not just use  
it?

Of course if there was a way to memap the file under ruby that would  
probably be the best…but I dunno how to do that.

–Kyle

---

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 9, 2007, 10:21pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/6 "2007-11-09T22:21:29Z")

</div>

7stud – wrote:

> Jonathan D. wrote:
> 
> > The lines are easy to  
> > find with grep, and I’m able to create the new line easily enough, but  
> > how can I do this without holding the entire file into memory? The  
> > files can be hundreds of megabytes.
> 
> File.foreach(“data.txt”) do |line|  
> #look for something in line  
> end

Here is what I originally wrote, and what I just tried. Am I missing  
something obvious? The first one duplicates the lines, and the second  
one does nothing. (doesn’t save the changes)

def change\_texture\_paths  
File.open(Dir.glob(ENV[‘HOME’] + “/\*.ma”).first, “r+”) do |file|  
file.grep( /setAttr “.fileTextureName” -type “string”/ ) do |line|  
texture\_file\_path = line[52…-2]  
texture\_file\_name = texture\_file\_path.split("/").last  
line = " setAttr “.fileTextureName” -type “string”  
“” + ENV[‘HOME’] + “/include/” + texture\_file\_name + “\n”  
file.print line  
end  
end

```
File.foreach(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |line|
  if (line =~ /setAttr ".fileTextureName" -type "string"/ )
    texture_file_path = line[52..-2]
    texture_file_name = texture_file_path.split("\/").last
    line = " setAttr \".fileTextureName\" -type \"string\" 

```

“” + ENV[‘HOME’] + “/include/” + texture\_file\_name + “\n”  
file.print line  
end  
end  
end

---

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 10, 2007, 1:54am UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/7 "2007-11-10T01:54:19Z")

</div>

Jonathan D. wrote:

> 7stud – wrote:
> 
> > Jonathan D. wrote:
> > 
> > > The lines are easy to  
> > > find with grep, and I’m able to create the new line easily enough, but  
> > > how can I do this without holding the entire file into memory? The  
> > > files can be hundreds of megabytes.
> > 
> > File.foreach(“data.txt”) do |line|  
> > #look for something in line  
> > end
> 
> Here is what I originally wrote, and what I just tried. Am I missing  
> something obvious? The first one duplicates the lines, and the second  
> one does nothing. (doesn’t save the changes)

1. Are you under the impression that every method in ruby has the same  
parameters?

2. When you see an error message, do you just ignore it?

---

<div class="post-metadata">

**Author:** ![jonathan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/aeb1de/32.png) [@jonathan](https://www.ruby-forum.com/u/jonathan)\
**Post date:** [November 9, 2007, 10:42pm UTC](https://www.ruby-forum.com/t/how-to-edit-a-file-without-loading-it-all-into-memory/119745/8 "2007-11-09T22:42:08Z")

</div>

mem map. not memap…but you get the idea.