# File position and buffers

**URL:** <https://www.ruby-forum.com/t/file-position-and-buffers/206316>\
**Category:** Ruby\
**Created:** [April 27, 2011, 10:02pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316 "2011-04-27T22:02:32Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [April 27, 2011, 10:02pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/1 "2011-04-27T22:02:32Z")

</div>

Hi all,

In a bit of a rut. Have a file with a lot of text. I want to seperate  
the text in this file as entries. Each entry that I would be seperating,  
would be done so using IO.pos and when that cursor reaches a certain  
character in the file, it will ideally place all the content before that  
character into a buffer. Then the cursor will continue reading until it  
hits that same character again and put that content into a buffer, so on  
and so forth. (Character I’ll be reading would be a greater than symbol)

Would I use a do iterator or use a while loop with a gets method? Or  
readlines perhaps?

File:

> entry 1  
> rubyrubyrubyrubyrubyrubyrubyruby  
> (newline here which I don’t want)  
> entry 2  
> rubyrubyrubyrubyrubyrubyrubyruby

Entry1 and entry2 will be in seperate buffers which I would be able to  
access again.

buffer1 = \>entry 1  
rubyrubyrubyrubyrubyrubyrubyruby

buffer2 = \>entry 2  
rubyrubyrubyrubyrubyrubyrubyruby

PS. The file is huge, so I don’t want to read it into memory. What is  
the best way to approach this? Any suggestions or comments would be  
helpful. Thanks!

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 12:33am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/2 "2011-04-28T00:33:48Z")

</div>

hi Cee -

this may well be WAY to simple for your needs, but it seems to me you  
could so something like this:

(0text.txt is a file with 7 lines that say rubyrubyrubyetc.)

f = “0text.txt”  
file = File.open(f)  
buffer = []  
bufferindex = 0

file.each\_line{|line|  
buffer[bufferindex] = line.chomp  
bufferindex += 1  
}

p buffer[0]  
p buffer[1]  
p buffer[2]  
#etc…

of course you could also set a maximum number of lines per buffer:

f = “0text.txt”  
file = File.open(f)  
buffer = Hash.new{|key, value| key[value]= []}  
bufferkey = 0  
maxbuflength = 3

file.each\_line{|line|  
if buffer[bufferkey].length == maxbuflength  
bufferkey +=1  
buffer[bufferkey] \<\< line.chomp  
else  
buffer[bufferkey] \<\< line.chomp  
end  
}

p buffer[0]  
p buffer[1]  
p buffer[2]

if the file’s extremely long i guess you’d want to write a method to  
dump the buffers at some point too.

maybe this is dumb, i hope not!  
cheers,

-j

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 2:08am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/3 "2011-04-28T02:08:44Z")

</div>

Cee J. wrote in post #995381:

> Hi all,
> 
> In a bit of a rut. Have a file with a lot of text. I want to seperate  
> the text in this file as entries. Each entry that I would be seperating,  
> would be done so using IO.pos and when that cursor reaches a certain  
> character in the file, it will ideally place all the content before that  
> character into a buffer. Then the cursor will continue reading until it  
> hits that same character again and put that content into a buffer, so on  
> and so forth. (Character I’ll be reading would be a greater than symbol)

There is absolutely no reason to use pos() to read that file.

> Would I use a do iterator or use a while loop with a gets method? Or  
> readlines perhaps?
> 
> File:
> 
> > entry 1  
> > rubyrubyrubyrubyrubyrubyrubyruby  
> > (newline here which I don’t want)

chomp() removes one newline, if present, at the end of a string.

You say your file looks like this:

> entry 1 \<—WHAT’S AT THE END OF THIS LINE??  
> rubyrubyrubyrubyruby \<—WHAT’S AT THE END OF THIS LINE??  
> (newline here which I don’t want)

Are there newlines at the end of each of those strings? Are you saying  
that your data is organized into paragraphs, i.e. separated by two  
newlines? Like this:

> entry1\n  
> rubyrubyruby\n  
> \n  
> entry2\n  
> rubyrubyruby\n  
> \n  
> entry3

A paragraph is defined as two consective newlines between lines. Note  
that in ruby the default line separator is one newline. But you can  
change that to two newlines–or any other character:

require ‘stringio’

str =\<\<ENDOFSTRING

> entry1  
> 11111111111

> entry2  
> 22222222222

> entry3  
> 33333333333  
> ENDOFSTRING

input = StringIO.new(str)  
$/ = “\n\n”

input.each do |para|  
p para.sub(/\n+ \z/xms, “”)  
end

–output:–  
“\>entry1\n11111111111”  
“\>entry2\n22222222222”  
“\>entry3\n33333333333”

> Entry1 and entry2 will be in seperate buffers which  
> I would be able to access again.
> 
> buffer1 = \>entry 1  
> rubyrubyrubyrubyrubyrubyrubyruby

> buffer2 = \>entry 2  
> rubyrubyrubyrubyrubyrubyrubyruby

So you want to read two paragraphs at a time?

e = input.enum\_for(:each)

e.each\_slice(2) do |buffer1, buffer2|  
p buffer1, buffer2  
end

–output:–  
“\>entry1\n11111111111\n\n”  
“\>entry2\n22222222222\n\n”  
“\>entry3\n33333333333\n”  
nil

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 2:50am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/4 "2011-04-28T02:50:44Z")

</div>

This shows the output better:

e = input.enum\_for(:each) #You can do this for a File too.

e.each\_slice(2) do |buffer1, buffer2|  
puts “buffer1: #{buffer1.inspect}”  
puts “buffer2: #{buffer2.inspect}”  
puts “-” \* 10  
end

## –output:– buffer1: “\>entry1\n11111111111\n\n” buffer2: “\>entry2\n22222222222\n\n”

## buffer1: “\>entry3\n33333333333\n” buffer2: nil

Before doing the sub() on buffer2, you will have to check if it’s nil:

if buffer2.nil?  
#don’t do a sub()  
else  
#do the sub()  
end

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 9:55am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/5 "2011-04-28T09:55:23Z")

</div>

On Wed, Apr 27, 2011 at 10:02 PM, Cee J. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Would I use a do iterator or use a while loop with a gets method? Or  
> access again.  
> helpful. Thanks!  
> One of the simplest approaches is to use Ruby’s ability to use  
> arbitrary record delimiters:

File.foreach file\_name, “\>” do |chunk|  
chunk.chomp! “\>”  
chunk.gsub! /\r\n?|\n/, ‘’ # remove line terminators

# if you need the leading “\>”:

# chunk[0,0] = “\>”

p chunk  
end

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 4:52pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/6 "2011-04-28T16:52:09Z")

</div>

You could use foreach checking if each line starts with ‘\>’. If it  
doesn’t  
you accumulate in a buffer; if it does you do something with the current  
buffer and start a new one.

Jesus  
El 27/04/2011 22:04, “Cee J.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) escribi:

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 4:06pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/7 "2011-04-28T16:06:51Z")

</div>

Thanks guys for your helpful comments. I will be more descriptive. I am  
an intern and my mentor wants me to use the IO.pos to read the  
characters of the file until the character reaches the “\>” symbol. SO  
upon the cursor reaching the “\>” symbol(which is the start of a new  
entry), he wants me to place that previous entry in a buffer. Here is  
the actual test file I am working with:

> gi|329295464|ref|NM\_2005745.3Acc1| Def1 zgc:65895 (zgc:65895), mRNA\n  
> AGCTCGGGGGCTCTAGCGATTTAAGGAGCGATGCGATCGAGCTGACCGTCGCG\n  
> \n  
> gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA\n  
> GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG\n  
> CGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG\n  
> \n  
> gi|3542945647|ref|NM\_7453343.5Acc3| Def3 zgc:65895 (zgc:65895), mRNA\n  
> CGTGCGGGGABCCGTACGTGCCGTGGGGGTTTAATAGCGCGCCATCTGAGCAG\n  
> TTAGTCGCTGACGCATGCACG\n  
> \n

7stud, you are right there are two consecutive newlines which I failed  
to mention. This should be the output of a buffer for one entry:

> gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA\n  
> GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG ← no “\n”  
> CGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG ← no “\n”

Notice how the newlines are gone. So with the exception of the header in  
each entry, the newlines should be gone and be placed in a buffer. I am  
lost on how to use the IO.pos and a file iterator to make sure each  
respective entry goes into a buffer without the file being indexed into  
memory.

Thanks in advance, I’m new to the language and trying to wrap my head  
around it.

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 8:06pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/8 "2011-04-28T20:06:51Z")

</div>

Robert K. wrote in post #995478:

> On Wed, Apr 27, 2011 at 10:02 PM, Cee J. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > Would I use a do iterator or use a while loop with a gets method? Or  
> > access again.  
> > helpful. Thanks!  
> > One of the simplest approaches is to use Ruby’s ability to use  
> > arbitrary record delimiters:
> 
> File.foreach file\_name, “\>” do |chunk|  
> chunk.chomp! “\>”  
> chunk.gsub! /\r\n?|\n/, ‘’ # remove line terminators

Cee J., are you reading the file in binary mode or text mode?

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 8:25pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/9 "2011-04-28T20:25:18Z")

</div>

7stud – wrote in post #995589:

> Cee J., are you reading the file in binary mode or  
> text mode?

If you don’t know, then show us the line in your code where you open the  
file.

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 7:47pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/10 "2011-04-28T19:47:01Z")

</div>

You still have not told us what you are supposed to do with the stuff  
you read in?? You can read a file line by line and print out each line  
as you go and the maximum amount of memory used will be one line’s  
worth. However, if you are supposed to store all the lines in an  
array, then you will read the whole file into memory.

> Thanks guys for your helpful comments. I will be more  
> descriptive. I am an intern and my mentor wants me to  
> use the IO.pos to read the characters of the file  
> until the character reaches the “\>” symbol.

What problems is that giving you? You can create a loop, read the  
character at pos(i), then increment i, and do what Jesús Gabriel y Galán  
suggested. You can use three buffers since there will always be three  
lines. The additional housekeeping of chomp()'ing some lines and not  
others is straight forward: you leave buffer1 alone, and you chomp()  
buffer2 once, and you chomp buffer3 twice (or use sub() on both with the  
same regex(see my previous post)).

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 8:29pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/11 "2011-04-28T20:29:04Z")

</div>

7stud – wrote in post #995596:

> 7stud – wrote in post #995589:
> 
> > Cee J., are you reading the file in binary mode or  
> > text mode?
> 
> If you don’t know, then show us the line in your code where you open the  
> file.

f = File.open(“test.fasta”, “r”)

Where test.fasta contains the entries i posted earlier..

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 28, 2011, 8:27pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/12 "2011-04-28T20:27:34Z")

</div>

7stud – wrote in post #995581:

> You still have not told us what you are supposed to do with the stuff  
> you read in?? You can read a file line by line and print out each line  
> as you go and the maximum amount of memory used will be one line’s  
> worth. However, if you are supposed to store all the lines in an  
> array, then you will read the whole file into memory.
> 
> > Thanks guys for your helpful comments. I will be more  
> > descriptive. I am an intern and my mentor wants me to  
> > use the IO.pos to read the characters of the file  
> > until the character reaches the “\>” symbol.

I am extracting text from each entry I read in, something I have figured  
out already. I want to read the file line by line and just store each  
entry into a buffer when it reaches the “\>” symbol. THen extract  
specific info from it later. The entry lengths all vary as there long  
and short lengths. File is in text mode.

> What problems is that giving you? You can create a loop, read the  
> character at pos(i), then increment i, and do what Jesús Gabriel y Galán  
> suggested.

Could you show me a simple example or refer me to a link?

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 3:05am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/13 "2011-04-29T03:05:27Z")

</div>

> 1. Read one entry, do something to the entry, then discard it and read  
> in the next entry.

This is what I want to do. Read one entry, extract information from it,  
then read next entry. He says using an array will take up a lot of  
memory so he said use a buffer.

> But, you will end up reading every entry twice, which  
> is stupid. The easiest way to read in the file and prepare each entry  
> is to set the input separator to “\n\n”, then use each() to read in a  
> paragraph, then use split(“\n”) to split each entry into lines, then add  
> back a \n to the first line.
> 
> Also, are you aware that this:
> 
> > gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA\n  
> > GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG ← no “\n”  
> > CGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG ← no “\n”
> 
> is equivalent to:
> 
> > gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA

GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATGCGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG

Yes I am aware of that - I just put “no \n” for emphasis. Regarding the  
pos(), I think he said to use it as a guide to help with the detection  
of each “\>” . Thanks for being patient and helping out.

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 1:31am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/14 "2011-04-29T01:31:21Z")

</div>

Cee J. wrote in post #995597:

> my mentor wants me to use the IO.pos to read the  
> characters of the file until the character reaches the “\>” symbol.

IO.pos() does not read in data, so you are going to have to ask your  
mentor what he means. You should also ask your mentor if this is a  
lesson in how not to do things. If he doesn’t reply in the affirmative,  
then you should find a new mentor.

> I am extracting text from each entry I read in, something I have figured  
> out already. I want to read the file line by line and just store each  
> entry into a buffer when it reaches the “\>” symbol. THen extract  
> specific info from it later.

You told us you were not supposed to read the whole file into memory.  
If you store every line in an array, then you will have read the whole  
file into memory. Once again, you are not being clear on what you want  
to do with the data. You need to tell us which of the following you  
want to do:

1. Store every entry in an array, so that after reading the whole file  
all the entries will be in the array, in order for you to be able to  
“extract specific info from it later”.

2. Read one entry, do something to the entry, then discard it and read  
in the next entry.

> The entry lengths all vary as there long  
> and short lengths. File is in text mode.

Ok.

> > What problems is that giving you? You can create a loop, read the  
> > character at pos(i), then increment i, and do what Jesús Gabriel y Galán  
> > suggested.
> 
> Could you show me a simple example or refer me to a link?

Not really. If you have to mix pos() into your code by any tortuous  
means necessary, you could use each\_byte to read the file char by char  
(that  
assumes your file contains all ascii characters), and when you find a  
‘\>’, you could seek() back to the start of the file, and use  
IO.sysread() to read this number of characters:

old\_pos = 0  
number\_of\_chars\_to\_read = pos() - old\_pos

Then do something like:

old\_pos = pos()

But, you will end up reading every entry twice, which  
is stupid. The easiest way to read in the file and prepare each entry  
is to set the input separator to “\n\n”, then use each() to read in a  
paragraph, then use split(“\n”) to split each entry into lines, then add  
back a \n to the first line.

Also, are you aware that this:

> gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA\n  
> GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG ← no “\n”  
> CGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG ← no “\n”

is equivalent to:

> gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA  
> GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATGCGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 5:35am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/15 "2011-04-29T05:35:15Z")

</div>

hi Cee -

copying the text you posted above into the file “0text.txt” and  
running this:

f = “0text.txt”  
file = File.open(f)  
buffer = []  
bufferindex = 0

file.each\_line(sep="\>"){|line|  
buffer[bufferindex] = line.chomp  
bufferindex += 1  
}

p buffer[0]  
p buffer[1]  
p buffer[2]  
p buffer[3]

i get this as output:

#=\> “\>”  
#=\> “gi|329295464|ref|NM\_2005745.3Acc1| Def1 zgc:65895 (zgc:65895),  
mRNA\n\nAGCTCGGGGGCTCTAGCGATTTAAGGAGCGATGCGATCGAGCTGACCGTCGCG\n\n\n\n\>”  
#=\> “gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895),  
mRNA\n\nGTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG\n\nCGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG\n\n\n\n\>”  
#=\> “gi|3542945647|ref|NM\_7453343.5Acc3| Def3 zgc:65895 (zgc:65895),  
mRNA\n\nCGTGCGGGGABCCGTACGTGCCGTGGGGGTTTAATAGCGCGCCATCTGAGCAG\n\nTTAGTCGCTGACGCATGCACG\n\n\n”

does this work for you? you could easily write ways to deal with,  
dump, and reset the buffers when they fill up. you can of course also  
clean up all the “\n”'s…

i agree with 7stud that using #.pos and #.gets seems like a long walk  
off a short pier. i’m pretty green myself, and there are probably  
better ways to iterate through the file, but #.each\_line(sep="\>") works  
just fine, and doesn’t eat up memory.

- j

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 4:58am UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/16 "2011-04-29T04:58:26Z")

</div>

If you don’t have to use pos(), then see my first post. At some point,  
you might ask him why he thinks that pos() would be of any help at all!

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 5:20pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/17 "2011-04-29T17:20:27Z")

</div>

7stud – wrote in post #995683:

> If you don’t have to use pos(), then see my first post. At some point,  
> you might ask him why he thinks that pos() would be of any help at all!

Thanks jake and 7stud for replying. I tried this in irb for your first  
post:

> > e = File.open(“test/test.fasta”).enum\_for(:each)  
> > =\> #Enumerable::Enumerator:0x1005777a8  
> > $/ = “\n\n”  
> > =\> “\n\n”

> Before doing the sub() on buffer2, you will have to check if it’s nil:

> if buffer2.nil?  
> #don’t do a sub()  
> else  
> #do the sub()  
> end

> > e.each\_slice(2) do |buf1, buf2|  
> > ?\> p buf1, buf2  
> > if buf2.nil?  
> > puts “Done”  
> > else  
> > ?\> buf2.sub(/\n+ \z/xms, “”)  
> > end  
> > end

Output:  
“\>gi|329299107|ref|NM\_2005745.3Acc1| Def1 zgc:65895 (zgc:65895),  
mRNA\nAGCTCGGGGGCTCTAGCGATTTAAGGAGCGATGCGATCGAGCTGACCGTCGCG\n\n”  
“\>gi|329299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895),  
mRNA\nGTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG\nCGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG\n\n”  
“\>gi|329299107|ref|NM\_7453343.5Acc3| Def3 zgc:65895 (zgc:65895),  
mRNA\nCGTGCGGGGABCCGTACGTGCCGTGGGGGTTTAATAGCGCGCCATCTGAGCAG\nTTAGTCGCTGACGCATGCACG\n”  
nil  
Done  
=\> nil

It still returns nil, am I doing what you suggested wrong?

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 7:49pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/18 "2011-04-29T19:49:58Z")

</div>

I suggest that people never use irb because it has too many quirks.

The first thing you need to realize is that ‘\>’ is  
not the separator you want to look for. That is the second bit of  
erroneous advice your mentor gave you. That’s because you don’t care  
what character marks the beginning of every entry, rather you care what  
character marks the end of every entry. The end of every entry in your  
file is marked by the string “\n\n”, so you should use that as your  
input line terminator. Remember, ruby uses “\n” for the input line  
separator by default, which means that when you read a file using  
IO#each, ruby reads lines–where the end of a line is marked by a  
newline. However, you can change the input line separator to the string  
“\n\n” (or any other string, e.g. “HELLO”) like this:

$/ = “\n\n”

Then when you read a file with each(), ruby will read up to and  
including the input line separator, and return that as a “line”.

Once you read in an entry, then you just need to do a little  
housekeeping and remove some of the “\n” characters.

require ‘stringio’

str =\<\<ENDOFSTRING

> gi|329295464|ref|NM\_2005745.3Acc1| Def1 zgc:65895 (zgc:65895), mRNA  
> AGCTCGGGGGCTCTAGCGATTTAAGGAGCGATGCGATCGAGCTGACCGTCGCG

> gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA  
> GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG  
> CGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG

> gi|3542945647|ref|NM\_7453343.5Acc3| Def3 zgc:65895 (zgc:65895), mRNA  
> CGTGCGGGGABCCGTACGTGCCGTGGGGGTTTAATAGCGCGCCATCTGAGCAG  
> TTAGTCGCTGACGCATGCACG

ENDOFSTRING

input = StringIO.new(str) #Now input is just like a File

input.each(sep = “\n\n”) do |para|  
buffer = ‘’

lines = para.split(“\n”)  
buffer \<\< lines.shift \<\< “\n”

lines.each do |line|  
buffer \<\< line  
end

puts buffer #…or do whatever else you need to do to buffer  
puts “-” \* 20  
end

p $/

–output:–

> gi|329295464|ref|NM\_2005745.3Acc1| Def1 zgc:65895 (zgc:65895), mRNA  
> AGCTCGGGGGCTCTAGCGATTTAAGGAGCGATGCGATCGAGCTGACCGTCGCG

* * *

> gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA  
> GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATGCGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG

* * *

> gi|3542945647|ref|NM\_7453343.5Acc3| Def3 zgc:65895 (zgc:65895), mRNA  
> CGTGCGGGGABCCGTACGTGCCGTGGGGGTTTAATAGCGCGCCATCTGAGCAGTTAGTCGCTGACGCATGCACG

* * *

“\n”

Note that specifying the new input line separator as an argument to  
each() serves to restore the original input line separator once the  
block has finished–which is a good thing.

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 8:31pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/19 "2011-04-29T20:31:53Z")

</div>

7stud – wrote in post #995821:

> I suggest that people never use irb because it has too many quirks.
> 
> The first thing you need to realize is that ‘\>’ is  
> not the separator you want to look for. That is the second bit of  
> erroneous advice your mentor gave you. That’s because you don’t care  
> what character marks the beginning of every entry, rather you care what  
> character marks the end of every entry. The end of every entry in your  
> file is marked by the string “\n\n”, so you should use that as your  
> input line terminator. Remember, ruby uses “\n” for the input line  
> separator by default, which means that when you read a file using  
> IO#each, ruby reads lines–where the end of a line is marked by a  
> newline.

I understand the logic, it makes sense. What if the file looked like  
this, where there is one newline seperating the entries? :

> gi|329295464|ref|NM\_2005745.3Acc1| Def1 zgc:65895 (zgc:65895), mRNA  
> AGCTCGGGGGCTCTAGCGATTTAAGGAGCGATGCGATCGAGCTGACCGTCGCG  
> gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895), mRNA  
> GTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATG  
> CGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG  
> gi|3542945647|ref|NM\_7453343.5Acc3| Def3 zgc:65895 (zgc:65895), mRNA  
> CGTGCGGGGABCCGTACGTGCCGTGGGGGTTTAATAGCGCGCCATCTGAGCAG  
> TTAGTCGCTGACGCATGCACG

Would an if-else(regarding"\n" and “\n\n”) do the trick? I wanted to  
write my code to where it would handle both scenarios. Or maybe:

case  
when “\n\n”  
`
when “\n”

end`

`
something to that extent? Suggestions?`

---

<div class="post-metadata">

**Author:** ![playballa23](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/b487fb/32.png) [@playballa23](https://www.ruby-forum.com/u/playballa23)\
**Post date:** [April 29, 2011, 10:38pm UTC](https://www.ruby-forum.com/t/file-position-and-buffers/206316/20 "2011-04-29T22:38:55Z")

</div>

hi Cee -

hmm, i’m getting a bit confused as to what exactly you’re trying to do

- but if you want to load all this stuff into a buffer without the  
newlines, and regardless of how many newlines you have between each  
entry (assuming that an “entry” is something that starts with “\>”) - i  
don’t see why this wouldn’t work:

f = “0text.txt”  
file = File.open(f)  
buffer = []  
bufferindex = 0

file.each(sep = “\>”){|line|  
buffer[bufferindex] = line  
bufferindex += 1  
}

## here you would do something more interesting

buffer.collect{|line|  
line = line.delete("\n")  
p “\>#{line}”  
}

which will return…  
“\>\>”  
“\>gi|329295464|ref|NM\_2005745.3Acc1| Def1 zgc:65895 (zgc:65895),  
mRNAAGCTCGGGGGCTCTAGCGATTTAAGGAGCGATGCGATCGAGCTGACCGTCGCG\>”  
“\>gi|456299107|ref|NM\_2342343.3Acc2| Def2 zgc:65895 (zgc:65895),  
mRNAGTCGCTGGGTCGAAAAGTGGTGCTATATCGCGGCTCGCGTCGATGTCGCGATGCGTGCGCGCGAGAGCGCGCTATGATGAAAGGATGAGAGAG\>”  
“\>gi|3542945647|ref|NM\_7453343.5Acc3| Def3 zgc:65895 (zgc:65895),  
mRNACGTGCGGGGABCCGTACGTGCCGTGGGGGTTTAATAGCGCGCCATCTGAGCAGTTAGTCGCTGACGCATGCACG”

…whether you have 0 or 100,000 newlines between each entry. is this  
not what you’re looking for?

-j

[Next page](https://www.ruby-forum.com/t/file-position-and-buffers/206316.md?page=2)
