Random Access using IO#pos in code blocks


#1

Hello everyone,
I’m 20 days new to Ruby, please forgive if I make any mistakes. I’m
on a project where I’m indexing certain words in a text document. So
I’m also storing the file position where the word occurs. But the
Problem is:
The IO#pos points to the end of the file all the while… Below is
the code I’m working on:

File.open(file_name) do |f|

f.readlines("\r\n\r\n").each do |para|

para.scan(/\b\w+\b/).each do |word|

 word = word.downcase.stem
 if (!stoplist.include? word) && (!word.empty?) #excludes empty

and frequent words

     unless freq.has_key?(word)
       freq[word] = [1,f.pos,file_name] # freq is a hash, that

stores an array containing index, position of word (THE PROBLEM)…
else
freq[word].to_a[0] += 1
freq[word].to_a<< f.pos << file_name
end

     unless wfreq.has_key?(word)
      wfreq[word] = [1,f.pos,file_name]
    else
      wfreq[word].to_a[0] += 1
      wfreq[word].to_a<< f.pos << file_name
    end

 end

end
end

File.open(file_name+".yaml",“w”){|f| YAML.dump(freq,f)}

Also it would be great if someone told me the replacement for the
deprecated ‘to_a’ method used above :slight_smile:

Any help is greatly appreciated


#2

On 28.04.2009 22:32, Arun K. wrote:

f.readlines("\r\n\r\n").each do |para|
The reason is in the line above.

      freq[word].to_a[0] += 1
 end

end
end

File.open(file_name+".yaml",“w”){|f| YAML.dump(freq,f)}

Also it would be great if someone told me the replacement for the
deprecated ‘to_a’ method used above :slight_smile:

Why do you convert an Array into an Array?

Kind regards

robert


#3

Arun K. wrote:

     unless freq.has_key?(word)
       freq[word] = [1,f.pos,file_name] # freq is a hash, that

stores an array containing index, position of word (THE PROBLEM)…
else
freq[word].to_a[0] += 1
freq[word].to_a<< f.pos << file_name
end

BTW, you can replace all that by:

freq[word] ||= [0]
freq[word][0] += 1
freq[word] << f.pos << file_name

As for the pos, since you’ve already slurped in the data you’ll need to
remember where you are within your buffer. Your outer loop could become
something like this:

para_pos = 0
f.readlines("\r\n\r\n").each do |para|

para_pos += para.size + 4
end

Unfortunately, I don’t think string#scan will give you offsets into the
strings found.

In ruby 1.8 you can write this:

pos = 0
while md = /\b\w+\b/.match(para[pos…-1])
word = md[0]
puts “Match #{word} at #{para_pos+pos+md.begin(0)}”
pos += md.end(0)

end

In ruby 1.9 (but not 1.8.6/1.8.7), Regexp.match takes a start pos, so
you could optimise it to this:

pos = 0
while md = /\b\w+\b/.match(para, pos)
word = md[0]
puts “Match #{word} at #{para_pos+md.begin(0)}”
pos = md.end(0)

end

However in ruby 1.9 the offsets used will be in terms of number of
characters, not number of bytes. It would be up to you to convert this
back into byte offsets into the file, if that’s what you’re after.


#4

Thank you so much for the kind responses! I’m pleased to be part of such
a
kind community :slight_smile:

On Wed, Apr 29, 2009 at 4:32 PM, Robert K.


#5

2009/4/29 Brian C. removed_email_address@domain.invalid:



end

String#scan is likely faster than manually matching portions with
#match. In both versions of Ruby you can do this to get the
/character/ offset:

irb(main):001:0> s=%{foo bar baz}
=> “foo bar baz”
irb(main):002:0> s.scan(/\w+/) { p $`.length }
0
4
8
=> “foo bar baz”

However in ruby 1.9 the offsets used will be in terms of number of
characters, not number of bytes. It would be up to you to convert this
back into byte offsets into the file, if that’s what you’re after.

This is an important point to remember!

Kind regards

robert


#6

freq[word] ||= [0]
freq[word][0] += 1
freq[word] << f.pos << file_name

I’ve tried doing it but since ‘freq’ is a hash it gives the following
error:

preprocessor.rb:32:in calc_frequency_word_list': undefined method[]=’
for 0:Fixnum (NoMethodError)
from copy of preprocessor.rb:25:in scan' from copy of preprocessor.rb:25:incalc_frequency_word_list’
from copy of preprocessor.rb:23:in each' from copy of preprocessor.rb:23:incalc_frequency_word_list’
from copy of preprocessor.rb:61


#7

Arun K. wrote:

freq[word] ||= [0]
freq[word][0] += 1
freq[word] << f.pos << file_name

I’ve tried doing it but since ‘freq’ is a hash it gives the following
error:

Show your actual code. The following code works just fine:

freq = {}
%w{foo bar baz bar}.each do |word|
freq[word] ||= [0]
freq[word][0] += 1
freq[word] << “pos” << “name”
end
puts freq.inspect

The error suggests that you have initialized freq[word] to 0, not to
[0].

Or perhaps you set freq = Hash.new(0), which is wrong in this case,
because the default element needs to be [0] not 0.

An alternative is to auto-initialize each hash element like this:

freq = Hash.new { |h,k| h[k] = [0] }
%w{foo bar baz bar}.each do |word|
freq[word][0] += 1
freq[word] << “pos” << “name”
end
puts freq.inspect


#8

Thank you so much for all the responses :slight_smile:


#9

Robert K. wrote:

String#scan is likely faster than manually matching portions with
#match. In both versions of Ruby you can do this to get the
/character/ offset:

irb(main):001:0> s=%{foo bar baz}
=> “foo bar baz”
irb(main):002:0> s.scan(/\w+/) { p $`.length }
0
4
8
=> “foo bar baz”

Well, my guess is that would be less efficient for large paragraphs,
since $` forces allocation of a new string containing all the text from
the start to the current point. But that reminds me, there is a global
variable containing a MatchData object: $~

So you can write:

irb(main):001:0> s=%{foo bar baz}
=> “foo bar baz”
irb(main):002:0> s.scan(/\w+/) { p $~.begin(0) }
0
4
8
=> “foo bar baz”

Regards,

Brian.


#10

2009/4/30 Brian C. removed_email_address@domain.invalid:

8
=> “foo bar baz”

Well, my guess is that would be less efficient for large paragraphs,
since $` forces allocation of a new string containing all the text from
the start to the current point.

Last time I checked the actual string buffer was shared so the
overhead is just a single instance. I do have to admit though that I
do not know when the object is allocated (i.e. at time of match or
when referencing $`).

8
=> “foo bar baz”

Also a good variant! (Btw, MatchData might be even more heavyweight
than a sub string.)

Kind regards

robert


#11

Robert K. wrote:

I do have to admit though that I
do not know when the object is allocated (i.e. at time of match or
when referencing $`).

Experiment suggests the MatchData is created immediately on the match,
and the string is instantiated lazily from that. This makes sense; it
would be very inefficient to allocate strings for $`, $1, $2, $3, … $’
when maybe none of them will be used. But the MatchData object has the
original string plus all the offsets.

def count(klass)
c = 0
ObjectSpace.each_object(klass) { c += 1 }
c
end

str = " foo bar baz "

c1 = [count(MatchData), count(String)]

str =~ /(\w+)/

c2 = [count(MatchData), count(String)]

x = $~

c3 = [count(MatchData), count(String)]

y = $`

c4 = [count(MatchData), count(String)]

puts [c1,c2,c3,c4].inspect

[[0, 188], [1, 188], [1, 188], [1, 189]]


#12

2009/4/29 Arun K. removed_email_address@domain.invalid:

freq[word] << “pos” << “name”
end
puts freq.inspect

This is a typical case where I would introduce a separate class or
even multiple classes because it makes life so much more readable.

WordPositon = Struct.new :file, :pos

WordStats = Struct.new :word, :positions do
def count; positions.size; end
end

freq = Hash.new {|h,word| h[word.freeze] = WordStat.new(word, [])}

freq[word].positions << WordPosition.new(file_name, pos)

Then you can do

freq.sort_by {|w,stat| stat.count}

Kind regards

robert


#13

2009/4/30 Brian C. removed_email_address@domain.invalid:

Robert K. wrote:

I do have to admit though that I
do not know when the object is allocated (i.e. at time of match or
when referencing $`).

Experiment suggests the MatchData is created immediately on the match,
and the string is instantiated lazily from that. This makes sense; it
would be very inefficient to allocate strings for $`, $1, $2, $3, … $’
when maybe none of them will be used. But the MatchData object has the
original string plus all the offsets.

Ah, good to know! Thanks for the experimenting!

“Tune in next week when you’ll hear Dr. Brian say: what’s this fuse
for?”
:wink:

Kind regards

robert