Hello everyone,
I’m 20 days new to Ruby, please forgive if I make any mistakes. I’m
on a project where I’m indexing certain words in a text document. So
I’m also storing the file position where the word occurs. But the
Problem is:
The IO#pos points to the end of the file all the while… Below is
the code I’m working on:
File.open(file_name) do |f|
f.readlines("\r\n\r\n").each do |para|
para.scan(/\b\w+\b/).each do |word|
word = word.downcase.stem
if (!stoplist.include? word) && (!word.empty?) #excludes empty
and frequent words
unless freq.has_key?(word)
freq[word] = [1,f.pos,file_name] # freq is a hash, that
stores an array containing index, position of word (THE PROBLEM)…
else
freq[word].to_a[0] += 1
freq[word].to_a<< f.pos << file_name
end
unless wfreq.has_key?(word)
wfreq[word] = [1,f.pos,file_name]
else
wfreq[word].to_a[0] += 1
wfreq[word].to_a<< f.pos << file_name
end
end
end
end
File.open(file_name+".yaml",“w”){|f| YAML.dump(freq,f)}
Also it would be great if someone told me the replacement for the
deprecated ‘to_a’ method used above 
Any help is greatly appreciated
On 28.04.2009 22:32, Arun K. wrote:
f.readlines("\r\n\r\n").each do |para|
The reason is in the line above.
freq[word].to_a[0] += 1
end
end
end
File.open(file_name+".yaml",“w”){|f| YAML.dump(freq,f)}
Also it would be great if someone told me the replacement for the
deprecated ‘to_a’ method used above 
Why do you convert an Array into an Array?
Kind regards
robert
Arun K. wrote:
unless freq.has_key?(word)
freq[word] = [1,f.pos,file_name] # freq is a hash, that
stores an array containing index, position of word (THE PROBLEM)…
else
freq[word].to_a[0] += 1
freq[word].to_a<< f.pos << file_name
end
BTW, you can replace all that by:
freq[word] ||= [0]
freq[word][0] += 1
freq[word] << f.pos << file_name
As for the pos, since you’ve already slurped in the data you’ll need to
remember where you are within your buffer. Your outer loop could become
something like this:
para_pos = 0
f.readlines("\r\n\r\n").each do |para|
…
para_pos += para.size + 4
end
Unfortunately, I don’t think string#scan will give you offsets into the
strings found.
In ruby 1.8 you can write this:
pos = 0
while md = /\b\w+\b/.match(para[pos…-1])
word = md[0]
puts “Match #{word} at #{para_pos+pos+md.begin(0)}”
pos += md.end(0)
…
end
In ruby 1.9 (but not 1.8.6/1.8.7), Regexp.match takes a start pos, so
you could optimise it to this:
pos = 0
while md = /\b\w+\b/.match(para, pos)
word = md[0]
puts “Match #{word} at #{para_pos+md.begin(0)}”
pos = md.end(0)
…
end
However in ruby 1.9 the offsets used will be in terms of number of
characters, not number of bytes. It would be up to you to convert this
back into byte offsets into the file, if that’s what you’re after.
Thank you so much for the kind responses! I’m pleased to be part of such
a
kind community
On Wed, Apr 29, 2009 at 4:32 PM, Robert K.
2009/4/29 Brian C. [email protected]:
…
…
end
String#scan is likely faster than manually matching portions with
#match. In both versions of Ruby you can do this to get the
/character/ offset:
irb(main):001:0> s=%{foo bar baz}
=> “foo bar baz”
irb(main):002:0> s.scan(/\w+/) { p $`.length }
0
4
8
=> “foo bar baz”
However in ruby 1.9 the offsets used will be in terms of number of
characters, not number of bytes. It would be up to you to convert this
back into byte offsets into the file, if that’s what you’re after.
This is an important point to remember!
Kind regards
robert
freq[word] ||= [0]
freq[word][0] += 1
freq[word] << f.pos << file_name
I’ve tried doing it but since ‘freq’ is a hash it gives the following
error:
preprocessor.rb:32:in calc_frequency_word_list': undefined method[]=’
for 0:Fixnum (NoMethodError)
from copy of preprocessor.rb:25:in scan' from copy of preprocessor.rb:25:incalc_frequency_word_list’
from copy of preprocessor.rb:23:in each' from copy of preprocessor.rb:23:incalc_frequency_word_list’
from copy of preprocessor.rb:61
Arun K. wrote:
freq[word] ||= [0]
freq[word][0] += 1
freq[word] << f.pos << file_name
I’ve tried doing it but since ‘freq’ is a hash it gives the following
error:
Show your actual code. The following code works just fine:
freq = {}
%w{foo bar baz bar}.each do |word|
freq[word] ||= [0]
freq[word][0] += 1
freq[word] << “pos” << “name”
end
puts freq.inspect
The error suggests that you have initialized freq[word] to 0, not to
[0].
Or perhaps you set freq = Hash.new(0), which is wrong in this case,
because the default element needs to be [0] not 0.
An alternative is to auto-initialize each hash element like this:
freq = Hash.new { |h,k| h[k] = [0] }
%w{foo bar baz bar}.each do |word|
freq[word][0] += 1
freq[word] << “pos” << “name”
end
puts freq.inspect
Thank you so much for all the responses
Robert K. wrote:
String#scan is likely faster than manually matching portions with
#match. In both versions of Ruby you can do this to get the
/character/ offset:
irb(main):001:0> s=%{foo bar baz}
=> “foo bar baz”
irb(main):002:0> s.scan(/\w+/) { p $`.length }
0
4
8
=> “foo bar baz”
Well, my guess is that would be less efficient for large paragraphs,
since $` forces allocation of a new string containing all the text from
the start to the current point. But that reminds me, there is a global
variable containing a MatchData object: $~
So you can write:
irb(main):001:0> s=%{foo bar baz}
=> “foo bar baz”
irb(main):002:0> s.scan(/\w+/) { p $~.begin(0) }
0
4
8
=> “foo bar baz”
Regards,
Brian.
2009/4/30 Brian C. [email protected]:
8
=> “foo bar baz”
Well, my guess is that would be less efficient for large paragraphs,
since $` forces allocation of a new string containing all the text from
the start to the current point.
Last time I checked the actual string buffer was shared so the
overhead is just a single instance. I do have to admit though that I
do not know when the object is allocated (i.e. at time of match or
when referencing $`).
8
=> “foo bar baz”
Also a good variant! (Btw, MatchData might be even more heavyweight
than a sub string.)
Kind regards
robert
Robert K. wrote:
I do have to admit though that I
do not know when the object is allocated (i.e. at time of match or
when referencing $`).
Experiment suggests the MatchData is created immediately on the match,
and the string is instantiated lazily from that. This makes sense; it
would be very inefficient to allocate strings for $`, $1, $2, $3, … $’
when maybe none of them will be used. But the MatchData object has the
original string plus all the offsets.
def count(klass)
c = 0
ObjectSpace.each_object(klass) { c += 1 }
c
end
str = " foo bar baz "
c1 = [count(MatchData), count(String)]
str =~ /(\w+)/
c2 = [count(MatchData), count(String)]
x = $~
c3 = [count(MatchData), count(String)]
y = $`
c4 = [count(MatchData), count(String)]
puts [c1,c2,c3,c4].inspect
[[0, 188], [1, 188], [1, 188], [1, 189]]
2009/4/29 Arun K. [email protected]:
freq[word] << “pos” << “name”
end
puts freq.inspect
This is a typical case where I would introduce a separate class or
even multiple classes because it makes life so much more readable.
WordPositon = Struct.new :file, :pos
WordStats = Struct.new :word, :positions do
def count; positions.size; end
end
freq = Hash.new {|h,word| h[word.freeze] = WordStat.new(word, [])}
…
freq[word].positions << WordPosition.new(file_name, pos)
…
Then you can do
freq.sort_by {|w,stat| stat.count}
Kind regards
robert
2009/4/30 Brian C. [email protected]:
Robert K. wrote:
I do have to admit though that I
do not know when the object is allocated (i.e. at time of match or
when referencing $`).
Experiment suggests the MatchData is created immediately on the match,
and the string is instantiated lazily from that. This makes sense; it
would be very inefficient to allocate strings for $`, $1, $2, $3, … $’
when maybe none of them will be used. But the MatchData object has the
original string plus all the offsets.
Ah, good to know! Thanks for the experimenting!
“Tune in next week when you’ll hear Dr. Brian say: what’s this fuse
for?”
Kind regards
robert
