# Direct access to similarity, idf, tf

**URL:** <https://www.ruby-forum.com/t/direct-access-to-similarity-idf-tf/174314>\
**Category:** Ferret\
**Created:** [August 24, 2009, 9:47pm UTC](https://www.ruby-forum.com/t/direct-access-to-similarity-idf-tf/174314 "2009-08-24T21:47:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rpacheco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e36b37/32.png) [@rpacheco](https://www.ruby-forum.com/u/rpacheco)\
**Post date:** [August 24, 2009, 9:47pm UTC](https://www.ruby-forum.com/t/direct-access-to-similarity-idf-tf/174314/1 "2009-08-24T21:47:47Z")

</div>

Hello,

I need access to similarity, tf, idf and some other content from the  
same level.

I’ve been studying the code for three days now and I’m almost sure that  
there’s no way to do that “directly” from ruby code.

As I _really need_ them, I was thinking about writing out some ruby code  
for the relevant ferret c code to get access to tf, idf and so on. Or  
maybe write a wrapper for lucene-java code, because it seams to be  
easier. But I do think I could help a bit ferret development writing the  
wrapper for ferret’s code.

So, I have some questions regarding these issues:

1. Is there an implemented way to get access to tf, idf, …? (so am I  
wrong?)

2. What do you think it’s easier: writing a wrapper for lucene-java or  
for ferret?

3. Will it be good for ferret to have this code I’m proposing written?

Thanks in advance

Ricardo Panaggio  
ps: sorry if my English is difficult to read =/

---

<div class="post-metadata">

**Author:** ![rpacheco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e36b37/32.png) [@rpacheco](https://www.ruby-forum.com/u/rpacheco)\
**Post date:** [September 23, 2009, 9:14pm UTC](https://www.ruby-forum.com/t/direct-access-to-similarity-idf-tf/174314/2 "2009-09-23T21:14:39Z")

</div>

I am very interested in this if you find a way to do it

Charles

---

<div class="post-metadata">

**Author:** ![rpacheco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e36b37/32.png) [@rpacheco](https://www.ruby-forum.com/u/rpacheco)\
**Post date:** [September 27, 2010, 9:40pm UTC](https://www.ruby-forum.com/t/direct-access-to-similarity-idf-tf/174314/3 "2010-09-27T21:40:45Z")

</div>

Her’s a monkey patch that should do the trick:

class Ferret::Index::IndexReader

TFIDF\_THRESH = 0.0

# return [doc\_id, [term, tfidf] ]

# doc\_id starts at 0

# tfidf drops values \> THRESH

# 

def each\_tfidf\_vec(field=:id, thresh=TFIDF\_THRESH, &block)

```
doc_freq = {} # [term] => doc_freq
terms(field).each { |term, df| doc_freq[term] = df }
num_terms = doc_freq.size

 (0...num_docs).each do |doc_id|
  tv_terms = term_vector(doc_id, field).terms
  tf_norm = tv_terms.size

  tfidf_vec = tv_terms.map do |tv_term|
    term = tv_term.text
    tf = tv_term.positions.size
    df = doc_freq[term]

    tfidf = (tf.to_f/tf_norm.to_f) * Math.log(num_docs.to_f/df.to_f)

    [term,tfidf] if tfidf > thresh
  end

  #remove nil values (tfidf < thresh)
  tfidf_vec.compact!

  yield doc_id, tfidf_vec
end

```

end

end