# Can't remove duplicate

**URL:** <https://www.ruby-forum.com/t/cant-remove-duplicate/173852>\
**Category:** Ferret\
**Created:** [August 17, 2009, 3:43am UTC](https://www.ruby-forum.com/t/cant-remove-duplicate/173852 "2009-08-17T03:43:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![femto](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/50afbb/32.png) [@femto](https://www.ruby-forum.com/u/femto)\
**Post date:** [August 17, 2009, 3:43am UTC](https://www.ruby-forum.com/t/cant-remove-duplicate/173852/1 "2009-08-17T03:43:26Z")

</div>

Hello all, I can’t remove duplicate,I’m using ferret  
to index log file in order to monitor application activity,  
what I want to do is index data based on the uniqueness of  
[filename,line](actullay should be [host,filename,line],  
the code is following:

if !$indexer

```
  field_infos = Ferret::Index::FieldInfos.new(:index =>

```

:untokenized\_omit\_norms,  
:term\_vector =\> :no)  
field\_infos.add\_field(:content, :store =\> :yes, :index =\> :yes)

```
  $indexer = Ferret::I.new(:path => index_dir,
                           :field_infos => field_infos,
                           :key => [:filename, :line],
                           :max_buffered_docs=>100)

  #$indexer ||= Ferret::I.new(:path=>index_dir, :key => ['filename',

```

‘line’], :max\_buffered\_docs=\>100) #unique host,file\_name,line  
#$indexer.field\_infos.add\_field(:time,  
# #:default\_boost =\> 20,  
# :store =\> :yes,  
# :index =\> :untokenized,  
# :term\_vector =\> :no)  
end  
but the problem is, I will index a new datum even if the [filename,line]  
is same, even I change :key =\> [“filename”, “line”], it also doesn’t  
work,  
what’s the problem? Thanks.