# "Fuzzy" matching of lines between CVS mailing lists

**URL:** <https://www.ruby-forum.com/t/fuzzy-matching-of-lines-between-cvs-mailing-lists/127487>\
**Category:** Ruby\
**Created:** [January 29, 2008, 7:03pm UTC](https://www.ruby-forum.com/t/fuzzy-matching-of-lines-between-cvs-mailing-lists/127487 "2008-01-29T19:03:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![philippeqc](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e9bcb4/32.png) [@philippeqc](https://www.ruby-forum.com/u/philippeqc)\
**Post date:** [January 29, 2008, 7:03pm UTC](https://www.ruby-forum.com/t/fuzzy-matching-of-lines-between-cvs-mailing-lists/127487/1 "2008-01-29T19:03:49Z")

</div>

People,

I am not sure if this is an appropriate place to ask this sort of  
question - there are probably dozens of different solutions with greatly  
varying amounts of time, effort and efficiency involved but since I like  
doing things in Ruby, I thought I would ask the gurus here:

I periodically receive new mailing lists in CVS format and I have to  
check for duplications of individual mailing addresses in the new list  
and the current list. The problem is, the there is no common format in  
the new lists because they come from different organisations - one list  
might have all data in capital letters, another might have last name and  
only initials, another might have last name and first name, another  
might have full state names and others a two character field - there are  
lots of variations. About the only thing that can relied on (ignoring  
case) is that the last name would be the same in both lists if there is  
a duplication. If I want to pattern match from the new list to the  
existing list, I have to be fairly flexible ie it is better to get false  
positives (because they can be quickly ignored by eyeballing) than false  
negatives (someone is mailed twice in the new merged list).

Suggestions? ideas? Should I just use the regular shell tools?

Thanks,

## Phil.

Philip R.

Pricom Pty Limited (ACN 003 252 275 ABN 91 003 252 275)  
GPO Box 3411  
Sydney NSW 2001  
Australia  
Fax: +61:(0)2-8221-9599  
E-mail: removed\_email\_address@domain.invalid

---

<div class="post-metadata">

**Author:** ![philippeqc](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e9bcb4/32.png) [@philippeqc](https://www.ruby-forum.com/u/philippeqc)\
**Post date:** [January 29, 2008, 8:35pm UTC](https://www.ruby-forum.com/t/fuzzy-matching-of-lines-between-cvs-mailing-lists/127487/2 "2008-01-29T20:35:54Z")

</div>

On 29.01.2008 19:03, Phil R. wrote:

> the new lists because they come from different organisations - one list  
> Suggestions? ideas? Should I just use the regular shell tools?  
>   
> Maybe a two step approach:

1. normalize data (e.g. rip off all whitespace, punctuation or just keep  
all characters and digits)

2. calculate something like the hamming distance between every two  
entries and flag those entries which have a distance less than a certain  
threshold

Downside is that step 2 takes O(n\*n)…

Kind regards

robert