# Finding duplicated lines in folder for refactoring purposes

**URL:** <https://www.ruby-forum.com/t/finding-duplicated-lines-in-folder-for-refactoring-purposes/139937>\
**Category:** Ruby\
**Created:** [June 9, 2008, 11:55am UTC](https://www.ruby-forum.com/t/finding-duplicated-lines-in-folder-for-refactoring-purposes/139937 "2008-06-09T11:55:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [June 9, 2008, 11:55am UTC](https://www.ruby-forum.com/t/finding-duplicated-lines-in-folder-for-refactoring-purposes/139937/1 "2008-06-09T11:55:55Z")

</div>

Hello.

I am wondering if anyone knows of a good trick to identify duplicated  
lines in a directory for refactoring purposes.

The idea is this: If I can get a listing of all lines, by file in a  
directory (recursively) then refactoring code could be focused  
eliminating “repeated” lines.

The solution would involve

- recursively searching all files in a given path
- displaying a list of repeated lines (ignoring case and whitespace)
- grouping results by path/filename combination
- sorting results by line repeat count
- ideally only displaying lines that are repeated at lease once…

I’m thinking this should exist either via a application, ruby script,  
or shell script.

Does anyone know if any ideas or solutions that are remotely close to  
this?

-Shannon

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [June 9, 2008, 1:13pm UTC](https://www.ruby-forum.com/t/finding-duplicated-lines-in-folder-for-refactoring-purposes/139937/2 "2008-06-09T13:13:00Z")

</div>

Try simian [[http://www.redhillconsulting.com.au/products/simian/](http://www.redhillconsulting.com.au/products/simian/) ].

Craig

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [June 9, 2008, 2:31pm UTC](https://www.ruby-forum.com/t/finding-duplicated-lines-in-folder-for-refactoring-purposes/139937/3 "2008-06-09T14:31:32Z")

</div>

2008/6/9 removed\_email\_address@domain.invalid [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> I am wondering if anyone knows of a good trick to identify duplicated  
> lines in a directory for refactoring purposes.

Do you mean “duplicated lines in files in a directory tree”? Ther  
term “duplicated lines in a directory” does not make much sense to me  
as a directory does not have “lines”.

> I’m thinking this should exist either via a application, ruby script,  
> or shell script.
> 
> Does anyone know if any ideas or solutions that are remotely close to  
> this?

If you have enough memory or few enough files you could do

# untested

require ‘find’  
require ‘set’

def normalize(line)  
l = line.strip  
l.gsub!(/\s+/, ’ ')  
l.downcase!  
l  
end

duplicates = Hash.new {|h,k| h[k] = Set.new}

Find.find dir do |file|  
File.foreach file do |line|  
duplicates[normalize(line)] \<\< file  
end  
end

duplicates.each do |line, files|  
puts line, files.sort.join(‘,’) if files.size \> 1  
end

Cheers

robert