Checking if two files are the same

Ruby
#1

I have a got a few folders which may have same files under different
names.
Is there any way I can find which these files are using ruby ?

The files with text (*.doc, *.txt …) should be pretty easy to check
but what about .pdf files, .exe files etc ?

I’m wondering if there is some sort of diff module that can do this.

#2

On Mon, Mar 7, 2011 at 11:48 AM, New C. [email protected] wrote:

I have a got a few folders which may have same files under different
names.
Is there any way I can find which these files are using ruby ?

The files with text (*.doc, *.txt …) should be pretty easy to check
but what about pdf files, exe files etc ?

I wondering if there is some sort of diff module that can do this.

There’s File.compare.

Depending on how many comparisons you’re going to do, it might be a
good idea to precompute checksums and compare the checksums.

#3

On 03/07/2011 06:09 AM, Xavier N. wrote:

Depending on how many comparisons you’re going to do, it might be a
good idea to precompute checksums and compare the checksums.

+1

#4

On Mon, Mar 7, 2011 at 11:09 AM, Xavier N. [email protected] wrote:

good idea to precompute checksums and compare the checksums.
I’m interested in any (Ruby) solutions (actual or ideas) for this, as
I have needed to do it in the past, and want to do something similar
in the very near future.

For comparing directories where the file names might have changed what
I’ve done in the past is to first match on file name, then for the
unmatching files in each directory see if there are any matches on
file size, and for those matches either make a direct File.compare (if
only two files match on a size) or compute checksums and use those to
exclude definitely unmatching files, and then use File.compare on what
(if anything) remains matching for that file size and checksum.

I assume something similar would work for finding duplicates in
general, not just comparing directories? (If there are likely to be
many matches on file size, then presumably one might as well compute
checksums for all files?)

#5

New C. wrote in post #985925:

I have a got a few folders which may have same files under different
names.
Is there any way I can find which these files are using ruby ?

Here is a little ruby script I use for finding and/or deleting duplicate
image and video files downloaded from my camera - it will work for any
sort of file.

#!/usr/bin/ruby -w
require ‘digest/sha1’
if ARGV[0] == “-d”
do_delete = true
ARGV.shift
end

seen = {}
dirs = ARGV.empty? ? ["#{ENV[“HOME”]}/Pictures"] : ARGV

dirs.each do |dir|
Dir["#{dir}/**/*"].sort.each do |fn|
next if File.directory?(fn)
hash = Digest::SHA1.file(fn).hexdigest
if seen[hash]
puts “#{fn} is dupe of #{seen[hash]}”
if do_delete
File.delete(fn)
puts “DELETED”
end
else
seen[hash] = fn
end
end
end

#6

On Tue, Mar 8, 2011 at 9:48 AM, Brian C. [email protected]
wrote:

require ‘digest/sha1’
next if File.directory?(fn)
end
end

For this idiom Hash#fetch can be used nicely:

irb(main):008:0> h={};
10.times {|i|
puts i
h.fetch(i % 3) {|x| printf “first %p\n”, i; h[x]=true; nil} and
printf “duplicate %p\n”, i}
0
first 0
1
first 1
2
first 2
3
duplicate 3
4
duplicate 4
5
duplicate 5
6
duplicate 6
7
duplicate 7
8
duplicate 8
9
duplicate 9
=> 10
irb(main):009:0>

… for arbitrary values of “nice”. :wink:

Kind regards

robert

#7

It looks pretty obfuscated to my eyes, but each to his own.

dirs.each do |dir|
Dir["#{dir}/**/*"].sort.each do |fn|
next if File.directory?(fn)
hash = Digest::SHA1.file(fn).hexdigest
if seen.fetch(hash) { seen[hash]=fn; false }
puts “#{fn} is dupe of #{seen[hash]}”
if do_delete
File.delete(fn)
puts “DELETED”
end
end
end
end

#8

On Mon, Mar 7, 2011 at 6:38 PM, Colin B. [email protected]
wrote:

(if anything) remains matching for that file size and checksum.
I have played with this as an exercise. The idea is to filter
candidates iteratively applying different criteria, from cheap to
expensive, until you arrive at the solution.

It is just a proof of concept in pseudocode, I wrote it off the top of
my head, it does not even run:

https://gist.github.com/859046

The code above assumes a generic scenario with m-n possible
duplicates, if a particular situation has details that can speed up
the process they should be taken into account of course.