Forum: Nitro question for unix wizzards

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
7a4e995e378ef66de0ceaea5e1381ee1?d=identicon&s=25 George Moschovitis (Guest)
on 2008-02-04 17:11
(Received via mailing list)
Dear devs,

2 small unix related questions.

does anyone know about:

- a quick and easy way to remove duplicate lines from a text files?
- a quick and easy way to decide if two image (binary) files are the
same
image/picture?

thanks in advance for your help!

-g.
407f866e9a1b959de956ae560239adfe?d=identicon&s=25 Jonathan Buch (Guest)
on 2008-02-04 17:32
(Received via mailing list)
Hi,

> - a quick and easy way to remove duplicate lines from a text files?

cat file | sort | uniq > file
# should work (if the sorting doesn't matter in the file)

cat file | uniq > file
# works if the duplicate lines are next to each other

> - a quick and easy way to decide if two image (binary) files are the same
> image/picture?

findimagedupes << search for that in google, I'm using one of those
perl scripts.

> thanks in advance for your help!

Hope that helps,

Jo
A6d3a37b5badfdd2f47655aa0e46604e?d=identicon&s=25 Eivind Eklund (Guest)
on 2008-02-04 17:37
(Received via mailing list)
On Feb 4, 2008 5:09 PM, George Moschovitis
<george.moschovitis@gmail.com> wrote:
> Dear devs,
>
> 2 small unix related questions.
>
> does anyone know about:
>
> - a quick and easy way to remove duplicate lines from a text files?

uniq < filename (assuming that the lines are consequtive; otherwise,
you need to do a sort first, or keep the lines in a hash in either
Ruby or Perl)

> - a quick and easy way to decide if two image (binary) files are the same
> image/picture?

Do you mean if they are identical?  cmp should do it.  Another
approach is to get a checksum of the files (e.g. using "openssl rmd160
<filename>") and see if the checksum is the same.

Eivind.
91e1fb8bd265b7629491ab64c42f0906?d=identicon&s=25 Reid Thompson (Guest)
on 2008-02-04 17:58
(Received via mailing list)
On Mon, 2008-02-04 at 18:09 +0200, George Moschovitis wrote:
> Dear devs,
>
> 2 small unix related questions.
>
> does anyone know about:
>
> - a quick and easy way to remove duplicate lines from a text files?
from http://www.student.northpark.edu/pemente/sed/sed1line.txt

# delete duplicate, nonconsecutive lines from a file. Beware not to
# overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'

> - a quick and easy way to decide if two image (binary) files are the
> same image/picture?
diff will tell you if the files are different also

rthompso@raker ~ $ diff 10000_Galaxies,_HST_Ultra_Deep.png
Deathvalleysky_nps_big.png
Files 10000_Galaxies,_HST_Ultra_Deep.png and Deathvalleysky_nps_big.png
differ
rthompso@raker ~ $ cp 10000_Galaxies,_HST_Ultra_Deep.png junk.png
rthompso@raker ~ $ diff 10000_Galaxies,_HST_Ultra_Deep.png junk.png
7a4e995e378ef66de0ceaea5e1381ee1?d=identicon&s=25 George Moschovitis (Guest)
on 2008-02-05 09:51
(Received via mailing list)
Thanks for the help   everyone!
This topic is locked and can not be replied to.