Awk to ruby 1.9

[atma]

Hello,

I’m a total newbe. I’m trying to figure out how to grab a specific Xth
character from a .log and put it into an array.

So far, I’ve got to this code snippet:

---

encoding: utf-8

class ReadIPs
def initialize
@read_ips = []
end

def get_ips(filename)
ips = [] # make a list of ip addressses
File.foreach(filename) do |line|
puts “#{line}” if line =~ /Ban/
end
end
end

ipadds = ReadIPs.new
a = ipadds.get_ips(“fail2ban.log”)

The output though parses lines the following:

2009-11-19 00:31:29,928 fail2ban.actions: WARNING [ssh-ipfw] Ban
203.169.139.171

Now, I’d like to isolate the IP and put it into an Array.

I would like to use the output from the cli: “$ grep Ban
fail2ban.log|awk -F “Ban” ‘{print $2}’”

60.12.200.xx
193.193.221.xx
85.72.xx.xx
124.207.xx.xx

regards

Panagiotis (atmosx) Atmatzidis

email: [email protected]
URL: http://www.convalesco.org
GnuPG ID: 0xFC4E8BB4
gpg --keyserver x-hkp://pgp.mit.edu --recv-keys 0xFC4E8BB4

[atma]

On Friday 04 December 2009 02:09:35 pm Panagiotis A. wrote:

def initialize

Now, I’d like to isolate the IP and put it into an Array.

It looks as though you’ve gotten that started…

def initialize
@read_ips = []
end

But you never seem to use this array.

def get_ips(filename)
ips = [] # make a list of ip addressses
File.foreach(filename) do |line|
puts “#{line}” if line =~ /Ban/
end
end

You never seem to use this array, either. You could do something like
this:

def get_ips(filename)
File.open filename do |file|
file.each_line.select{|line| line =~ /Ban/}
end
end

That will at least return an array of lines containing Ban. But you
already
know what each line looks like. Here’s another way, that looks kind of
like
what you started:

def get_ips(filename)
ips = []
File.open filename do |file|
file.each_line do |line|
if line =~ /Ban\s+(\S+)$/
ips << $1.chomp
end
end
end
ips
end

I would like to use the output from the cli: “$ grep Ban fail2ban.log|awk
-F “Ban” ‘{print $2}’”

You could probably figure out a way to do this in Ruby, and it probably
wouldn’t be too much worse than the awk version, but if you really want
to do
it that way, try backticks. If this is the output you’re expecting:

60.12.200.xx
193.193.221.xx
85.72.xx.xx
124.207.xx.xx

you could probably get away with:

ips = grep Ban fail2ban.log | awk -F "BAN" '{print $2}'.each_line.to_a

By the way, all of this is giving you ips as strings. If you’re wanting
to
actually manipulate the ips at all, I’d suggest looking at IPAddr. (You
probably don’t, but just in case…)

[atma]

On 04 Δεκ 2009, at 10:27 μ.μ., David M. wrote:

class ReadIPs
end

Now, I’d like to isolate the IP and put it into an Array.

It looks as though you’ve gotten that started…

def initialize
@read_ips = []
end

But you never seem to use this array.

Oh, you mean the “<<” flags. Right, I saw fractions of code but I was
not sure what “<<” meant.

def get_ips(filename)
ips = []
File.open filename do |file|
file.each_line do |line|
if line =~ /Ban\s+(\S+)$/
ips << $1.chomp
end
end
end
ips
end

Thanks for code snippet. I see now how should ips be used in order to
create an array.

85.72.xx.xx
124.207.xx.xx

you could probably get away with:

ips = grep Ban fail2ban.log | awk -F "BAN" '{print $2}'.each_line.to_a

By the way, all of this is giving you ips as strings. If you’re wanting to
actually manipulate the ips at all, I’d suggest looking at IPAddr. (You
probably don’t, but just in case…)

I have found this code snippet, with a slight change runs under ruby1.9

def ip(filename)
ips = []
File.read(filename).lines.to_a.each do |place|
sf = 0
while sfn =
place.index(/(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/,sf)
sf = sfn + 3
ips << $&
end
end
return ips
end

Thanks for your pointers
regards

Panagiotis (atmosx) Atmatzidis

email: [email protected]
URL: http://www.convalesco.org
GnuPG ID: 0xFC4E8BB4
gpg --keyserver x-hkp://pgp.mit.edu --recv-keys 0xFC4E8BB4

[atma]

You can certainly recreate the awk functionality. But for the lazy
programmer who likes the tool that already exists try something like:

awk_test.txt
this is some text 4

---

search_term = “some”
data = 2
file = “awk_test.txt”

open("|awk -F ‘#{search_term}’ ‘{print $#{data}}’ #{file}"){|f| @results = f.gets}
=> " \ttext\t4\n"

puts @results
text 4

You can also use the IO class to create pipe to a unix command like
awk or agrep
in this example(code I have in my .irbrc file):

#you need to have TEV agrep installed to use this code
def agrep_with_string(regex, string)
temp_file = ‘temp_file_for_agrep’
File.open(temp_file, ‘w’){|f| f.puts string}
str = “agrep --color -e ‘#{regex.source}’ #{temp_file}”
IO.popen(str){|io| @hits = io.readlines}
File.delete(temp_file)
@hits
end

[atma]

On Fri, Dec 4, 2009 at 3:09 PM, Panagiotis A.
[email protected] wrote:

I would like to use the output from the cli: “$ grep Ban fail2ban.log|awk -F “Ban” ‘{print $2}’”

60.12.200.xx
193.193.221.xx
85.72.xx.xx
124.207.xx.xx

$ echo ‘2009-11-19 00:31:29,928 fail2ban.actions: WARNING [ssh-ipfw]
Ban 203.169.139.171’ | awk -F “Ban” ‘{print $2}’
203.169.139.171

$ echo ‘2009-11-19 00:31:29,928 fail2ban.actions: WARNING [ssh-ipfw]
Ban 203.169.139.171’ | ruby -ne ‘print split(“Ban”)[1]’
203.169.139.171

def get_ips(filename)
ips = [] # make a list of ip addressses
File.foreach(filename) do |line|
puts “#{line}” if line =~ /Ban/
end
end

Incorporating that into your program snippet:

def get_ips(filename)
ips = []
File.foreach(filename) do |line|
next unless line =~ /Ban/
ips << line.split(“Ban”)[1]
end
ips
end

(Though in a real program, rather than split like above, I would
probably use a capturing regex to grab the IP address.)

Unless the file is huge, I would probably use scan:

def banned_ips_from(filename)
File.read(filename).scan(/Ban (.*)/).flatten
end

[atma]

On 05.12.2009 10:07, Panagiotis A. wrote:

On 04 2009, at 10:27, David M. wrote:

On Friday 04 December 2009 02:09:35 pm Panagiotis A. wrote:

The output though parses lines the following:

2009-11-19 00:31:29,928 fail2ban.actions: WARNING [ssh-ipfw] Ban
203.169.139.171

I would like to use the output from the cli: “$ grep Ban fail2ban.log|awk
-F “Ban” ‘{print $2}’”

end
  return ips

end

It seems that is making things unnecessary complicated (there is a
superfluous to_a in there etc.). That should be sufficient for your
case:

require ‘set’

def ips(file_name)
ips = Set.new # [] if you do want dups

File.foreach file_name do |line|
ip = line[/Ban\s+(\d{1,3}(?:.\d{1,3}){3})/, 1] and
ips << ip
end

ips
end

Note: I am using a Set here in order to avoid reporting duplicates. If
you want duplicates just use an array as indicated above.

Other than that this is what the code does: File.foreach iterates a file
line by line. The expression line[/…/, 1] extracts capturing group 1
of the regexp if it matches. Otherwise you get nil. Then I use that
information immediately with “and” to decide whether to add to the set
or not. The regular expression is not as selective as the one you have
found in terms of the IP but it does match the “Ban” so it should be
safer (after all, you know the format of the file). Also, the piece you
found seems to iterate while in your file there is at most one IP per
line (if your example covers all options).

Plus, the iteration with index is awkward and inefficient. In Ruby you
would rather do

place.scan /…/ do |ip|
ips << ip
end

But as I said, scanning is not necessary if there is at most a single IP
per line.

And, last but not least a more modular solution in which you do not need
an Array or Set but rather use Ruby’s block feature to deal with
individual IPs:

def ban_ips(file_name)
File.foreach file_name do |line|
ip = line[/Ban\s+(\d{1,3}(?:.\d{1,3}){3})/, 1] and
yield ip
end
end

With that you can do

ban_ips “fail2ban.log” do |ip|
printf “found IP %-15s\n”, ip
end

or

ips = []

ban_ips “fail2ban.log” do |ip|
ips << ip
end

puts “found #{ips.size} IPs”

Welcome to the wonderful world of Ruby!

Kind regards

robert

[atma]

On 06.12.2009 20:01, Panagiotis A. wrote:

Good evening (it’s 8:48 pm here)!

Are you in Greece? It seems the country is shaken by riots currently.
I hope, no more people get killed in the course of action. It’s enough
that a young boy died a year ago.

True. There is 1 ip per line and some duplicates but there’s a catch also. There are some IP’s that are captured more than 1 time with the “Ban” flag. Which means that they were captured in a different time. Fail2ban blocks the IP for a couple of minutes in order to avoid the password brute-force which is taking place. After 3 minutes Unbans the ip. So the usual kind of log is this:

2009-11-15 15:19:35,222 fail2ban.actions: WARNING [ssh-ipfw] Ban 195.66.191.75
2009-11-15 15:29:35,643 fail2ban.actions: WARNING [ssh-ipfw] Unban 195.66.191.75
2009-11-16 07:46:59,854 fail2ban.actions: WARNING [ssh-ipfw] Ban 203.172.184.130
2009-11-16 07:57:00,085 fail2ban.actions: WARNING [ssh-ipfw] Unban 203.172.184.130

[*I leave the IP’s intact because these are actual SSH attacks and… if the admin don’t care for his host, neither do I.]

So at this point, I need really to display duplicates and maybe issue a bold warning when an IP appears more than 5 times. It means that your host is probably targeted.

In that case you probably want to use a Hash for counting like this

ips = Hash.new 0

…
if ips[ip] += 1 >= 5
$stderr.puts “WARNING: potential attack from #{ip}!”
end
…

Welcome to the wonderful world of Ruby!

Thanks!!! THe learning process is for sure much easier than Objective-C, the syntax much more straight forward, but some concepts I’m still struggling to understand them! Thanks for your reply though, it was very enlightening.

You’re welcome!

PS. yes I know I can install SNORT and get over it, but it’s much funnier creating your own programs!

Absolutely!

Oh, this mailing list is really good

Thanks a bunch! This group is among the friendlies places I know for
exchange of technical thoughts.

Kind regards

robert

[atma]

Good evening (it’s 8:48 pm here)!

First I would like to thank you all for the responses. I’ve got really
much more than I hoped too when I sent the post.

On 06 Δεκ 2009, at 2:30 μ.μ., Robert K. wrote:

-F “Ban” ‘{print $2}’"
return ips
ip = line[/Ban\s+(\d{1,3}(?:.\d{1,3}){3})/, 1] and
ips << ip
end

ips
end

Note: I am using a Set here in order to avoid reporting duplicates. If you want duplicates just use an array as indicated above.

Yes. I did not write this function myself. I found it on the internet,
and adjust it just a little bit to make it work with 1.9 version. I have
no idea at this point what place.index does neither “set” which you
used. Today I reached the “hashes” chapter and still did not finnish it
all! But I’m eager to write actual programs that’s why I started working
on small project in parallel with the book. It’s a good practice helps
you understand things although it may turn a bit confusing at times.

Other than that this is what the code does: File.foreach iterates a file line by line. The expression line[/…/, 1] extracts capturing group 1 of the regexp if it matches. Otherwise you get nil. Then I use that information immediately with “and” to decide whether to add to the set or not. The regular expression is not as selective as the one you have found in terms of the IP but it does match the “Ban” so it should be safer (after all, you know the format of the file). Also, the piece you found seems to iterate while in your file there is at most one IP per line (if your example covers all options).

True. There is 1 ip per line and some duplicates but there’s a catch
also. There are some IP’s that are captured more than 1 time with the
“Ban” flag. Which means that they were captured in a different time.
Fail2ban blocks the IP for a couple of minutes in order to avoid the
password brute-force which is taking place. After 3 minutes Unbans the
ip. So the usual kind of log is this:

2009-11-15 15:19:35,222 fail2ban.actions: WARNING [ssh-ipfw] Ban
195.66.191.75
2009-11-15 15:29:35,643 fail2ban.actions: WARNING [ssh-ipfw] Unban
195.66.191.75
2009-11-16 07:46:59,854 fail2ban.actions: WARNING [ssh-ipfw] Ban
203.172.184.130
2009-11-16 07:57:00,085 fail2ban.actions: WARNING [ssh-ipfw] Unban
203.172.184.130

[*I leave the IP’s intact because these are actual SSH attacks and… if
the admin don’t care for his host, neither do I.]

So at this point, I need really to display duplicates and maybe issue a
bold warning when an IP appears more than 5 times. It means that your
host is probably targeted.

Plus, the iteration with index is awkward and inefficient. In Ruby you would rather do

place.scan /…/ do |ip|
ips << ip
end

But as I said, scanning is not necessary if there is at most a single IP per line.

I’ll keep that in mind. I never used the .scan function anyway.

With that you can do

ban_ips “fail2ban.log” do |ip|
printf “found IP %-15s\n”, ip
end

Although I dislike the printf use here, this is a code snippet that I
understand entirely, which is a good thing

Welcome to the wonderful world of Ruby!

Thanks!!! THe learning process is for sure much easier than Objective-C,
the syntax much more straight forward, but some concepts I’m still
struggling to understand them! Thanks for your reply though, it was very
enlightening.

Kind regards

robert

–
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

PS. yes I know I can install SNORT and get over it, but it’s much
funnier creating your own programs! Oh, this mailing list is really
good

Panagiotis (atmosx) Atmatzidis

email: [email protected]
URL: http://www.convalesco.org
GnuPG ID: 0xFC4E8BB4
gpg --keyserver x-hkp://pgp.mit.edu --recv-keys 0xFC4E8BB4

[atma]

Hi,

Am Samstag, 05. Dez 2009, 05:09:35 +0900 schrieb Panagiotis A.:

2009-11-19 00:31:29,928 fail2ban.actions: WARNING [ssh-ipfw] Ban 203.169.139.171

I would like to use the output from the cli: “$ grep Ban fail2ban.log|awk -F “Ban” ‘{print $2}’”

The correct translation of `-F Ban “{print $2}”’ would be

ary = line.split /Ban/
puts ary[ 2]

I would prefer something like

if line =~ /Ban\s+/ then
puts $’.split.first
end

You can play around that for the rest of the day:

ary = line.split
while ary.any? do
if ary.shift == “Ban” then
puts ary.first
break
end
end

I don’t think it’s worth here the effort of IP address parsing but
thats surely a matter of taste.

Bertram

[atma]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,
On 07 Δεκ 2009, at 5:33 μ.μ., Bertram S. wrote:

---

puts ary[ 2]
while ary.any? do
if ary.shift == “Ban” then
puts ary.first
break
end
end

I don’t think it’s worth here the effort of IP address parsing but
thats surely a matter of taste.

Bertram

First, thanks for the code.

The parsing that I use was ready actually and works quite nice. I’m not
able to write regexp at that level yet. That’s why I used it.

–
Bertram S.
Stuttgart, Deutschland/Germany
*
Discover String#notempty? at http://raa.ruby-lang.org/project/step.

Panagiotis (atmosx) Atmatzidis

email: [email protected]
URL: http://www.convalesco.org
GnuPG ID: 0xFC4E8BB4
gpg --keyserver x-hkp://pgp.mit.edu --recv-keys 0xFC4E8BB4

---

The wise man said: “Never argue with an idiot. They bring you down to
their level and beat you with experience.”

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAksdLsoACgkQrghUb/xOi7QRCACdFTT8hkstbPlj62mTTQyqkOJ0
DqQAn1w/mEOiVDPk1lgDHCCMVk15m/Ew
=rCzl
-----END PGP SIGNATURE-----

[atma]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
On 07 Δεκ 2009, at 12:50 π.μ., Robert K. wrote:

On 06.12.2009 20:01, Panagiotis A. wrote:

Good evening (it’s 8:48 pm here)!

Are you in Greece? It seems the country is shaken by riots currently. I hope, no more people get killed in the course of action. It’s enough that a young boy died a year ago.

Yes I am Greek. True, it’s a mess. This years, all the riots was just a
reminder of last year’s chaos. Where I live however, it’s okay, no
broken stores or burned cars around here, nothing happens.

ips = Hash.new 0

…
if ips[ip] += 1 >= 5
$stderr.puts “WARNING: potential attack from #{ip}!”
end
…

I used a similar approach, which is used by the book also. I’ve created
another file (like a library of sorts) and created new hash which keeps
track of the number an IP shows up.

statistics

class Stats
def self.count_ip_display(ips)
counts = Hash.new(0)
for ip in ips
counts[ip] += 1
end
sort = counts.sort_by {|ip, display| display}
#top_ten = sort.last(10).reverse
#top_ten
len = sort.length
ips
sort
end
end

I see that your sample above, is like " Hasn.new 0". I guess that’s just
another syntax for Hash.new(0) right?

Oh, this mailing list is really good

Thanks a bunch! This group is among the friendlies places I know for exchange of technical thoughts.

Kind regards

robert

Yes, it’s nice to feel good with other people. Having a Linux background
I’ve faced too many “elite-driven” hostile communities in the past to
understand the value of a healthy (of trolls, etc) ML

–
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Panagiotis (atmosx) Atmatzidis

email: [email protected]
URL: http://www.convalesco.org
GnuPG ID: 0xFC4E8BB4
gpg --keyserver x-hkp://pgp.mit.edu --recv-keys 0xFC4E8BB4

---

The wise man said: “Never argue with an idiot. They bring you down to
their level and beat you with experience.”

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAksdMBAACgkQrghUb/xOi7SbgACggQbt+oqU08lU4jT566JF0S2s
uDsAnRIJEn3dbqQrin+mQXHJItz8vVsF
=H/tM
-----END PGP SIGNATURE-----

[atma]

On Mon, Dec 7, 2009 at 4:40 PM, Panagiotis A.
[email protected] wrote:

I see that your sample above, is like " Hash.new 0". I guess that’s just another syntax for Hash.new(0) right?

Yes, parentheses are (usually) optional in Ruby.

Paul S.
http://www.nomadicfun.co.uk

[email protected]