Unwanted scan() method behavior

Ruby
1

Greeting to everyone from Greece!

I’m writing a string which will make a list of ‘fail2ban.log’ captured
IP addresses dump them into SQLite and them display some statistics via
Sinatra. I’m using scan() method to grab the needed lines. The lines I’d
like to grab are like this:

2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Ban
78.xxx.xxx.17x

Although these are ssh brute-force login attempts I’d switch numbers
with ‘x’ letters in the above sample.

A typical fail2ban.log file is like this:

–
2011-07-23 02:03:50,741 fail2ban.server : INFO Changed logging target
to /var/log/fail2ban.log for Fail2ban v0.8.4
2011-07-23 02:03:50,743 fail2ban.jail : INFO Creating new jail
‘ssh-ipfw’
2011-07-23 02:03:50,745 fail2ban.jail : INFO Jail ‘ssh-ipfw’ uses
poller
2011-07-23 02:03:50,853 fail2ban.filter : INFO Added logfile =
/var/log/secure.log
2011-07-23 02:03:50,856 fail2ban.filter : INFO Set maxRetry = 3
2011-07-23 02:03:50,859 fail2ban.filter : INFO Set findtime = 600
2011-07-23 02:03:50,861 fail2ban.actions: INFO Set banTime = 600
2011-07-23 02:03:51,030 fail2ban.jail : INFO Jail ‘ssh-ipfw’ started
2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Ban
78.xxx.xxx.17x
2011-07-23 02:14:51,441 fail2ban.actions: WARNING [ssh-ipfw] Unban
78.xxx.xxx.17x
2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Ban
2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Ban
2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Ban
2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Unban
2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Unban
2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Unban

Here is my script so far:

class Myzonereport
attr_reader :logfile

def initialize(logfile)
raise “No fail2ban log file found!” if (logfile.empty?)
@logfile = logfile
end

def readlog
puts “I can’t read the log file” unless (File.readable?(@logfile) ||
File.empty?(@logfile))
log = File.read(@logfile)
log.scan(/^(\d{4}-\d\d-\d\d).?(\d{2}:\d{2}:\d{2},\d{3}).?(Ban).*?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})/).each
do |date, time, string, ip|
puts “id: #{time} | date: #{date} | IP: #{ip}”
end
end

end

x = Myzonereport.new(‘fail2ban.log’)
puts x.readlog

My problem though is that the output is printed 2 times. The first time
in the form I want using puts and a second time in ‘raw mode’. Like
this:

id: 23:37:50,235 | date: 2011-08-09 | IP:
id: 02:09:32,868 | date: 2011-08-10 | IP:

2011-07-23
02:04:51,107
Ban

2011-07-23
05:22:45,963
Ban

2011-07-23
12:07:25,377
Ban

[]

I can’t tell why this happens. Should I use another method in order to
grab the pattern I want? Is this scan’s default behavior? I’m getting
same results if I don’t use any (puts or other) method in the loop.

Best Regards & thanks in advance for your time

–
Panagiotis A.

personal: [email protected]
lists: [email protected]
blog: http://www.convalesco.org

The wise man said: “Never argue with an idiot. They bring you down to
their level and beat you with experience.”

2

Did you mean to use log.scan(//).each{} instead of
log.scan(//){} ? Note that your version calls each on the
results of scan, instead of passing a block to scan.

The return value of each is simply the array it was called on itself.

The return value of scan w/o block is array of groups matched by regex
(arrays), w/ block - the string itself.

– Matma R.

3

I re-read your mail and realized that I don’t really understand what
you’re trying to accomplish, and my previous mail wasn’t probably
really useful. Sorry.

– Matma R.

4

I would say, you should only call
x.readlog
and not
puts x.readlog

You call put in the method which gives you the desired output. The ‘raw’
output is the return value of the readlog method, which you don’t want
to be
printed.

2011/8/14 Panagiotis A. [email protected]

5

Hello,

Thanks for reply,

On 14 Αυγ 2011, at 9:06 μ.μ., Bartosz Dziewoński wrote:

I re-read your mail and realized that I don’t really understand what
you’re trying to accomplish, and my previous mail wasn’t probably
really useful. Sorry.

I don’t blame you for that, neither would I probably.

– Matma R.

Here’s a better explanation:

$ sed ‘s/[0-9]{1,3}(.[0-9]{1,3}){3}/(127.0.0.1)/’ <fail2ban.log

fail2ban.log
$ cat fail2ban.log

2011-08-07 23:32:09,210 fail2ban.server : INFO Changed logging target
to /var/log/fail2ban.log for Fail2ban v0.8.4
2011-08-07 23:32:09,237 fail2ban.jail : INFO Creating new jail
‘ssh-ipfw’
2011-08-07 23:32:09,239 fail2ban.jail : INFO Jail ‘ssh-ipfw’ uses
poller
2011-08-07 23:32:09,373 fail2ban.filter : INFO Added logfile =
/var/log/secure.log
2011-08-07 23:32:09,376 fail2ban.filter : INFO Set maxRetry = 3
2011-08-07 23:32:09,379 fail2ban.filter : INFO Set findtime = 600
2011-08-07 23:32:09,381 fail2ban.actions: INFO Set banTime = 600
2011-08-07 23:32:09,659 fail2ban.jail : INFO Jail ‘ssh-ipfw’ started
2011-08-08 07:37:00,199 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-08 07:37:04,328 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-08 07:47:00,650 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-08 07:47:05,248 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-09 00:25:38,918 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-09 00:35:39,631 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-09 03:58:18,229 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-09 04:08:18,904 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-09 06:04:18,785 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-09 06:14:19,403 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-09 09:43:32,351 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-09 09:53:32,964 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-09 23:07:48,462 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-09 23:17:49,334 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-09 23:37:50,235 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-09 23:47:50,446 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-10 02:09:32,868 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-10 02:19:33,067 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-10 13:41:46,288 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-10 13:51:47,117 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-10 22:50:44,647 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-10 23:00:45,106 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-10 23:18:48,976 fail2ban.actions: WARNING [ssh-ipfw] Ban
127.0.0.1
2011-08-10 23:28:49,140 fail2ban.actions: WARNING [ssh-ipfw] Unban
127.0.0.1
2011-08-11 07:32:36,636 fail2ban.server : INFO Changed logging target
to /var/log/fail2ban.log for Fail2ban v0.8.4
2011-08-11 07:32:36,664 fail2ban.jail : INFO Creating new jail
‘ssh-ipfw’
2011-08-11 07:32:36,666 fail2ban.jail : INFO Jail ‘ssh-ipfw’ uses
poller
2011-08-11 07:32:36,800 fail2ban.filter : INFO Added logfile =
/var/log/secure.log
2011-08-11 07:32:36,802 fail2ban.filter : INFO Set maxRetry = 3
2011-08-11 07:32:36,806 fail2ban.filter : INFO Set findtime = 600
2011-08-11 07:32:36,808 fail2ban.actions: INFO Set banTime = 600
2011-08-11 07:32:36,974 fail2ban.jail : INFO Jail ‘ssh-ipfw’ started

$ cat myzonereport.rb

#!/usr/bin/env ruby

encoding: UTF-8

ZoneReport version v.02-alpha

[email protected]

#require ‘socket’

class Myzonereport
attr_reader :logfile

def initialize(logfile)
raise “No fail2ban log file found!” if (logfile.empty?)
@logfile = logfile
@list = Hash.new
end

def readlog
puts “I can’t read the log file” unless (File.readable?(@logfile) ||
File.empty?(@logfile))
log = File.read(@logfile)
log.scan(/^(\d{4}-\d\d-\d\d).?(\d{2}:\d{2}:\d{2},\d{3}).?(Ban).*?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})/).each
do |date, time, string, ip|
puts “id: #{time} | date: #{date} | IP: #{ip}”
end
end

def rename
time = Time.new
date = time.strftime(“%Y-%m-%d”)
if (File.exists?(@logfile) && File.readable?(@logfile))
File.rename(@logfile, @logfile + “-” + date + ‘.log’)
File.new(@logfile)
else
puts “File ‘#{@logfile}’ does not exist or it’s not readable!”
end
end

end

x = Myzonereport.new(‘fail2ban.log’)
puts x.readlog[1m[3m%[23m[1m[0m

$ ruby myzonereport.rb

atma@angel:/Volumes/atmosx/Programming/test/ > ruby myzonereport.rb
id: 07:37:00,199 | date: 2011-08-08 | IP: 127.0.0.1
id: 07:37:04,328 | date: 2011-08-08 | IP: 127.0.0.1
id: 00:25:38,918 | date: 2011-08-09 | IP: 127.0.0.1
id: 03:58:18,229 | date: 2011-08-09 | IP: 127.0.0.1
id: 06:04:18,785 | date: 2011-08-09 | IP: 127.0.0.1
id: 09:43:32,351 | date: 2011-08-09 | IP: 127.0.0.1
id: 23:07:48,462 | date: 2011-08-09 | IP: 127.0.0.1
id: 23:37:50,235 | date: 2011-08-09 | IP: 127.0.0.1
id: 02:09:32,868 | date: 2011-08-10 | IP: 127.0.0.1
id: 13:41:46,288 | date: 2011-08-10 | IP: 127.0.0.1
id: 22:50:44,647 | date: 2011-08-10 | IP: 127.0.0.1
id: 23:18:48,976 | date: 2011-08-10 | IP: 127.0.0.1
2011-08-08
07:37:00,199
Ban
127.0.0.1
2011-08-08
07:37:04,328
Ban
127.0.0.1
2011-08-09
00:25:38,918
Ban
127.0.0.1
2011-08-09
03:58:18,229
Ban
127.0.0.1
2011-08-09
06:04:18,785
Ban
127.0.0.1
2011-08-09
09:43:32,351
Ban
127.0.0.1
2011-08-09
23:07:48,462
Ban
127.0.0.1
2011-08-09
23:37:50,235
Ban
127.0.0.1
2011-08-10
02:09:32,868
Ban
127.0.0.1
2011-08-10
13:41:46,288
Ban
127.0.0.1
2011-08-10
22:50:44,647
Ban
127.0.0.1
2011-08-10
23:18:48,976
Ban
127.0.0.1

I would like to output to be just:

id: 07:37:00,199 | date: 2011-08-08 | IP: 127.0.0.1
id: 07:37:04,328 | date: 2011-08-08 | IP: 127.0.0.1
id: 00:25:38,918 | date: 2011-08-09 | IP: 127.0.0.1
id: 03:58:18,229 | date: 2011-08-09 | IP: 127.0.0.1
id: 06:04:18,785 | date: 2011-08-09 | IP: 127.0.0.1
id: 09:43:32,351 | date: 2011-08-09 | IP: 127.0.0.1
id: 23:07:48,462 | date: 2011-08-09 | IP: 127.0.0.1
id: 23:37:50,235 | date: 2011-08-09 | IP: 127.0.0.1
id: 02:09:32,868 | date: 2011-08-10 | IP: 127.0.0.1
id: 13:41:46,288 | date: 2011-08-10 | IP: 127.0.0.1
id: 22:50:44,647 | date: 2011-08-10 | IP: 127.0.0.1
id: 23:18:48,976 | date: 2011-08-10 | IP: 127.0.0.1

Hope this example makes the issue more clear.

Best Regards!

–
Panagiotis A.

personal: [email protected]
lists: [email protected]
blog: http://www.convalesco.org

The wise man said: “Never argue with an idiot. They bring you down to
their level and beat you with experience.”

6

On Mon, Aug 15, 2011 at 12:48 AM, Panagiotis A.
[email protected] wrote:

output is the return value of the readlog method, which you don’t want to be
printed.

I had the feeling that it was something that obvious that I’m missing but I
focused so much on the function that I missed it!

A few more remarks: since those files can be large a better choice is
usually to use File.foreach. You can also shorten your regexp a bit
by using non capturing groups with repetition markers.

Example (untested):

File.foreach @logfile do |line|
if
/^(\d{4}(?:-\d\d){2}).?(\d{2}(?::\d{2}){2},\d{3}).?(Ban).*?(\d{1,3}(?:.\d{1,3}){3})/
=~ line
puts “id: #$2 | date: #$1 | IP: #$3”
end
end

Kind regards

robert

7

Hello,

On 15 Αυγ 2011, at 6:06 μ.μ., Robert K. wrote:

Example (untested):
robert

–
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Thanks for the pointers Robert! I’ll do some checks and probably
implement your solution which seems more appropriate,

Best Regards

–
Panagiotis A.

personal: [email protected]
lists: [email protected]
blog: http://www.convalesco.org

The wise man said: “Never argue with an idiot. They bring you down to
their level and beat you with experience.”

8

Hello,

On 14 Αυγ 2011, at 10:22 μ.μ., Gunther D. wrote:

I would say, you should only call
x.readlog
and not
puts x.readlog

You call put in the method which gives you the desired output. The ‘raw’
output is the return value of the readlog method, which you don’t want to be
printed.

I had the feeling that it was something that obvious that I’m missing
but I focused so much on the function that I missed it!

Thanks :slight_smile:

2011-07-23 02:04:51,107 fail2ban.actions: WARNING [ssh-ipfw] Ban
2011-07-23 02:03:50,743 fail2ban.jail : INFO Creating new jail
78.xxx.xxx.17x
Here is my script so far:
puts “I can’t read the log file” unless (File.readable?(@logfile) ||

2011-07-23

–
Panagiotis A.

personal: [email protected]
lists: [email protected]
blog: http://www.convalesco.org

The wise man said: “Never argue with an idiot. They bring you down to
their level and beat you with experience.”