Extract from output (regular expressions)

Ruby
1

Hi guys,

As a beginner, I’m looking for a way about how to extract selected
values from captured output.

Suppose, I call external command in this way:

def shorewall
begin
cmd = ‘shorewall show ipa’
ipa, s = Open3.capture2(cmd)
if s.success?
return ipa
end
rescue StandardError => e
return “No good, #{e.message}.”
end
end

After that, shorewall contains:

Shorewall 4.5.17.1 per-IP Accounting at fw1 - Sun Jun 9 18:38:05 EST
2013

Showing table: cowboys
IP: 192.168.200.1 SRC packets: 4196607 bytes: 292644635 DST packets:
7224498 bytes: 762829278
IP: 192.168.200.2 SRC packets: 77289 bytes: 4799573 DST packets: 324472
bytes: 481821874
IP: 192.168.200.3 SRC packets: 122875 bytes: 14531084 DST packets:
145748 bytes: 170332152
IP: 192.168.200.4 SRC packets: 142254 bytes: 18836983 DST packets:
106100 bytes: 26420351
IP: 192.168.200.5 SRC packets: 3424090 bytes: 181079976 DST packets:
6948051 bytes: 1798270528
IP: 192.168.200.6 SRC packets: 2758 bytes: 234311 DST packets: 12875
bytes: 18882957
IP: 192.168.200.9 SRC packets: 74277 bytes: 8864576 DST packets: 21888
bytes: 5834302
IP: 192.168.200.10 SRC packets: 583 bytes: 44836 DST packets: 623 bytes:
177505
IP: 192.168.200.253 SRC packets: 157 bytes: 11899 DST packets: 156
bytes: 12834

Showing table: indians
IP: 10.0.0.2 SRC packets: 1108147 bytes: 105913984 DST packets: 2698261
bytes: 3740346675
IP: 10.0.0.3 SRC packets: 362460 bytes: 145188605 DST packets: 300909
bytes: 136230642

Now, I would like to put this to the new array but under two conditions:

  1. only lines starting with IP should be considered
  2. only IP address and digital values should be extracted

So, the final array should look like this one:

192.168.200.1 4196607 292644635 7224498 762829278
192.168.200.2 77289 4799573 324472 481821874
192.168.200.3 122875 14531084 145748 170332152
192.168.200.4 142254 18836983 106100 26420351
192.168.200.5 3424090 181079976 6948051 1798270528
192.168.200.6 2758 234311 12875 18882957
192.168.200.9 74277 8864576 21888 5834302
192.168.200.10 583 44836 623 177505
192.168.200.253 157 11899 156 12834
10.0.0.2 1108147 105913984 2698261 3740346675
10.0.0.3 362460 145188605 300909 136230642

Or, perhaps, doesn’t have to be an array, I’m open for suggestions.
Later, I will process each line separately and transfer all records to
rrd database.

Cheers,
Thomas

2

This outputs it into a 2D Array and displays the results to the console,
if that helps.

Code:

require ‘pp’
a = File.foreach( ‘test.txt’ ).map do |line|
if line =~ /^IP: /
line.scan( /[\d.]+/ )
else
nil
end
end.to_a.compact

pp a

Output:

[[“192.168.200.1”, “4196607”, “292644635”, “7224498”, “762829278”],
[“192.168.200.2”, “77289”, “4799573”, “324472”, “481821874”],
[“192.168.200.3”, “122875”, “14531084”, “145748”, “170332152”],
[“192.168.200.4”, “142254”, “18836983”, “106100”, “26420351”],
[“192.168.200.5”, “3424090”, “181079976”, “6948051”, “1798270528”],
[“192.168.200.6”, “2758”, “234311”, “12875”, “18882957”],
[“192.168.200.9”, “74277”, “8864576”, “21888”, “5834302”],
[“192.168.200.10”, “583”, “44836”, “623”, “177505”],
[“192.168.200.253”, “157”, “11899”, “156”, “12834”],
[“10.0.0.2”, “1108147”, “105913984”, “2698261”, “3740346675”],
[“10.0.0.3”, “362460”, “145188605”, “300909”, “136230642”]]

3

Oops, I don’t know why I had “to_a” in there, works fine without it.

4

Look up Array#flatten

5

If you want it to just be an array of space-delimited strings, you could
make this change in the code.

Instead of this:
line.scan( /[\d.]+/ )

This:
line.scan( /[\d.]+/ ).join(’ ')

6

Thank you, it works for me but have one more question. Is there a way to
make it one dimensional array? I’m fine with 2d array, just a curiosity.

Something like this:

[“192.168.200.1, 4196607, 292644635, 7224498, 762829278”,
“192.168.200.2, 77289, 4799573, 324472, 481821874”,
“192.168.200.3, 122875, 14531084, 145748, 170332152”,
“192.168.200.4, 142254, 18836983, 106100, 26420351”,
“192.168.200.5, 3424090, 181079976, 6948051, 1798270528”,
“192.168.200.6, 2758, 234311, 12875, 18882957”,
“192.168.200.9, 74277, 8864576, 21888, 5834302”,
“192.168.200.10, 583, 44836, 623, 177505”,
“192.168.200.253, 157, 11899, 156, 12834”,
“10.0.0.2, 1108147, 105913984, 2698261, 3740346675”,
“10.0.0.3, 362460, 145188605, 300909, 136230642”]

Cheers,
Thomas

7

FYI the reason I’d use a 2D array for that kind of thing is because you
then have this option for looping through the data:

output.each do |ip, src_packets, src_bytes, dst_packets, dst_bytes|
#Whatever you want with each set of data…
end

8

On Wed, Jun 12, 2013 at 12:22 AM, Joel P. [email protected]
wrote:

FYI the reason I’d use a 2D array for that kind of thing is because you
then have this option for looping through the data:

output.each do |ip, src_packets, src_bytes, dst_packets, dst_bytes|
#Whatever you want with each set of data…
end

I’d give even more structure to the data: I use a Hash as element type
in
the Array plus I’d use a Hash on root level to distinguish tables.

#!/usr/bin/ruby

$w = true

require ‘ipaddr’

data = Hash.new {|h,k| h[k] = []}
current = nil

ARGF.each_line do |line|
case line
when /Showing table:\s*(\w+)/
current = data[$1]
when /^IP/
h = {}

line.scan /(\w+):\s*([\d.]+)/ do |match|
  key = $1.to_sym
  val = $2

  h[key] = case val
    when /\A\d+\z/
    Integer(val)
  when /\A\d{1,3}(?:\.\d{1,3}){3}\z/
    IPAddr.new val
  else
    val
  end
end

current << h

end
end

require ‘pp’
pp data

$ ./ip.rb fw.data
{“cowboys”=>
[{:IP=>#<IPAddr: IPv4:192.168.200.1/255.255.255.255>,
:packets=>7224498,
:bytes=>762829278},
{:IP=>#<IPAddr: IPv4:192.168.200.2/255.255.255.255>,
:packets=>324472,
:bytes=>481821874},
{:IP=>#<IPAddr: IPv4:192.168.200.3/255.255.255.255>,
:packets=>145748,
:bytes=>170332152},
{:IP=>#<IPAddr: IPv4:192.168.200.4/255.255.255.255>,
:packets=>106100,
:bytes=>26420351},
{:IP=>#<IPAddr: IPv4:192.168.200.5/255.255.255.255>,
:packets=>6948051,
:bytes=>1798270528},
{:IP=>#<IPAddr: IPv4:192.168.200.6/255.255.255.255>,
:packets=>12875,
:bytes=>18882957},
{:IP=>#<IPAddr: IPv4:192.168.200.9/255.255.255.255>,
:packets=>21888,
:bytes=>5834302},
{:IP=>#<IPAddr: IPv4:192.168.200.10/255.255.255.255>,
:packets=>623,
:bytes=>177505},
{:IP=>#<IPAddr: IPv4:192.168.200.253/255.255.255.255>,
:packets=>156,
:bytes=>12834}],
“indians”=>
[{:IP=>#<IPAddr: IPv4:10.0.0.2/255.255.255.255>,
:packets=>2698261,
:bytes=>3740346675},
{:IP=>#<IPAddr: IPv4:10.0.0.3/255.255.255.255>,
:packets=>300909,
:bytes=>136230642}]}

In case indentation breaks I put it on github

Kind regards

robert