Cdtjkyj

I am examining Bro logs and would like to find a way to use AWK to grab IPs from a tracker link and search the Bro logs for those IPs automatically, printing the results of each match into [custom-name].txt file.

MORE INFO:

I am new to awk but trying to piece things together. I am not beholden to awk if there is a better way to do this. Efficient would be good.

Right now I am manually using an Excel function to concatenate the column of bad IPs (which are updated regularly from the link) into a super long awk OR string:

cat http* | awk '/123.456.789.012/ || /2nd IP/ || /3rd IP/' | sort -n > file.txt

Currently, the http.log has the normal Bro headers... TS UID ID.orig_h ID.orig_p etc.

In layman's terms, what I'm trying to do is:

If the IP on line 1 of link is found in http.log, print results to line_1_ip.txt, If the IP on line 2 of link is found in http.log, print results to line_2_ip.txt, et al.

A snippet of the IPs would look like:

#############################################################################################

# abuse.ch ZeuS IP blocklist "BadIPs" (excluding hijacked sites and free hosting providers) #

#                                                                                           #

# For questions please refer to https://zeustracker.abuse.ch/blocklist.php                  #

#############################################################################################



103.230.84.239

103.241.0.100

103.26.128.84

103.4.52.150

103.7.59.135

104.218.48.99

104.219.232.12

104.236.55.19

107.150.58.84

108.61.193.211

It is just a text file.

When it comes to actually referencing a remote webpage to pull IPs from, I am honestly not sure where to even start. But if I can get it to pull from a local text file, that will give me a starting point from which to work.

I'm not entirely sure on what you want to parse since I haven't seen the input format but, if you just want to read a list of IPs and print to the corresponding output file, you could try this:

awk 'NR==FNR{a[$1]=NR; next} {

    for(ip in a){ 

        if($0~ip){ 

            print >> "line_"a[ip]"_ip.txt" 

        } 

    }

}}' <(wget -O - "https://zeustracker.abuse.ch/blocklist.php?download=badips" 2>/dev/null | 

    grep -Ev "#|^ *$") http.log 

That's a little complex though, so let's break it down:

• 
  wget -O - https://zeustracker.abuse.ch/blocklist.php?download=badips 2>/dev/null | grep -Ev "#|^ *$" : this will download the list of IPs, remove any lines that contain # or are empty and print to standard output.


• 
  <(command) : This is called process substitution and is a trick that some shells (bash, zsh and ksh, but maybe others as well) have which enables you to treat the output of a process as an input file to another command.


• So, the awk snipet sees two "files", the first is the output of the wget command piped trough grep and the second is the http.log file.


• 
  NR==FNR{a[$1]=NR; next} : NR is the current input line number and FNR is the same but of the current file. The two will be equal only while the first file is being read. This will, therefore, create the array a whose keys are the IPs and whose value is the current line number.


• 
  for(ip in a) : iterate over the a array, saving each of its keys (the IPs) as ip.


• 
  if($0~ip){} : if this line ($0) matches this ip.


• 
  print >> "line_"a[ip]"_ip.txt" : print the line into a file called line_, the number of the input line where this IP was seen and _ip.txt. The >> ensures that the file appended to and not overwritten.

Note that bro comes with bro-cut to extract data from its log files.

You could do:

bro-cut id.orig_h id.resp_h < conn.log | grep -wFf <(

  curl 'https://zeustracker.abuse.ch/blocklist.php?download=badips' |

  grep '^[^#]'

)

With awk:

awk '

  NR == FNR {

    if (/^[^#]/) bad[$0]=""

    next

  }

  ($1 in bad) || ($2 in bad)

' <(curl 'https://zeustracker.abuse.ch/blocklist.php?download=badips') 

  <(bro-cut id.orig_h id.resp_h < conn.log)

(more efficient as using hash tables instead of painstaking string matching in a loop).

Also note that the emerging threat rule-set has rules that track those Zeus tracker lists (and others). So if you use snort/surricata with that rule set in addition to bro-ids, doing it with bro would end up being redundant.