Monitoring pf logs with Gource

tcpdump pflog0 through GourceEver wanted to see your OpenBSD pf(4) logs in a cinematic way?

This post will demonstrate the use of Gource (software version control visualization tool) as a means to visualize our pf firewall logs.

The examples in this post were carried out on OpenBSD piping output from pflog(4) to a Linux workstation with OpenGL extensions support.

Background

Having organised a couple of capture the flag (CTF) competitions in the past, we have looked into ways of providing the by-standing audience with visual feeds and informational streams regarding the competition progress and player activity.

Back when we were preparing AthCon CTF, we came across this cool way to visualise our CVS history with Gource. Poking around Gource, we realized it supports a custom log format as input, and then it hit us! We could transform network activity data into a pipe-delimited format to create an impressive visualization of real-time activity on the wire, providing an incredible view of in-game action.

Today, this technique is impressively useful (pun intended) to monitor packets that are logged by OpenBSD's Packet Filter.

Logging packets on the firewall

OpenBSD's pf(4) provides the ability to log packets that match certain rules.

In the firewall ruleset /etc/pf.conf, filter actions (block,match,pass) can be accompanied by the log parameter, to generate a log message in addition to performing the action specified.

Typically, a pf ruleset begins (or ends) with a block action, such as:

block log

Note that, when logging match or pass rules, only the initial packet of each connection is logged, unless log (all) or no state option is specified.

The standard way to monitor those packets is running tcpdump against either the log files produced by pflogd (usually /var/log/pflog) or by directly accessing the packet filter logging interface (usually pflog0) for a real-time view of logged packets. Both uses are demonstrated below:

# run against pflog file (playback)
tcpdump -r /var/log/pflog 

# run against pflog0 interface (realtime)
tcpdump -i pflog0 

Depending on the activity one decides to log on the firewall, the resulting visualizations of gource can differ significantly.

UPDATE: All tcpdump commands on this post are for the tcpdump utility that is included with OpenBSD. Bear in mind that tcpdump is different on other platforms and will not operate correctly on pflog files.

The gource

Gource(1) is an OpenGL-based 3D visualization tool intended for visualizing activity on source control repositories.

Git history on Gource screenshot

When gource is fed with logs from a repository, such as cvs or git, the repository is displayed as a tree, forming branches to represent the directory structure and leaves/nodes to display files. Usernames from contributors to the source code appear and disappear as they contribute to specific files.

The tool supports data input in a custom log format:

timestamp|username|type|file|colour

Our idea was to use the username and file fields for source and destination IP respectively. We put together a quick awk script to process the output of tcpdump and transform it into the custom log format supported by gource.

The awk

The output extracted from tcpdump is transformed into gource's custom log format, using this quickly written tcpdump2gource.awk script:

#!/usr/bin/awk -f
# Convert `tcpdump -ttnql` output into gource custom format
{
  timestamp=$1
  sip=$2;
  proto=$5;
  dip=substr($4, 1,length($4)-1);
  direction=$3;
  type="";
  match(dip, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)
  dport=substr(dip,RLENGTH+2);
  gsub(".[0-9]*$","",timestamp);
  gsub(".[0-9]*$","",dip);
  network=dip;
  gsub(".[0-9]*$","",network);
  gsub(".[0-9]*$","",sip);
  printf("%s|%s|A|%s/%s.%s|#FF00ff\n",timestamp,sip,dip,dport,proto);
  # fflush();
}

Putting them all together

During our initial takes we noticed that pipe buffering was affecting our visuals. Thankfully, there are ways to disable i/o buffering:

  • tcpdump features a command line flag that makes stdout line-buffered.
  • awk is launched using stdbuf, which is part of the coreutils package.

With the awk script and the pflog logfile in place, visualizing logged packets is a breeze:

tcpdump -ttnql -r pflog.log 'proto \tcp' or 'proto \udp' \
| stdbuf -i0 -o0 -e0 awk -f tcpdump2gource.awk \
| gource --multi-sampling --no-vsync --title "pflog0 activity" --key \
  --realtime --highlight-users --highlight-dirs --user-friction 0.2 \
  --user-scale 0.8 --log-format custom --disable-auto-rotate -i 0 -

Simple tcpdump pflog0 through Gource

This is the simplest visual you can get from pflog activity, creating a tree hierarchy of attacked hosts and ports for each host. It is by no means complete, but leaves a lot of options for customization and improvement.

Tweaking to taste

Fiddling with the awk script, different tcpdump filters and the various command line options supported by gource can bring up other revelations:

For example, replacing the last statement in tcpdump2gource.awk with

printf("%s|%s|A|%s/%s/%s:%s.%s|#FF00ff\n",
       timestamp,sip,dip,sip,sip,proto,dport);

will produce further node structures in the hierarchy tree that is rendered from gource, visually grouping traffic by source IP (in addition to the destination IP and port attacked).

tcpdump pflog0 through Gource

Further adapting the script allows monitoring live traffic. Running sudo tcpdump on the firewall may not be advised, but it sure is a lot of fun:

ssh pf.example.net \
"sudo tcpdump -ttnql -i pflog0 'proto \tcp' or 'proto \udp'" \
 | stdbuf -i0 -o0 -e0 awk -f tcpdump2gource.awk \
 | gource --multi-sampling --no-vsync --title "pflog0 activity" --key \
   --realtime --highlight-users --user-friction 0.2 --user-scale 0.8 \
   -e 0.5 --log-format custom -i 300 --max-files 500 -

And here is a short video demonstrating how this looks (notice an nmap scan popping up at around 0:33)

Final words

At Echothrust Solutions, we regularly come up with irregular ideas on putting things to work beautifully in unexpected ways.

Far from being the ideal tool for visualizing network traffic, adapting this method has brought good results both for presentation and for monitoring purposes. Between its caveats, when bulk traffic occurs, gource will sometimes fall behind, struggling to produce a consistent visual representation; one can use the speed control function to have gource advance to real time.

Feeding network activity to gource conveniently highlights network connections on a constantly monitored interface:

  • Irregular network activity causes new bursts of movement in the visualization, making the traffic easier to spot.
  • The visual grouping of different network streams provides a formidable bird's eye view of the network activity, difficult to obtain from tcpdump text output.
  • Visualisation playback can be fast-forwarded using the speed control function (keyboard +/-).

Yes, we mostly use it for fun, but it has also been praised during CTF contests; participants adore watching their packets projected.

UPDATE: This post received much feedback from readers. I would like to thank Mike Cole for his corrections, @raffaelmarty for his remarks and, last but not least, @bsdnow for airing this on their show.

References