Using syslog and Echofish to detect persistent threats on your networks
Have you checked your server logs lately? Did you see those "odd" requests from arbitrary IPs that appear to perform a single request and "vanish"? Have you ever wondered how many of those are actually random? Do they return ? How often?
No matter which service you expose to the internet (http, ssh, smtp, imap), you are certain to notice protocol-aware requests (e.g. valid HTTP get request) from random IP addresses hitting your public services.
The following blog post focuses around answering these questions and the ways we utilize the Abuser module of Echofish to identify persistent attackers on our services, that would otherwise stay unnoticed.
Echofish can ease the process of identifying abuse on services through the Abuser module. Abuser is focused on extracting IP addresses from syslog messages that indicate policy violations.
The extraction is configured and handled by user-defined Triggers, a combination of match rules and extraction rules. These Triggers enable Echofish to continuously gather intelligence from matching messages and report abuse findings as Incidents, grouped by the extracted IP address.
Ever since we introduced the Abuser module in Echofish, persistent threats have become easier to spot. This is mainly due to the fact that Incidents make more sense when grouped by source IP.
Real world example
A real case example can better explain this functionality; one of the abuser triggers we have created, extracts the IP address of clients that attempted authentication through the Postfix MTA and failed. The trigger essentially keeps track of this "abuse" signature and increases the counter for each offending IP.
So for the following messages,
postfix/smtpd: lost connection after AUTH from unknown[x.x.x.x] postfix/smtpd: lost connection after AUTH from unknown[x.x.x.x] postfix/smtpd: lost connection after AUTH from unknown[x.x.x.x]
the IP `x.x.x.x` will have an incident counter of 3. Now you might think that you can do that with a simple `grep | wc` and you might be right. However, when these log entries span within the timeframe of a year, you can't possibly `grep` all those log files. Do you even keep them around handy? In most of the cases this would be a 'no'.
With the Abuser module, monitoring those Incidents from Echofish is a simple process:
- log into Echofish
- go to abuser incidents
- on the table filters enter "First Occurrence: <=2015-01-01" and on "Last Occurence: >=2015-02"
- order by incident counter in descending order
- and voilà, we get a list of offenders which persisted on attacking one of our systems.
The screenshot shows two different abusers with quite high a number of failed authentication attempts, possibly brute forcing. Watching the detailed view of the incident we can see how the requests were distributed into approximately a dozen requests per day.
Keep in mind this requires the prior creation of a trigger - this is how ours looks like for those interested:
- Facility: 2
- Severity: 6
- Program: postfix/smtpd
- Msg: lost connection after AUTH from %[%]
- Pattern: /\[(.*?)\]/
- Grouping: 1
- Capture: 1
- Description: lost connection after AUTH from unknown[zz.yy.xx.ww]
- Occurrence: 5
- Priority: 1
It is worth noting that for the period between 2014-03-13 and 2015-01-16 we have registered 99,966 unique abuser IPs. However, for the time period of one month (from 2015-01-16 until 2015-02-16) we have ~16.5k unique abuser IPs. Out of those 16.5k IPs, 1.800 IPs were also on the first subset.
What's more, combining the results from the Abuser module with a bit of shell scripting, pf and openbgp, one can automatically block offenders based on specific criteria. After we implemented this on our network we had a significant drop in our bandwidth consumption, received spam, smtp relay tests, along with the added value that our logs became a lot cleaner with less nuisances for us to worry about during our daily ops.
We hope you enjoyed this post; we will try to follow up with some more interesting statistics and details about the setup. If you have any questions you can reach me through Twitter @randomuid. If you require professional assistance with your Echofish, Syslog or OpenBSD servers feel free to contact us.
- That grumpy BSD guy: The Hail Mary Cloud And The Lessons Learned (I suggest you read the entire series).