If you work for any sort of provider (hosting, isp, etc), chances are that you’ve experienced a DoS/DDoS against a customer or internal system. If so, you know how frustrating it can be to track down the source of the attack unless you have an expensive platform such as from Arbor Networks.
NfSen is an invaluable open source tool that can be used to identify and alert on specific attacks. It does this by receiving Netflow data from a Cisco or Juniper router.
NfSen is available from: http://nfsen.sourceforge.net
NfDump (1.5.8 or greater) is available from: http://nfdump.sourceforge.net
NfDump (1.5.8 or greater) is available from: http://nfdump.sourceforge.net
Besides the above applications, you will also need the following:
- Perl > 5.6.0
- PHP > 4.1
- Mail::Header and Mail::Internet (from CPAN)
- RRDTool (for graph generation)
- a RTBH (Real Time Black Hole) system to drop the traffic as close to ingress as possible.
Once you have those installed, we can move onto plugins. By itself, NfSen gives you visibility into your network, but alone, it doesn’t alert you about anomalies. That’s where my Dynamic DDoS Detector (DDD) plugin comes in.
DDD can be downloaded from http://www.synacknetworks.com/ddd/ddd.zip. This zip file contains the Perl module (ddd.pm) and the MySQL structure.
DDD is loosely based on a plugin by John Fraizer named newddosdetect. Other than the concept, it’s been almost entirely rewritten, hence the new name.
The main reason for this plugin is that we recently deployed dedicated Netflow probes through our network. These take a feed from an inline tap and generate Netflow data, which is then relayed to our collectors. These probes work fairly well, however, they continue to see an attack even though it may have been mitigated via RTBH.
Since they continue to see the attack traffic, our collector continued to alert, causing a flood of emails until we manually edited the existing plugin and reloaded NfSen. Now, all we need to do is insert the attacker’s IP in our exclusion list.
To install, you just need to do the following steps:
1) Create the MySQL database from the ddd.mysql file. You can use the cli or phpMyAdmin.
2) Add any IP blocks you want to exclude (in CIDR format)
3) Adjust the filters, if needed
4) Drop the ddd.pm file into your NfSen plugins directory
5) Edit ddd.pm and change the following
### database login infomy $dsn = 'dbi:mysql:ddd:127.0.0.1:3306';my $user = 'dbuser';my $pass = 'dbpass'; |
# Specify the notification email(s).if ($DEBUG > 0){ $emails = "user\@example.com"; print "Emails = $emails\n\n";}else{ $emails = "distro-list\@example.com";} |
# beginning of the subject line for alertsmy $subject = "";if ($DEBUG > 0){ $subject = '[DEBUG] DDoS Alert:';}else{ $subject = "DDoS Early Warning:";} |
6) Edit nfsen.conf to include the plugin.
@plugins = ( # profile # module [ 'live', 'ddd'],); |
7) Reload NfSen
bin/nfsen reload |
Once done, NfSen will run the plugin every expire cycle (default is every 5 minutes). If you watch /var/log/messages, you should see it run with output similar to the following sanitized log.
Jan 11 01:45:15 hostname nfsen[9261]: 90 channels/alerts to profileJan 11 01:45:16 hostname nfsen[9261]: Update profile Compromised-Customers in group .Jan 11 01:45:17 hostname nfsen[9261]: Update profile botnet in group .Jan 11 01:45:18 hostname nfsen[9261]: Update profile live in group .Jan 11 01:45:18 hostname nfsen[9261]: Update profile AKAMAI in group BREAKDOWNJan 11 01:45:19 hostname nfsen[9261]: Update profile DNS-SERVERS in group BREAKDOWNJan 11 01:45:20 hostname nfsen[9261]: Update profile IRC-Probable-Targets in group BREAKDOWNJan 11 01:45:21 hostname nfsen[9261]: Update profile Private-IPs in group BREAKDOWNJan 11 01:45:22 hostname nfsen[9261]: Update profile Protocols-at-a-glance in group BREAKDOWNJan 11 01:45:23 hostname nfsen[9261]: Update profile Services-at-a-glance in group BREAKDOWNJan 11 01:45:23 hostname nfsen[9477]: Plugin Cycle: Time: 201001110140, Profile: live, Group: ., Module: PortTracker,Jan 11 01:45:25 hostname nfsen[9477]: Plugin Cycle: Time: 201001110140, Profile: live, Group: ., Module: ddd,Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/32 )Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm run: Checking Packets > 50K/flow-setJan 11 01:45:25 hostname nfsen[9477]: ddd.pm run: NF Filter: ((packets > 50000) AND NOT (proto tcp AND NOT (port 110 OR port 25)) AND NOT proto esp) AND (NOT net A.B.C.D/32 )Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm end: Packets > 50K/flow-setJan 11 01:45:25 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------Jan 11 01:45:25 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking UDP Packets > 80K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto udp and not out if 0 AND (NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for UDP Packets > 80K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: UDP Packets > 80K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/28 AND NOT net A.B.C.D/32)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking TCP Packets > 100K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto tcp and not out if 0 AND (NOT net A.B.C.D/28 AND NOT net A.B.C.D/32)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for TCP Packets > 100K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: TCP Packets > 100K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions:Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking ICMP Packets > 50K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto icmp and not out if 0Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for ICMP Packets > 50K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: ICMP Packets > 50K/destJan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/28 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking TCP SYN StormJan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto tcp and packets eq 1 and flags S and not flags A and bpp 48 and not out if 0 AND (NOT net A.B.C.D/28 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/32)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: No matching flows found for TCP SYN StormJan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: TCP SYN StormJan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm exclusions: (NOT net A.B.C.D/23 AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: Checking UDP Packets > 70K / IPJan 11 01:45:26 hostname nfsen[9477]: ddd.pm run: NF Filter: proto udp and not out if 0 AND (NOT net A.B.C.D/23 AND NOT net A.B.C.D/30 AND NOT net A.B.C.D/32 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/22 AND NOT net A.B.C.D/24 AND NOT net A.B.C.D/23 AND NOT net A.B.C.D/23)Jan 11 01:45:26 hostname nfsen[9477]: ddd.pm end: UDP Packets > 70K / IPJan 11 01:45:26 hostname nfsen[9477]: ddd.pm -------------------------------------------------------------Jan 11 01:45:26 hostname nfsen[9261]: Run expire at Mon Jan 11 01:45:00 2010Jan 11 01:45:26 hostname nfsen[9261]: Expire profile Compromised-Customers group . low water mark: 90%%Jan 11 01:45:27 hostname nfsen[9261]: Expire profile botnet group . low water mark: 90%%Jan 11 01:45:27 hostname nfsen[9261]: Expire profile live group . low water mark: 90%%Jan 11 01:45:27 hostname nfsen[9261]: Expire profile AKAMAI group BREAKDOWN low water mark: 90%%Jan 11 01:45:27 hostname nfsen[9261]: Expire profile IRC-Probable-Targets group BREAKDOWN low water mark: 90%%Jan 11 01:45:27 hostname nfsen[9261]: End expire at Mon Jan 11 01:45:00 2010 |
You shouldn’t see any errors here. If you do, most likely the IPs in your exclusion table aren’t in CIDR format. Even a host needs to be in that format (i.e. 192.168.1.1/32).
When an attack is detected, we send an email to the address listed in the plugin with the body composed of the relevant details. The following is an alert from last year indicating a UDP attack against our customer on port 113.
Date flow start Duration Proto Src IP Addr:Port Dst IP Addr:Port Flags Tos Packets Bytes pps bps Bpp Flows2009-11-16 03:07:15.269 133.556 UDP AA.AAA.AAA.AAA:35198 -> XX.XXX.XX.XX:113 ...... 0 187678 8.6 M 1405 517127 46 2Summary: total flows: 2, total bytes: 8.6 M, total packets: 187678, avg bps: 517127, avg pps: 1405, avg bpp: 46Time window: 2009-11-16 03:03:28 - 2009-11-16 03:10:06Total flows processed: 803235, Blocks skipped: 0, Bytes read: 43449712Sys: 0.174s flows/second: 4590622.6 Wall: 0.170s flows/second: 4724217.0 |
Additionally, the alert is stored in the database for future queries.
With this information, we can then use NfSen to pull up details for the suspected attack. We’ll want to do a query such as “host XX.XXX.XX.XX”, where xx is the destination of the attack (our customer). This allows us to see all sources because some attackers could be below our thresholds.
Chances are that if we go back to the beginning of the attack (or a few cycles before), we’ll find IRC traffic directly preceding the attack. IRC is typically on ports 6660 through 6669, or port 7000).
This usually indicates one of two things. 1) Someone was talking smack on IRC and got slapped, or 2) This customer got infected by an old piece of malware and when it tried to login to the command and control channel, it had an old password, resulting in an automatic attack.
In either case, we mitigate the attack in our RTBH system, apply an IRC ACL to the customer interface, and send a ticket to our security group to contact the customer so they can clean the infection.
Some other ideas we’ve been kicking around for future versions of DDD is the ability to automatically inject a prefix into our mitigation VRF so that our Arbor implementation can scrub the traffic instead of manually blackholing it.
Also, version 1.5 will include the ability to auto expire attack exclusions, in addition to permanent exclusions. You don’t want to permanently exclude an attackers or attack destination because you’ll be blind to future attacks. Additionally, NfSen can only accept filters that are so long.
This is a fairly simple, yet effective way to identify attacks on your network. Since NfSen is always looking 5 minutes into the past, you won’t be able to prevent an outage, but you can shorten the duration, lessening the impact.
Be cautious what you blackhole with this. Sure, most alerts are true attacks, but some can just be anomalous, yet valid traffic. Things like breaking news could possibly trigger an alert if a lot of users suddenly start visiting a small number of web sites. Be sure you know what an IP is associated with before null routing it. You don’t want to create an outage. You also probably don’t want to null route a customer unless an attack is affecting large portions of the network and is absolutely necessary.
You should also get buy-in from management before implementing new policies, such as ACLs to block IRC traffic, or even RTBH.
