DDoS auto-mitigation best practices (for eyeball networks)
Could the community share some DDoS auto-mitigation best practices for eyeball networks, where the target is a residential broadband subscriber? I'm not asking so much about the customer communication as much as configuration of any thresholds or settings, such as: - minimum traffic volume before responding (for volumetric attacks) - minimum time to wait before responding - filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)? - time before mitigation is automatically removed - and if the attack should recur shortly thereafter, time to respond and remove again - use of an upstream provider(s) mitigation services versus one's own mitigation tools - network placement of mitigation (presumably upstream as possible) - and anything else I ask about best practice for broadband subscribers on eyeball networks because it's different environment than data center and hosting environments or when one's network is being used to DDoS a target. Regards, Frank
How does he/she become target? How does IP address gets exposed? I guess simplest way is to reboot modem and hope to get new ip (or call n request) Mehmet
On Sep 19, 2015, at 12:54, Frank Bulk <frnkblk@iname.com> wrote:
Could the community share some DDoS auto-mitigation best practices for eyeball networks, where the target is a residential broadband subscriber? I'm not asking so much about the customer communication as much as configuration of any thresholds or settings, such as: - minimum traffic volume before responding (for volumetric attacks) - minimum time to wait before responding - filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)? - time before mitigation is automatically removed - and if the attack should recur shortly thereafter, time to respond and remove again - use of an upstream provider(s) mitigation services versus one's own mitigation tools - network placement of mitigation (presumably upstream as possible) - and anything else
I ask about best practice for broadband subscribers on eyeball networks because it's different environment than data center and hosting environments or when one's network is being used to DDoS a target.
Regards,
Frank
Often it's an argument in some sort of online game or a poor loser. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Mehmet Akcin" <mehmet@akcin.net> To: "Frank Bulk" <frnkblk@iname.com> Cc: nanog@nanog.org Sent: Saturday, September 19, 2015 3:09:47 PM Subject: Re: DDoS auto-mitigation best practices (for eyeball networks) How does he/she become target? How does IP address gets exposed? I guess simplest way is to reboot modem and hope to get new ip (or call n request) Mehmet
On Sep 19, 2015, at 12:54, Frank Bulk <frnkblk@iname.com> wrote:
Could the community share some DDoS auto-mitigation best practices for eyeball networks, where the target is a residential broadband subscriber? I'm not asking so much about the customer communication as much as configuration of any thresholds or settings, such as: - minimum traffic volume before responding (for volumetric attacks) - minimum time to wait before responding - filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)? - time before mitigation is automatically removed - and if the attack should recur shortly thereafter, time to respond and remove again - use of an upstream provider(s) mitigation services versus one's own mitigation tools - network placement of mitigation (presumably upstream as possible) - and anything else
I ask about best practice for broadband subscribers on eyeball networks because it's different environment than data center and hosting environments or when one's network is being used to DDoS a target.
Regards,
Frank
On Sep 19, 2015, at 4:09 PM, Mehmet Akcin <mehmet@akcin.net> wrote:
How does he/she become target? How does IP address gets exposed?
Call of Duty. No I am not kidding. The majority of the DDOS attacks we see targeted at Residential customers, are because said Customer was running their mouth / being an A**Hole in Call of Duty, and they pissed off someone with access to a botnet, etc.. The other is people gambling for real money and they start to win, so someone else knocks they offline so they won't lose. As for how they get their IP, not really sure as I suck as those games, but I've heard it is pretty trivial to do so. -Patrick
99% of the attacks we see are gaming related -- somehow the other players know the IP and then attack our customer for an advantage in the game, or retribution. Most DHCP servers (correctly) give the same IP address if the CPE is rebooted. Ours are one of those. =) Frank -----Original Message----- From: Mehmet Akcin [mailto:mehmet@akcin.net] Sent: Saturday, September 19, 2015 3:10 PM To: Frank Bulk <frnkblk@iname.com> Cc: nanog@nanog.org Subject: Re: DDoS auto-mitigation best practices (for eyeball networks) How does he/she become target? How does IP address gets exposed? I guess simplest way is to reboot modem and hope to get new ip (or call n request) Mehmet
On Sep 19, 2015, at 12:54, Frank Bulk <frnkblk@iname.com> wrote:
Could the community share some DDoS auto-mitigation best practices for eyeball networks, where the target is a residential broadband subscriber? I'm not asking so much about the customer communication as much as configuration of any thresholds or settings, such as: - minimum traffic volume before responding (for volumetric attacks) - minimum time to wait before responding - filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)? - time before mitigation is automatically removed - and if the attack should recur shortly thereafter, time to respond and remove again - use of an upstream provider(s) mitigation services versus one's own mitigation tools - network placement of mitigation (presumably upstream as possible) - and anything else
I ask about best practice for broadband subscribers on eyeball networks because it's different environment than data center and hosting environments or when one's network is being used to DDoS a target.
Regards,
Frank
Most video games utilize peer-to-peer traffic (which is why many require port forwarding/UPnP), so the attacker has the IP addresses of all of their peers in their firewall logs. There are even 'gaming routers' that specialize in gaming this peer-to-peer system for competitive advantages, such as specifically blocking the IPs of players you don't want to play against: https://netduma.com/why/for-gamers/ Once an attacker has identified his target, getting the IP is as simple as joining/being in an online game with that player. On Mon, Sep 21, 2015 at 5:00 AM, <frnkblk@iname.com> wrote:
99% of the attacks we see are gaming related -- somehow the other players know the IP and then attack our customer for an advantage in the game, or retribution.
Most DHCP servers (correctly) give the same IP address if the CPE is rebooted. Ours are one of those. =)
Frank
-----Original Message----- From: Mehmet Akcin [mailto:mehmet@akcin.net] Sent: Saturday, September 19, 2015 3:10 PM To: Frank Bulk <frnkblk@iname.com> Cc: nanog@nanog.org Subject: Re: DDoS auto-mitigation best practices (for eyeball networks)
How does he/she become target? How does IP address gets exposed?
I guess simplest way is to reboot modem and hope to get new ip (or call n request)
Mehmet
On Sep 19, 2015, at 12:54, Frank Bulk <frnkblk@iname.com> wrote:
Could the community share some DDoS auto-mitigation best practices for eyeball networks, where the target is a residential broadband subscriber? I'm not asking so much about the customer communication as much as configuration of any thresholds or settings, such as: - minimum traffic volume before responding (for volumetric attacks) - minimum time to wait before responding - filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)? - time before mitigation is automatically removed - and if the attack should recur shortly thereafter, time to respond and remove again - use of an upstream provider(s) mitigation services versus one's own mitigation tools - network placement of mitigation (presumably upstream as possible) - and anything else
I ask about best practice for broadband subscribers on eyeball networks because it's different environment than data center and hosting environments or when one's network is being used to DDoS a target.
Regards,
Frank
----- Original Message ----- From: Frank Bulk <frnkblk@iname.com> To: nanog@nanog.org Cc: Sent: Saturday, September 19, 2015 12:54 PM Subject: DDoS auto-mitigation best practices (for eyeball networks) Could the community share some DDoS auto-mitigation best practices for eyeball networks, where the target is a residential broadband subscriber? I'm not asking so much about the customer communication as much as configuration of any thresholds or settings, such as: - minimum traffic volume before responding (for volumetric attacks) - minimum time to wait before responding - filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)? - time before mitigation is automatically removed - and if the attack should recur shortly thereafter, time to respond and remove again - use of an upstream provider(s) mitigation services versus one's own mitigation tools - network placement of mitigation (presumably upstream as possible) - and anything else I ask about best practice for broadband subscribers on eyeball networks because it's different environment than data center and hosting environments or when one's network is being used to DDoS a target. Regards, Frank Frank, If you figure out a way to protect residential-BB-clients, I would love to know! Regards, ./Randy
On 20 Sep 2015, at 2:54, Frank Bulk wrote:
- minimum traffic volume before responding (for volumetric attacks) - minimum time to wait before responding
These are situationally-specific.
- filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)?
If one has flowspec capabilities, mitigating only the attack traffic is preferred, if at all possible. If one only has S/RTBH, blocking the attack sources is preferred, if at all possible. Some operators resort to D/RTBH (thereby completing the DDoS) because they don't have layer-4 granularity in their mitigation tools (e.g., flowspec), or even if they have S/RTBH, the number of sources can be daunting in relation to their hardware capabilities. I'm not a big fan of this approach, as it completes the DDoS on the victim, but understand why some operators do this.
- time before mitigation is automatically removed
This is situationally-specific.
- and if the attack should recur shortly thereafter, time to respond and remove again
This is situationally-specific. Some operators keep track of the frequency, and will 'fire' customers who're attack-magnets. I'm not a big fan of this approach, either, but understand why some operators do it (simple economics).
- use of an upstream provider(s) mitigation services versus one's own mitigation tools
This is situationally-specific. Some operators take advantage of these services when on-offer.
- network placement of mitigation (presumably upstream as possible)
Peering/transit edges, generally. Some do it on upstream networks which provide such a service, as you indicated.
- and anything else
There's no one-size-fits-all answer for most of these questions; your capabilities and tolerances may be very different from those of another operator. Network infrastructure-based techniques, such as S/RTBH, D/RTBH, and flowspec are generally what's used in these situations, in addition to QoS (see below). A great deal (not all) of the operationally-significant attacks against individual broadband users these days seem to be UDP reflection/amplification attacks. Policing non-timesync ntp at one's edges via QoS is pretty straightforward and minimizes the risk of overblocking, as there's a packet-size to use as an additional classifier beyond protocol/port. Some operators, as Jared Mauch has mentioned here previously, are policing common UDP port-pairs used in other flavors of UDP reflection/amplification attacks at their edges, as well (Jared is pleased with the results on his network). It might be possible to get this sort of thing instituted on one's upstreams. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On 09/19/15 at 02:54pm, Frank Bulk wrote:
Could the community share some DDoS auto-mitigation best practices for eyeball networks, where the target is a residential broadband subscriber?
o kie dough kie
I'm not asking so much about the customer communication as much as configuration of any thresholds or settings, such as: - minimum traffic volume before responding (for volumetric attacks)
i prefer zero tolerance ... i tarpit all incoming tco-based attacks and probes that was not allowed incoming tcp traffic to port 25 or port 80 or port blah example iptables rules ... linux and iptables + tarpits is free # IPtables-BlackList.net/Howto - ingress filters - allow established - check for blacklist - limit udp and icmp reply ( tough problem to solve ) - allow to port 80 ( keep webserver separate from dns, smtp, etc ) - tarpit all new tcp incoming connections - drop all other new incoming connections - there is no need log millions of ddos attack pacekets per second unless you want to fill up your disk which helps the ddos attacks to be a successful attack - for icmp and udp ... you will need your ISP to help block it limiting incoming icmp/udp is sorta pointless since those packets already come down the wire however, you still do NOT want to respond to those packets either so you will have to limit to just a handful per second, little more per hour, and higher limit per day for icmp ... turn off broadcast ping responses on all devices for udp ... make sure the apps are properly configured dns, snmp, ntp, nfs, x11, etc uses udp your dns servers might need to be accessible from outside all other udp-based servers should be internal only - to protect against arp-based attacks .... build/patch/configure your hardware/routers/switches properly - install monitoring tools to watch for whatever you're paranoid about - man-in-the-middle .. trivial to detect and prevent - sniffers ( hard to detect )
- minimum time to wait before responding
zero wait ...
- filter percentage: 100% of the traffic toward target (or if volumetric, just a certain percentage)?
you will always, 100% fail volumetric attacks
- time before mitigation is automatically removed
you can have iptables remove a particular ddos attacker automatically or manually i prefer manually so i can see what it's doing
- and if the attack should recur shortly thereafter, time to respond and remove again
zero wait time .. zero tolerance per example iptables rules above
- use of an upstream provider(s) mitigation services versus one's own mitigation tools
i haven't found too many ISP willing to allow customers to put a customer firewall in their facility just before it comes down to the wire to customers bldg this is required if customers want to properly mitigate icmp-based and udp-based ddos attacks
- network placement of mitigation (presumably upstream as possible) - and anything else
mitigation solutions should be a gateway firewall and host-based mitigation if you can install another firewall at the ISP, thats good too and you still need a gateway firewall and the host based firewall
I ask about best practice for broadband subscribers on eyeball networks because it's different environment than data center and hosting environments or when one's network is being used to DDoS a target.
add corp environment, hospitality environment, govt environment, etc etc to the list too - free wifi, hotel based wifi or hardwire is probably the easiest way to send the unsuspecting victim home with a trojan that will phone home ( the attacker ) when the victim plugs the cracked box into the secure corp network nah.... ddos attacks are ddos attacks ... usually harmless ... it probably doesn't matter to the attackers what they're attacking you are constantly under 24x7x365 low level ddos attacks if you are being targeted by somebody that wants to get you, you'd have a problem if they're better at attacking than you are at defending your servers ... they're done if they have a bigger budget to pay for all the necessary bandwidth needed to take your servers offline - if you know who they are, call the ISP and the cops ----- other "basic best practices" - have a good security policy ... even if just for yourself hide the laptop in your trunk using a brown bag and NOT an obvious laptop bag - always use encrypted services... never clear text - use ssh, openssl, smtps, pop3s, imap3s - dozens of other best practices security rules - always have a incremental daily backup that is kept for months - always have a hot swap backup just in case .... etc .. etc ... ---- you should also keep track of who is attacking your servers so that law enforcement can followup if needed you should also know which src address might be spoofed and which ddos attackers are using their real src ip tracing the originating source of spoofed address requires the help of the various upstream ISP magic pixie dust alvin # # DDoS-Mitigator.com # IPtables-BlackList.net/Howto #
participants (9)
-
alvin nanog -
Chase Christian -
Frank Bulk -
frnkblk＠iname.com
-
Mehmet Akcin -
Mike Hammett -
Patrick Muldoon -
Randy -
Roland Dobbins