automatic rtbh trigger using flow data
Hi, does anyone know how to use flow data to trigger a rtbh (remotely triggered blackhole) route using bgp ? .I'm thinking we could use quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack. Btw, I already have nfsen running and we receive real-time alters of various types of attacks, high volume, high ports, etc. and then we telnet into a cisco trigger router and drop a few lines of code into it and then bgp does the rest within seconds, the upstream providers learn of this route via communities and they rtbh it in their cloud, BUT, I would like my alerts to do this automatically. that would be very nice. Any guidance would be appreciated. -Aaron
fastnetmon does exactly what you’re looking for. https://fastnetmon.com/ <https://fastnetmon.com/> there is also an open source version https://github.com/pavel-odintsov/fastnetmon <https://github.com/pavel-odintsov/fastnetmon> my best —vicente
On Aug 30, 2018, at 12:52 PM, Aaron Gould <aaron1@gvtc.com> wrote:
Hi, does anyone know how to use flow data to trigger a rtbh (remotely triggered blackhole) route using bgp ? …I’m thinking we could use quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack.
Btw, I already have nfsen running and we receive real-time alters of various types of attacks, high volume, high ports, etc… and then we telnet into a cisco trigger router and drop a few lines of code into it and then bgp does the rest within seconds, the upstream providers learn of this route via communities and they rtbh it in their cloud, BUT, I would like my alerts to do this automatically… that would be very nice. Any guidance would be appreciated.
-Aaron
There are software that combine your needs altogether. I'm sure there are others. WANGuard from Andrisoft (https://www.andrisoft.com/software/wanguard) Fastnetmon (https://fastnetmon.com/) From: NANOG <nanog-bounces@nanog.org> On Behalf Of Aaron Gould Sent: Thursday, August 30, 2018 12:53 PM To: Nanog@nanog.org Subject: automatic rtbh trigger using flow data Hi, does anyone know how to use flow data to trigger a rtbh (remotely triggered blackhole) route using bgp ? ...I'm thinking we could use quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack. Btw, I already have nfsen running and we receive real-time alters of various types of attacks, high volume, high ports, etc... and then we telnet into a cisco trigger router and drop a few lines of code into it and then bgp does the rest within seconds, the upstream providers learn of this route via communities and they rtbh it in their cloud, BUT, I would like my alerts to do this automatically... that would be very nice. Any guidance would be appreciated. -Aaron
Wow, 4 replies for fastnetmon, thanks Ryan, Vincente, Job and Kushal I'll look into it -Aaron From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Aaron Gould Sent: Thursday, August 30, 2018 2:53 PM To: Nanog@nanog.org Subject: automatic rtbh trigger using flow data Hi, does anyone know how to use flow data to trigger a rtbh (remotely triggered blackhole) route using bgp ? .I'm thinking we could use quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack. Btw, I already have nfsen running and we receive real-time alters of various types of attacks, high volume, high ports, etc. and then we telnet into a cisco trigger router and drop a few lines of code into it and then bgp does the rest within seconds, the upstream providers learn of this route via communities and they rtbh it in their cloud, BUT, I would like my alerts to do this automatically. that would be very nice. Any guidance would be appreciated. -Aaron
Aaron Gould wrote : Hi, does anyone know how to use flow data to trigger a rtbh (remotely triggered blackhole) route using bgp ? ...I'm thinking we could use quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack.
Look at Exabgp : https://github.com/Exa-Networks/exabgp That's what I use in here : https://arneill-py.sacramento.ca.us/cbbc/ to inject the prefixes in BGP. I block the attacker's addresses, not the victim but if you are willing to write your own scripts it does the job. Michel. TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
Aaron Gould wrote : Hi, does anyone know how to use flow data to trigger a rtbh (remotely
Thanks, but what if the attacker is many... like thousands ? ...isn't that typically what we see, is tons and tons of sources (hence distributed....dos) ? -Aaron -----Original Message----- From: Michel Py [mailto:michel.py@tsisemi.com] Sent: Thursday, August 30, 2018 3:17 PM To: Aaron Gould; Nanog@nanog.org Subject: RE: automatic rtbh trigger using flow data triggered blackhole) route using bgp ? ...I'm thinking we could use
quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack.
Look at Exabgp : https://github.com/Exa-Networks/exabgp That's what I use in here : https://arneill-py.sacramento.ca.us/cbbc/ to inject the prefixes in BGP. I block the attacker's addresses, not the victim but if you are willing to write your own scripts it does the job. Michel. TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
Aaron Gould wrote : Hi, does anyone know how to use flow data to trigger a rtbh (remotely
Exactly Aaron. No provider will allow a customer to null route a source IP address. I could only assume that a null route on Michel's network is tanking the packets at their edge to 192.0.2.1 (discard/null0). -- Ryan Hamel Senior Support Engineer ryan.hamel@quadranet.com | +1 (888) 578-2372 QuadraNet Enterprises, LLC. | Dedicated Servers, Colocation, Cloud -----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of Aaron Gould Sent: Thursday, August 30, 2018 1:38 PM To: 'Michel Py' <michel.py@tsisemi.com>; Nanog@nanog.org Subject: RE: automatic rtbh trigger using flow data Thanks, but what if the attacker is many... like thousands ? ...isn't that typically what we see, is tons and tons of sources (hence distributed....dos) ? -Aaron -----Original Message----- From: Michel Py [mailto:michel.py@tsisemi.com] Sent: Thursday, August 30, 2018 3:17 PM To: Aaron Gould; Nanog@nanog.org Subject: RE: automatic rtbh trigger using flow data triggered blackhole) route using bgp ? ...I'm thinking we could use
quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack.
Look at Exabgp : https://github.com/Exa-Networks/exabgp That's what I use in here : https://arneill-py.sacramento.ca.us/cbbc/ to inject the prefixes in BGP. I block the attacker's addresses, not the victim but if you are willing to write your own scripts it does the job. Michel. TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
Aaron Gould wrote : Thanks, but what if the attacker is many... like thousands ? ...isn't that typically what we see, is tons and tons of sources (hence distributed....dos) ?
At this very moment I blacklist ~ 56,000 individual /32s and historically it has been up to 135,000 at times. It's not a problem for most routers, unless you're on one of these old clunkers with un-upgradable TCAM and a full feed (if you are, you don't have much time left anyway).
Ryan Hamel wrote : Exactly Aaron. No provider will allow a customer to null route a source IP address.
Yes, unless you have your own router on their side of the link and pay for it, or have your own VRF on their router which is not going to be cheap either.
I could only assume that a null route on Michel's network is tanking the packets at their edge to 192.0.2.1 (discard/null0).
Aaron Gould wrote : Hi, does anyone know how to use flow data to trigger a rtbh (remotely
Correct, and I clearly understand its limitations, paragraph below taken from https://arneill-py.sacramento.ca.us/cbbc/ There indeed is a value in blacklisting the IP address of the host being attacked and feed that with the appropriate community to the upstream that will accept it as it is part of your own space. You sacrifice one host to save the bandwidth to the rest. That being said, if the DDOS targets your entire IP range, none of these will help. I have to withstand DDOS attacks all the time, can the CBBC feed help ? It depends on the type of attack; the CBBC feed is not designed as DDOS mitigation tool. There is no such thing as a free lunch : your ISP will not take the full CBBC feed for free when they can make you pay big bucks for their own one. The CBBC does not prevent the DDOS attack to get to you, it may help with attacks that are based on PPS, not raw bandwidth. What the CBBC does is to block the offending traffic at the router level, so it is blocked before it even reaches your server / firewall. However, the CBBC does not prevent the DDOS traffic from coming to you, so if you have a slow connection to the Internet and the DDOS sends more bandwidth than you have, you still are down. However, if the DDOS is based not on bandwidth but on a higher-level protocol such as DNS or HTTPS, it helps by taking the load off the server. Michel. -Aaron -----Original Message----- From: Michel Py [mailto:michel.py@tsisemi.com] Sent: Thursday, August 30, 2018 3:17 PM To: Aaron Gould; Nanog@nanog.org Subject: RE: automatic rtbh trigger using flow data triggered blackhole) route using bgp ? ...I'm thinking we could use
quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack.
Look at Exabgp : https://github.com/Exa-Networks/exabgp That's what I use in here : https://arneill-py.sacramento.ca.us/cbbc/ to inject the prefixes in BGP. I block the attacker's addresses, not the victim but if you are willing to write your own scripts it does the job. Michel. TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
Michel Py wrote:
Aaron Gould wrote : Hi, does anyone know how to use flow data to trigger a rtbh (remotely triggered blackhole) route using bgp ? ...I'm thinking we could use quagga or a script of some sort to interact with a router to advertise to bgp the /32 host route of the victim under attack. Look at Exabgp : https://github.com/Exa-Networks/exabgp That's what I use in here : https://arneill-py.sacramento.ca.us/cbbc/ to inject the prefixes in BGP. I block the attacker's addresses, not the victim but if you are willing to write your own scripts it does the job.
Michel.
I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity. Using flow data, that sounds like an interesting direction to take this into, so thank you! Joe
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga
I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity.
I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you!
The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close. Michel. TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, if so I'd like to see the list too Even so, this would not stop the attacks from hitting my front door, my side of my Internet uplink...when paying for a 30 gigs CIR and paying double for megabits per second over that, up to the ceiling of 100 gig every bit that hits my front door over 30 gig would cost me extra, remotely triggering based on my victim IP address inside my network would be my solution to saving money But stopping the attack even on my side of my Internet up like would at least stop it from proliferating throughout my internal network which is also costing me when it affects cell towers, etc. Aaron On Aug 30, 2018, at 6:43 PM, Michel Py <michel.py@tsisemi.com> wrote:
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga
I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity.
I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you!
The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close.
Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
On 31 Aug 2018, at 6:47, Aaron Gould wrote:
I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, i
Using S/RTBH to drop attack sources has been a valid and useful mitigation tactic for close to 20 years. Any kind of modern router scales up to large numbers of sources; and note that S/RTBH isn't limited to /32s. It's discussed in this .pdf preso: <https://app.box.com/s/xznjloitly2apixr5xge> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Fri 2018-Aug-31 06:59:29 +0700, Roland Dobbins <rdobbins@arbor.net> wrote:
On 31 Aug 2018, at 6:47, Aaron Gould wrote:
I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, i
Using S/RTBH to drop attack sources has been a valid and useful mitigation tactic for close to 20 years. Any kind of modern router scales up to large numbers of sources; and note that S/RTBH isn't limited to /32s.
It's discussed in this .pdf preso:
I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream. _keeps dreaming_ -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
On 31 Aug 2018, at 22:15, Hugo Slabbert wrote:
I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream.
<https://pc.nanog.org/static/published/meetings/NANOG71/1447/20171003_Levy_Operationalizing_Isp_v2.pdf> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
fre. 31. aug. 2018 17.16 skrev Hugo Slabbert <hugo@slabnet.com>:
I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream.
_keeps dreaming_
We just need a signal to drop UDP for a prefix. The same as RTBH but only for UDP. This would prevent all volumetric attacks without the end user being cut off completely. Besides from some games, VPN and VoIP, they would have an almost completely normal internet experience. DNS would go through the ISP servers and only be affected if the user is using a third party service. Regards Baldur
No ISP is in the business of filtering traffic unless the client pays the hefty fee since someone still has to tank the attack. I also don’t think there is destination prefix IP filtering in flowspec, which could seriously cause problems. From: NANOG <nanog-bounces@nanog.org> On Behalf Of Baldur Norddahl Sent: Saturday, September 01, 2018 5:18 PM To: nanog@nanog.org Subject: Re: automatic rtbh trigger using flow data fre. 31. aug. 2018 17.16 skrev Hugo Slabbert <hugo@slabnet.com<mailto:hugo@slabnet.com>>: I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream. _keeps dreaming_ We just need a signal to drop UDP for a prefix. The same as RTBH but only for UDP. This would prevent all volumetric attacks without the end user being cut off completely. Besides from some games, VPN and VoIP, they would have an almost completely normal internet experience. DNS would go through the ISP servers and only be affected if the user is using a third party service. Regards Baldur
On Sun 2018-Sep-02 00:39:40 +0000, Ryan Hamel <Ryan.Hamel@quadranet.com> wrote:
No ISP is in the business of filtering traffic unless the client pays the hefty fee since someone still has to tank the attack.
If I can tag an RTBH community on a /32, what's the additional lost revenue in letting me be more granular and get down to the specific flows I want dropped? "drop all traffic to x/32" would drop *more* traffic than "drop any traffic from address y to x/32, protocol TCP, port n".
I also don’t think there is destination prefix IP filtering in flowspec, which could seriously cause problems.
What now? Unless I'm misunderstanding what you're saying, it's right in the spec[1]: A flow specification NLRI must be validated such that it is considered feasible if and only if: a) The originator of the flow specification matches the originator of the best-match unicast route for the destination prefix embedded in the flow specification. b) There are no more specific unicast routes, when compared with the flow destination prefix, that have been received from a different neighboring AS than the best-match unicast route, which has been determined in step a). By originator of a BGP route, we mean either the BGP originator path attribute, as used by route reflection, or the transport address of the BGP peer, if this path attribute is not present. The underlying concept is that the neighboring AS that advertises the best unicast route for a destination is allowed to advertise flow- spec information that conveys a more or equally specific destination prefix. Thus, as long as there are no more specific unicast routes, received from a different neighboring AS, which would be affected by that filtering rule. The neighboring AS is the immediate destination of the traffic described by the flow specification. If it requests these flows to be dropped, that request can be honored without concern that it represents a denial of service in itself. Supposedly, the traffic is being dropped by the downstream autonomous system, and there is no added value in carrying the traffic to it. -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal [1] https://tools.ietf.org/html/rfc5575
From: NANOG <nanog-bounces@nanog.org> On Behalf Of Baldur Norddahl Sent: Saturday, September 01, 2018 5:18 PM To: nanog@nanog.org Subject: Re: automatic rtbh trigger using flow data
fre. 31. aug. 2018 17.16 skrev Hugo Slabbert <hugo@slabnet.com<mailto:hugo@slabnet.com>>:
I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream.
_keeps dreaming_
We just need a signal to drop UDP for a prefix. The same as RTBH but only for UDP. This would prevent all volumetric attacks without the end user being cut off completely.
Besides from some games, VPN and VoIP, they would have an almost completely normal internet experience. DNS would go through the ISP servers and only be affected if the user is using a third party service.
Regards
Baldur
Roland Dobbins wrote : I'm well aware of what's mentioned in the Arbor report, thanks!
I would not have guessed :P
Ryan Hamel wrote : No ISP is in the business of filtering traffic unless the client pays the hefty fee since someone still has to tank the atack.
I agree. In the end, it tends to favor who has the biggest one. I meant the biggest bandwidth, of course. I the foreseeable future, no blacklist system is going to replace what Arbor and consorts can provide : a pipe big enough to either route the DDOS attack to null0 or even better route it to somewhere it can be analyzed further. Michel. TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
This is not true. Some of our transits do RTBH for free. For example Cogent. They will not do FlowSpec. Maybe their equipment can not do it or for some other reason. However RTBH is a simple routing hack that can be implemented on any router. The traffic is dropped right at the edge and is never transported on the transit provider network. In that sense it also protects the transit network. RTBH only for UDP would also be a very simple hack on many routers. It might not be FlowSpec, but it may have most of the benefit, in a much simplified way. Regards Baldur søn. 2. sep. 2018 02.39 skrev Ryan Hamel <Ryan.Hamel@quadranet.com>:
No ISP is in the business of filtering traffic unless the client pays the hefty fee since someone still has to tank the attack.
I also don’t think there is destination prefix IP filtering in flowspec, which could seriously cause problems.
*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Baldur Norddahl *Sent:* Saturday, September 01, 2018 5:18 PM *To:* nanog@nanog.org *Subject:* Re: automatic rtbh trigger using flow data
fre. 31. aug. 2018 17.16 skrev Hugo Slabbert <hugo@slabnet.com>:
I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream.
_keeps dreaming_
We just need a signal to drop UDP for a prefix. The same as RTBH but only for UDP. This would prevent all volumetric attacks without the end user being cut off completely.
Besides from some games, VPN and VoIP, they would have an almost completely normal internet experience. DNS would go through the ISP servers and only be affected if the user is using a third party service.
Regards
Baldur
Baldur, Modifying the routing table with a next-hop change from a community, is different than having a line card filtering packets at layer 4, of course most if not all carriers will support it. Instead of doing normal TCAM route lookups, you’re getting into packet inspection territory, which is something completely different. Just quickly reading the ASR 9K documentation, it can only support 3K rules per system. Juniper – 8K, Alcatel-Lucent – 512. That’s pretty low considering I can put many /32s into a routing table very easily and without hassle. As I said before, no ISP is going to offer such filtering services for free when DDoS mitigation is a cash cow. Ryan Hamel From: NANOG <nanog-bounces@nanog.org> On Behalf Of Baldur Norddahl Sent: Sunday, September 02, 2018 1:42 AM To: nanog@nanog.org Subject: Re: automatic rtbh trigger using flow data This is not true. Some of our transits do RTBH for free. For example Cogent. They will not do FlowSpec. Maybe their equipment can not do it or for some other reason. However RTBH is a simple routing hack that can be implemented on any router. The traffic is dropped right at the edge and is never transported on the transit provider network. In that sense it also protects the transit network. RTBH only for UDP would also be a very simple hack on many routers. It might not be FlowSpec, but it may have most of the benefit, in a much simplified way. Regards Baldur søn. 2. sep. 2018 02.39 skrev Ryan Hamel <Ryan.Hamel@quadranet.com<mailto:Ryan.Hamel@quadranet.com>>: No ISP is in the business of filtering traffic unless the client pays the hefty fee since someone still has to tank the attack. I also don’t think there is destination prefix IP filtering in flowspec, which could seriously cause problems. From: NANOG <nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>> On Behalf Of Baldur Norddahl Sent: Saturday, September 01, 2018 5:18 PM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: automatic rtbh trigger using flow data fre. 31. aug. 2018 17.16 skrev Hugo Slabbert <hugo@slabnet.com<mailto:hugo@slabnet.com>>: I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream. _keeps dreaming_ We just need a signal to drop UDP for a prefix. The same as RTBH but only for UDP. This would prevent all volumetric attacks without the end user being cut off completely. Besides from some games, VPN and VoIP, they would have an almost completely normal internet experience. DNS would go through the ISP servers and only be affected if the user is using a third party service. Regards Baldur
I would redirect the packet to a VRF with one global drop UDP ACL. That scales perfectly. There is probably many ways to implement such a feature. søn. 2. sep. 2018 11.07 skrev Ryan Hamel <Ryan.Hamel@quadranet.com>:
Baldur,
Modifying the routing table with a next-hop change from a community, is different than having a line card filtering packets at layer 4, of course most if not all carriers will support it. Instead of doing normal TCAM route lookups, you’re getting into packet inspection territory, which is something completely different.
Just quickly reading the ASR 9K documentation, it can only support 3K rules per system. Juniper – 8K, Alcatel-Lucent – 512. That’s pretty low considering I can put many /32s into a routing table very easily and without hassle.
As I said before, no ISP is going to offer such filtering services for free when DDoS mitigation is a cash cow.
Ryan Hamel
*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Baldur Norddahl *Sent:* Sunday, September 02, 2018 1:42 AM *To:* nanog@nanog.org *Subject:* Re: automatic rtbh trigger using flow data
This is not true. Some of our transits do RTBH for free. For example Cogent.
They will not do FlowSpec. Maybe their equipment can not do it or for some other reason.
However RTBH is a simple routing hack that can be implemented on any router. The traffic is dropped right at the edge and is never transported on the transit provider network. In that sense it also protects the transit network.
RTBH only for UDP would also be a very simple hack on many routers.
It might not be FlowSpec, but it may have most of the benefit, in a much simplified way.
Regards
Baldur
søn. 2. sep. 2018 02.39 skrev Ryan Hamel <Ryan.Hamel@quadranet.com>:
No ISP is in the business of filtering traffic unless the client pays the hefty fee since someone still has to tank the attack.
I also don’t think there is destination prefix IP filtering in flowspec, which could seriously cause problems.
*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Baldur Norddahl *Sent:* Saturday, September 01, 2018 5:18 PM *To:* nanog@nanog.org *Subject:* Re: automatic rtbh trigger using flow data
fre. 31. aug. 2018 17.16 skrev Hugo Slabbert <hugo@slabnet.com>:
I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream.
_keeps dreaming_
We just need a signal to drop UDP for a prefix. The same as RTBH but only for UDP. This would prevent all volumetric attacks without the end user being cut off completely.
Besides from some games, VPN and VoIP, they would have an almost completely normal internet experience. DNS would go through the ISP servers and only be affected if the user is using a third party service.
Regards
Baldur
Aaron Gould wrote : I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, if so I'd like to see the list too.
I emailed you. For years I ran it at home on a Cisco 1841, 100,000 BGP prefixes is nothing these days. I am not surprised that Joe pushes that to some CPEs.
Even so, this would not stop the attacks from hitting my front door, my side of my Internet uplink...when paying for a 30 gigs CIR and paying double for megabits per second over that, up to the ceiling of 100 gig every bit that hits my front door over 30 gig would cost me extra, remotely triggering based on my victim IP address inside my network would be my solution to saving money.
I agree. If you want to get a real use of source blacklisting, to save bandwidth, you probably went to rent a U in a rack at your upstream(s) to block it there. I never did it past 1GE, and I have never measured seriously the bandwidth it would save, would be curious to know. I think the two approaches are complementary to each other though. Michel. On Aug 30, 2018, at 6:43 PM, Michel Py <michel.py@tsisemi.com> wrote:
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga
I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity.
I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you!
The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close.
Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
Most of the solutions mentioned are paid, or fastnetmon is partially paid. And the thing you want is paid i believe.... Nice tool though, not saying anything against it. However.... My personal view is, as long as you can store your flow info in a timeseries database (like influxdb and NOT SQL LIKE!!!!!!!) you can do whatever you want with the (raw) data. And create custom triggers for different calculations. Flows are on the fly and are coming in constantly, you could have a calculation like group by srcip and whatever protocol you want or just srcip, and make a calculation for every x seconds or minutes. As i mentioned the flow data is a constant stream, so you could have it triggered as fast as you want. (and the nice thing is, with sflow, you also get as path, peer as, localpref,community (if enabled). You could group by anything.. :) I admit it takes a bit more time to setup but the outcome is amazing ;) (especially if you graph it then with grafana) And in your case it would be a script that does a influxdb command to make the calculations and if the outcome shows an IP meeting the thresholds you have set in the calculation, you trigger a script that adjusts the route to be announced to your upstream with the correct (rtbh) community. ( as i mentioned, as long as you have the "raw" flows, you can do anything ) Good luck, whatever you choose :) On 31-08-18 02:14, Michel Py wrote:
Aaron Gould wrote : I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, if so I'd like to see the list too. I emailed you. For years I ran it at home on a Cisco 1841, 100,000 BGP prefixes is nothing these days. I am not surprised that Joe pushes that to some CPEs.
Even so, this would not stop the attacks from hitting my front door, my side of my Internet uplink...when paying for a 30 gigs CIR and paying double for megabits per second over that, up to the ceiling of 100 gig every bit that hits my front door over 30 gig would cost me extra, remotely triggering based on my victim IP address inside my network would be my solution to saving money. I agree. If you want to get a real use of source blacklisting, to save bandwidth, you probably went to rent a U in a rack at your upstream(s) to block it there. I never did it past 1GE, and I have never measured seriously the bandwidth it would save, would be curious to know. I think the two approaches are complementary to each other though.
Michel.
On Aug 30, 2018, at 6:43 PM, Michel Py <michel.py@tsisemi.com> wrote:
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity. I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you! The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close.
Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
From experience, sflows are horribly inaccurate for DDoS detection, since the volume could disrupt the control plane and render the process useless, thus not giving data to the external system to act upon it. You can't get any better than mirroring your inbound transit, and sampling the output to a sensor. I have also spent some time on a spare server running netmap and influxdb, it simply could not keep up, nor could I write any useful queries against it. Which is why I'm deciding on using pf_ring ft (flow table) and just let it calculate all the data to be dumped into a MySQL memory table, and also harvesting ntop's NDPI protocol decoding, to enhance the detection rules. My ideal setup is to break things down into dedicated programs like flow exporter, detection, and response. I also want to make clear to Michel, that colo'ing a router at an ISP is no different than plugging it into your local router, your uplink will get saturated beyond what it can physically handle with only the ACLs protecting the other side, but if your clients are also receiving traffic on the same uplink as the attack, it's a denial of service to them. -----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of H I Baysal Sent: Friday, August 31, 2018 2:09 AM To: Michel Py <michel.py@tsisemi.com>; Aaron Gould <aaron1@gvtc.com>; michel@arneill-py.sacramento.ca.us Cc: Nanog@nanog.org Subject: Re: automatic rtbh trigger using flow data Most of the solutions mentioned are paid, or fastnetmon is partially paid. And the thing you want is paid i believe.... Nice tool though, not saying anything against it. However.... My personal view is, as long as you can store your flow info in a timeseries database (like influxdb and NOT SQL LIKE!!!!!!!) you can do whatever you want with the (raw) data. And create custom triggers for different calculations. Flows are on the fly and are coming in constantly, you could have a calculation like group by srcip and whatever protocol you want or just srcip, and make a calculation for every x seconds or minutes. As i mentioned the flow data is a constant stream, so you could have it triggered as fast as you want. (and the nice thing is, with sflow, you also get as path, peer as, localpref,community (if enabled). You could group by anything.. :) I admit it takes a bit more time to setup but the outcome is amazing ;) (especially if you graph it then with grafana) And in your case it would be a script that does a influxdb command to make the calculations and if the outcome shows an IP meeting the thresholds you have set in the calculation, you trigger a script that adjusts the route to be announced to your upstream with the correct (rtbh) community. ( as i mentioned, as long as you have the "raw" flows, you can do anything ) Good luck, whatever you choose :) On 31-08-18 02:14, Michel Py wrote:
Aaron Gould wrote : I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, if so I'd like to see the list too. I emailed you. For years I ran it at home on a Cisco 1841, 100,000 BGP prefixes is nothing these days. I am not surprised that Joe pushes that to some CPEs.
Even so, this would not stop the attacks from hitting my front door, my side of my Internet uplink...when paying for a 30 gigs CIR and paying double for megabits per second over that, up to the ceiling of 100 gig every bit that hits my front door over 30 gig would cost me extra, remotely triggering based on my victim IP address inside my network would be my solution to saving money. I agree. If you want to get a real use of source blacklisting, to save bandwidth, you probably went to rent a U in a rack at your upstream(s) to block it there. I never did it past 1GE, and I have never measured seriously the bandwidth it would save, would be curious to know. I think the two approaches are complementary to each other though.
Michel.
On Aug 30, 2018, at 6:43 PM, Michel Py <michel.py@tsisemi.com> wrote:
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity. I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you! The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close.
Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
I think your experience has to do more with your setup than sflow or influxdb... my sflow data, that i push into influxdb. is 1 on 1 accurate with the interface utilization ( even on group by per source ip ) Arista performs fine with sflow.... Don't know what brand you used. I'm getting 300 mbit of constant (sflow) data in influxdb.... ''' it simply could not keep up, nor could I write any useful queries against it ''' again, i really think this has to do with the setup or the queries you tried to do. (and (i think ) you tried sflow exporter on a server, i think the topic was from a appliance as an exporter) If you still want to give influxdb+ sflow a try, i'm more than willing to help you out ;) On 31-08-18 11:33, Ryan Hamel wrote:
From experience, sflows are horribly inaccurate for DDoS detection, since the volume could disrupt the control plane and render the process useless, thus not giving data to the external system to act upon it. You can't get any better than mirroring your inbound transit, and sampling the output to a sensor.
I have also spent some time on a spare server running netmap and influxdb, it simply could not keep up, nor could I write any useful queries against it. Which is why I'm deciding on using pf_ring ft (flow table) and just let it calculate all the data to be dumped into a MySQL memory table, and also harvesting ntop's NDPI protocol decoding, to enhance the detection rules. My ideal setup is to break things down into dedicated programs like flow exporter, detection, and response.
I also want to make clear to Michel, that colo'ing a router at an ISP is no different than plugging it into your local router, your uplink will get saturated beyond what it can physically handle with only the ACLs protecting the other side, but if your clients are also receiving traffic on the same uplink as the attack, it's a denial of service to them.
-----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of H I Baysal Sent: Friday, August 31, 2018 2:09 AM To: Michel Py <michel.py@tsisemi.com>; Aaron Gould <aaron1@gvtc.com>; michel@arneill-py.sacramento.ca.us Cc: Nanog@nanog.org Subject: Re: automatic rtbh trigger using flow data
Most of the solutions mentioned are paid, or fastnetmon is partially paid. And the thing you want is paid i believe.... Nice tool though, not saying anything against it. However....
My personal view is, as long as you can store your flow info in a timeseries database (like influxdb and NOT SQL LIKE!!!!!!!) you can do whatever you want with the (raw) data. And create custom triggers for different calculations.
Flows are on the fly and are coming in constantly, you could have a calculation like group by srcip and whatever protocol you want or just srcip, and make a calculation for every x seconds or minutes. As i mentioned the flow data is a constant stream, so you could have it triggered as fast as you want.
(and the nice thing is, with sflow, you also get as path, peer as, localpref,community (if enabled). You could group by anything.. :)
I admit it takes a bit more time to setup but the outcome is amazing ;) (especially if you graph it then with grafana) And in your case it would be a script that does a influxdb command to make the calculations and if the outcome shows an IP meeting the thresholds you have set in the calculation, you trigger a script that adjusts the route to be announced to your upstream with the correct (rtbh) community. ( as i mentioned, as long as you have the "raw" flows, you can do anything )
Good luck, whatever you choose :)
On 31-08-18 02:14, Michel Py wrote:
Aaron Gould wrote : I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, if so I'd like to see the list too. I emailed you. For years I ran it at home on a Cisco 1841, 100,000 BGP prefixes is nothing these days. I am not surprised that Joe pushes that to some CPEs.
Even so, this would not stop the attacks from hitting my front door, my side of my Internet uplink...when paying for a 30 gigs CIR and paying double for megabits per second over that, up to the ceiling of 100 gig every bit that hits my front door over 30 gig would cost me extra, remotely triggering based on my victim IP address inside my network would be my solution to saving money. I agree. If you want to get a real use of source blacklisting, to save bandwidth, you probably went to rent a U in a rack at your upstream(s) to block it there. I never did it past 1GE, and I have never measured seriously the bandwidth it would save, would be curious to know. I think the two approaches are complementary to each other though.
Michel.
On Aug 30, 2018, at 6:43 PM, Michel Py <michel.py@tsisemi.com> wrote:
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity. I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you! The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close.
Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
On 31 Aug 2018, at 16:33, Ryan Hamel wrote:
From experience, sflows are horribly inaccurate for DDoS detection, since the volume could disrupt the control plane and render the process useless, thus not giving data to the external system to act upon it.
On the contrary, flow telemetry in general works quite well for DDoS detection/classification/traceback, and is widely utilized for such purposes; it has been for many years. I'm not a big fan of s/Flow comparatively speaking, but it and NetFlow, IPFIX, et. al. have proven themselves over the years, assuming that the flow export parameters on the exporting devices are configured correctly, and the collection/analysis systems are configured optimally. Flow telemetry is management-plane, not control-plane. Implementing network infrastructure self-protection BCPs such as iACLs is definitely recommended in general. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Ryan Hamel wrote : I also want to make clear to Michel, that colo'ing a router at an ISP is no different than plugging it into your local router, your uplink will get saturated beyond what it can physically handle with only the ACLs protecting the other side, but if your clients are also receiving traffic on the same uplink as the attack, it's a denial of service to them.
I understand, but there still is a difference : Bandwith inside the colo is cheaper than the WAN link; conceptually you could have multiple fat interfaces at the colo and hope to filter the unwanted traffic before it hits where it generally hurts : on the WAN link. Would save money only if the transit is billed separately from the WAN link traffic, and every situation is different. Michel. -----Original Message----- From: NANOG <nanog-bounces@nanog.org> On Behalf Of H I Baysal Sent: Friday, August 31, 2018 2:09 AM To: Michel Py <michel.py@tsisemi.com>; Aaron Gould <aaron1@gvtc.com>; michel@arneill-py.sacramento.ca.us Cc: Nanog@nanog.org Subject: Re: automatic rtbh trigger using flow data Most of the solutions mentioned are paid, or fastnetmon is partially paid. And the thing you want is paid i believe.... Nice tool though, not saying anything against it. However.... My personal view is, as long as you can store your flow info in a timeseries database (like influxdb and NOT SQL LIKE!!!!!!!) you can do whatever you want with the (raw) data. And create custom triggers for different calculations. Flows are on the fly and are coming in constantly, you could have a calculation like group by srcip and whatever protocol you want or just srcip, and make a calculation for every x seconds or minutes. As i mentioned the flow data is a constant stream, so you could have it triggered as fast as you want. (and the nice thing is, with sflow, you also get as path, peer as, localpref,community (if enabled). You could group by anything.. :) I admit it takes a bit more time to setup but the outcome is amazing ;) (especially if you graph it then with grafana) And in your case it would be a script that does a influxdb command to make the calculations and if the outcome shows an IP meeting the thresholds you have set in the calculation, you trigger a script that adjusts the route to be announced to your upstream with the correct (rtbh) community. ( as i mentioned, as long as you have the "raw" flows, you can do anything ) Good luck, whatever you choose :) On 31-08-18 02:14, Michel Py wrote:
Aaron Gould wrote : I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, if so I'd like to see the list too. I emailed you. For years I ran it at home on a Cisco 1841, 100,000 BGP prefixes is nothing these days. I am not surprised that Joe pushes that to some CPEs.
Even so, this would not stop the attacks from hitting my front door, my side of my Internet uplink...when paying for a 30 gigs CIR and paying double for megabits per second over that, up to the ceiling of 100 gig every bit that hits my front door over 30 gig would cost me extra, remotely triggering based on my victim IP address inside my network would be my solution to saving money. I agree. If you want to get a real use of source blacklisting, to save bandwidth, you probably went to rent a U in a rack at your upstream(s) to block it there. I never did it past 1GE, and I have never measured seriously the bandwidth it would save, would be curious to know. I think the two approaches are complementary to each other though.
Michel.
On Aug 30, 2018, at 6:43 PM, Michel Py <michel.py@tsisemi.com> wrote:
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity. I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you! The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close.
Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
Instead of rtbh I would suggest blocking/rate limiting common ports used in DDoS attacks. That will block 90% of the DDoS attacks. We recently open sourced a BGP Flowspec based tool for DDoS Mitigation. It applies Flowspec rules per victim IP Addr. https://github.com/racompton/docker-auto-flowspec ~Pratik Lotia -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of H I Baysal Sent: Friday, August 31, 2018 3:09 AM To: Michel Py; Aaron Gould; michel@arneill-py.sacramento.ca.us Cc: Nanog@nanog.org Subject: Re: automatic rtbh trigger using flow data Most of the solutions mentioned are paid, or fastnetmon is partially paid. And the thing you want is paid i believe.... Nice tool though, not saying anything against it. However.... My personal view is, as long as you can store your flow info in a timeseries database (like influxdb and NOT SQL LIKE!!!!!!!) you can do whatever you want with the (raw) data. And create custom triggers for different calculations. Flows are on the fly and are coming in constantly, you could have a calculation like group by srcip and whatever protocol you want or just srcip, and make a calculation for every x seconds or minutes. As i mentioned the flow data is a constant stream, so you could have it triggered as fast as you want. (and the nice thing is, with sflow, you also get as path, peer as, localpref,community (if enabled). You could group by anything.. :) I admit it takes a bit more time to setup but the outcome is amazing ;) (especially if you graph it then with grafana) And in your case it would be a script that does a influxdb command to make the calculations and if the outcome shows an IP meeting the thresholds you have set in the calculation, you trigger a script that adjusts the route to be announced to your upstream with the correct (rtbh) community. ( as i mentioned, as long as you have the "raw" flows, you can do anything ) Good luck, whatever you choose :) On 31-08-18 02:14, Michel Py wrote:
Aaron Gould wrote : I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, if so I'd like to see the list too. I emailed you. For years I ran it at home on a Cisco 1841, 100,000 BGP prefixes is nothing these days. I am not surprised that Joe pushes that to some CPEs.
Even so, this would not stop the attacks from hitting my front door, my side of my Internet uplink...when paying for a 30 gigs CIR and paying double for megabits per second over that, up to the ceiling of 100 gig every bit that hits my front door over 30 gig would cost me extra, remotely triggering based on my victim IP address inside my network would be my solution to saving money. I agree. If you want to get a real use of source blacklisting, to save bandwidth, you probably went to rent a U in a rack at your upstream(s) to block it there. I never did it past 1GE, and I have never measured seriously the bandwidth it would save, would be curious to know. I think the two approaches are complementary to each other though.
Michel.
On Aug 30, 2018, at 6:43 PM, Michel Py <michel.py@tsisemi.com> wrote:
Joe Maimon wrote : I use a bunch of scripts plus a supervisory sqlite3 database process all injecting into quagga I have the sqlite part planned, today I'm using a flat file :-( I know :-(
Also aimed at attacker sources. I feed it with honeypots and live servers, hooked into fail2ban and using independent host scripts. Not very sophisticated, the remotes use ssh executed commands to add/delete. I also setup a promiscuous ebgp RR so I can extend my umbrella to CPE with diverse connectivity. I would like to have your feed. How many attacker prefixes do you currently have ?
Using flow data, that sounds like an interesting direction to take this into, so thank you! The one thing we can share here is the attacker prefixes. The victim prefixes are unique to each of us but I expect our attacker prefixes to be very close.
Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On 31 Aug 2018, at 23:53, Lotia, Pratik M wrote:
Instead of rtbh I would suggest blocking/rate limiting common ports used in DDoS attacks.
This isn't an 'instead of', it's an 'in addition to'. And it must be done judiciously; many operators doing this have concentrated on common port-pairs observed in UDP reflection/amplification attacks. It's important to understand that any kind of packet of any protocol/ports (if such concepts apply on the protocol in question) can be used to launch DDoS attacks. We've many tools in the toolbox, and should use them in a situationally-appropriate manner. And when we're using techniques like QoSing down certain ports/protocols, we must err on the side of caution, lest we cause larger problems than the attacks themselves. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
many operators doing this have concentrated on common port-pairs observed in UDP reflection/amplification attacks.
Yes, because that's a great starting point.
And when we're using techniques like QoSing down certain ports/protocols, we must err on the side of caution,
Arbor report mentions volumetric attacks using DNS, NTP form 75+% of the attacks. Then QoSing certain ports and protocols is the best way to start with. ~Pratik Lotia -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Roland Dobbins Sent: Friday, August 31, 2018 11:13 AM To: NANOG list Subject: Re: automatic rtbh trigger using flow data On 31 Aug 2018, at 23:53, Lotia, Pratik M wrote:
Instead of rtbh I would suggest blocking/rate limiting common ports used in DDoS attacks.
This isn't an 'instead of', it's an 'in addition to'. And it must be done judiciously; many operators doing this have concentrated on common port-pairs observed in UDP reflection/amplification attacks. It's important to understand that any kind of packet of any protocol/ports (if such concepts apply on the protocol in question) can be used to launch DDoS attacks. We've many tools in the toolbox, and should use them in a situationally-appropriate manner. And when we're using techniques like QoSing down certain ports/protocols, we must err on the side of caution, lest we cause larger problems than the attacks themselves. ----------------------------------- Roland Dobbins <rdobbins@arbor.net> E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On 1 Sep 2018, at 1:20, Lotia, Pratik M wrote:
Arbor report mentions volumetric attacks using DNS, NTP form 75+% of the attacks.
I'm well aware of what's mentioned in the Arbor report, thanks! ;>
Then QoSing certain ports and protocols is the best way to start with.
The point is that when applying broad policies of this nature, one must be very conservative, else one can cause larger problems on a macro scale. Internet ateriosclerosis is a significant issue. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
(I think this is all about volumetric attacks btw...it's my belief that slow-and-low attacks are continually occurring and are going largely unnoticed...i'll speak for myself) Few years ago we began seeing certain ports used as attack vectors, thus we began our internet boundary policers for these ports... as time went on, we add to that list of ports. Some ports as we know, like dns, and I think ntp from time to time (dang, sorry, lol) are used in amplification, and so, we can't police legit ports too slowly or real stuff is affected... so that's what Roland probably meant by "judiciously" We also have inside this set of qos tools at the internet boundary, an ever-growing acl that we call "repeat victims"... we have grown to understand that, if a customer ip address is attack once, it's likely it will be attacked again... There are new attacked ports all the time, so sometimes, an attack gets through... which is causing me to think about an overall UDP limit on my internet boundary ports... since most attacks are udp-based*....furthermore, along with that overarching udp limit, I may mark internet-sourced-udp with a certain marking dscp/exp so that as it travels through my internet network, it will be the first to get dropped (? Wred ? work well for udp?) during congestion when an attack gets through -Aaron * btw, what can you experts tell me about tcp-based volumetric attacks... please help me to understand... does tcp have an inherent inability to ramp-up to massive speeds/loads with it's sliding window and must-rcv-ack-before sending more segments ?? I ask since I heard this years ago about tcp and I wonder if this is why -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Roland Dobbins Sent: Friday, August 31, 2018 12:13 PM To: NANOG list Subject: Re: automatic rtbh trigger using flow data On 31 Aug 2018, at 23:53, Lotia, Pratik M wrote:
Instead of rtbh I would suggest blocking/rate limiting common ports used in DDoS attacks.
This isn't an 'instead of', it's an 'in addition to'. And it must be done judiciously; many operators doing this have concentrated on common port-pairs observed in UDP reflection/amplification attacks. It's important to understand that any kind of packet of any protocol/ports (if such concepts apply on the protocol in question) can be used to launch DDoS attacks. We've many tools in the toolbox, and should use them in a situationally-appropriate manner. And when we're using techniques like QoSing down certain ports/protocols, we must err on the side of caution, lest we cause larger problems than the attacks themselves. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Fri 2018-Aug-31 13:35:29 -0500, Aaron Gould <aaron1@gvtc.com> wrote:
* btw, what can you experts tell me about tcp-based volumetric attacks... please help me to understand... does tcp have an inherent inability to ramp-up to massive speeds/loads with it's sliding window and must-rcv-ack-before sending more segments ?? I ask since I heard this years ago about tcp and I wonder if this is why
UDP, depending on the application, can be reflected and amplified. Generally on the TCP side you can try SYN or ACK floods, but you're not going to get an amplified reflection. -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
On 1 Sep 2018, at 1:43, Hugo Slabbert wrote:
Generally on the TCP side you can try SYN or ACK floods, but you're not going to get an amplified reflection.
Actually, TCP reflection/amplification has been on the increase; the attacker is guaranteed at least 4:1 amplification in most circumstances, the number of reflectors/amplifiers is for all practical purposes infinite, and they're mostly legitimate, non-broken services/applications. And as always, it's important to note that with all reflection/amplification attacks, the root of the issue is the lack of universal source-address validation (SAV). Without the ability to spoof, there would be no reflection/amplification attacks. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Sun 2018-Sep-02 10:09:32 +0700, Roland Dobbins <rdobbins@arbor.net> wrote:
On 1 Sep 2018, at 1:43, Hugo Slabbert wrote:
Generally on the TCP side you can try SYN or ACK floods, but you're not going to get an amplified reflection.
Actually, TCP reflection/amplification has been on the increase; the attacker is guaranteed at least 4:1 amplification in most circumstances, the number of reflectors/amplifiers is for all practical purposes infinite, and they're mostly legitimate, non-broken services/applications.
Fair. I guess in terms of common reflect/amp vector at $dayjob we just see UDP-based significantly more frequently on large volumetric attacks given the amp factor on some vectors is so huge. Some relevant reading I need to revisit: https://www.usenix.org/sites/default/files/conference/protected-files/woot14... https://www.usenix.org/system/files/conference/woot14/woot14-kuhrer.pdf
And as always, it's important to note that with all reflection/amplification attacks, the root of the issue is the lack of universal source-address validation (SAV). Without the ability to spoof, there would be no reflection/amplification attacks.
ACK, pun intended.
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
On 1 Sep 2018, at 1:35, Aaron Gould wrote:
I may mark internet-sourced-udp with a certain marking dscp/exp so that as it travels through my internet network, it will be the first to get dropped (? Wred ? work well for udp?) during congestion when an attack gets through
You can use flow telemetry analysis to look at the UDP non-initial fragments destined for any access networks under your control; you'll likely see that they comprise a tiny portion of the overall traffic mix, and they're most commonly associated with large DNS answers. Once you've determined the numbers, you can police down the non-initial fragments destined for the access networks you control (don't do this on transit traffic!) to whatever small percentage makes the most sense, with a bit of extra headroom. 1% of link bandwidth works for several operators. In that QoS policy, you exempt well-known/well-run open DNS recursor farms like Google DNS, OpenDNS, et. al. (and possibly your own, depending on your topology, etc.), and any other legitimate source CIDRs which generate appreciable amounts of non-initial fragments. When a reflection/amplification attack which involves non-initial fragments hits, the QoS policy will sink a significant proportion of the attack. It doesn't help with your peering links, but keeps the traffic off your core and off the large network(s). Again, don't apply this across-the-board; only do it for access networks within your span of administrative control.
* btw, what can you experts tell me about tcp-based volumetric attacks...
TCP reflection/amplification. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Fri, Aug 31, 2018 at 11:09:19AM +0200, H I Baysal wrote:
My personal view is, as long as you can store your flow info in a timeseries database (like influxdb and NOT SQL LIKE!!!!!!!) you can do whatever you want with the (raw) data. And create custom triggers for different calculations.
For one of our customers I've deployed good old pmacct + MySQL (using memory engine) backend for DDoS detection purposes. It has some drawbacks (e.g. one has to frequently delete old records to keep tables fit and fast) but it allows asking complex SQL queries against these short term data (e.g. different detection logic per subnets) or precompute with triggers.
Flows are on the fly and are coming in constantly, you could have a calculation like group by srcip and whatever protocol you want or just srcip,
Beware of high cardinality issues when facing random src IP floods. BTW, once again pmacct (with some glue) is nice for feeding flow data into time series database. It can pre aggregate and pre filter low volume flows to reduce storage requirements. -- Paweł Małachowski
I also looked at PMACCT, however I have to say that I am way more pleased with the current setup. PMACCT relies on bgp session etc, while you actually get that info from your (s)flow. No need for lookup. And I really didn't like the idea of SQL like DB, I know i could push it to influx. But i thought this could be done way simpler and i think i did exactly that. I store the "raw" info for 7 days, as detailed as possible. then downsample for a week again, a month and a year. And the picture rendering api function of grafana makes it possible for me to do for example /dst AS# or /peer #AS or /as-path AS# in a chatbot, and grafana actually pushes the graph to telegram in our case, but it could be any bot. So at peering conferences, i can see with 1 message how much traffic we do either to a peer, destination or if traffic passes your AS. I don't see any (pre-made) system beating this :P On 01-09-18 01:51, Paweł Małachowski wrote:
On Fri, Aug 31, 2018 at 11:09:19AM +0200, H I Baysal wrote:
My personal view is, as long as you can store your flow info in a timeseries database (like influxdb and NOT SQL LIKE!!!!!!!) you can do whatever you want with the (raw) data. And create custom triggers for different calculations. For one of our customers I've deployed good old pmacct + MySQL (using memory engine) backend for DDoS detection purposes. It has some drawbacks (e.g. one has to frequently delete old records to keep tables fit and fast) but it allows asking complex SQL queries against these short term data (e.g. different detection logic per subnets) or precompute with triggers.
Flows are on the fly and are coming in constantly, you could have a calculation like group by srcip and whatever protocol you want or just srcip, Beware of high cardinality issues when facing random src IP floods.
BTW, once again pmacct (with some glue) is nice for feeding flow data into time series database. It can pre aggregate and pre filter low volume flows to reduce storage requirements.
participants (11)
-
Aaron Gould -
Baldur Norddahl -
H I Baysal -
Hugo Slabbert -
Joe Maimon -
Lotia, Pratik M -
Michel Py -
Paweł Małachowski -
Roland Dobbins -
Ryan Hamel -
Vicente De Luca