Am 27.04.2016 um 18:09 schrieb Hank Nussbacher <hank@efes.iucc.ac.il>:

On 27/04/2016 18:58, John Kristoff wrote:

...

On Thu, 21 Apr 2016 09:46:13 +0200 Martin Bacher <ti14m028@technikum-wien.at> wrote:

...

- Intra-AS BGP FlowSpec deployment: Who is running it? For which kind of attacks are you using it? Are you only dropping or rate-limiting certain traffic or are you also using the redirect/remark capabilities? What are the limitations from your perspective? Are you facing any operational issues? How are you injecting the FlowSpec routes? Unless you received a number of private responses, perhaps the lack of public responses is telling. Geant runs a Firewall of Demand based on BGP Flowspec (Juniper routers). You can read more about it here: http://www.geant.org/Networks/Network_Operations/PublishingImages/Pages/Fire... https://www.terena.org/activities/tf-csirt/meeting44/Firewall%20on%20Demand_... Thank you Hank. That’s a pretty nice intra AS implementation with a nice interface for customers.

Cheers, Martin

Regards, Hank

...

I've heard of a few networks doing this and there is some public record of it being used, including one instance where a bad rule was behind a serious outage:

<https://support.cloudflare.com/hc/en-us/articles/200172446-CloudFlare-Post-Mortem-from-Outage-on-March-3-2013>

...

- Inter-AS: Who is running Inter-AS FlowSpec deployments? What is your experience? Are there any concerns regarding Inter-AS deployments? Has anyone done interop tests? You might mine public, archived BGP data and see if there are any traffic filtering rules present (they are encoded in extended communities, which are optional, transitive).

We once tried to coordinate an Inter-AS flow-spec project, but it failed miserably due to lack of interest. For posterity, here is the project page:

<https://www.cymru.com/jtk/misc/community-fs.html>

Literally the only people who were interested in it at the time was one of the spec's co-authors. :-)

Since then, we have tried a more modest approach using the well known BGP RTBH technique:

<https://www.cymru.com/jtk/misc/utrs.html>

This has been much more successful and since we've started we've probably had about a dozen networks express interest in flow-spec rules. Verification of rules is potentially tricky, but widespread interest still lags in my estimation.

...

- How are you detecting DDoS attacks (Netflow, in-line probes, ..?) and which applications are you using for the analysis (Peakflow, Open-Source tools, ..?) Not speaking for anyone in particular, but don't forget about user complaints. In some cases a network may not notice (or care) if an attack is below a certain threshold for their network, but above a stress point downstream.

John

Hello Tyler, thanks for your reply.

Am 28.04.2016 um 17:37 schrieb Tyler Haske <tyler.haske@gmail.com>:

Martin,

...

Last but not least: I am also looking for anonymized statistical data about DDoS attacks which I could use in the thesis. I am mainly interested in data about the type of attacks, attack time, sources, source and destination ports, and so on. I know this something which is generally not shared, so I would really appreciate it if someone would be able to share such data.

Many companies are extremely reluctant to share their attack data. But that's OK, because there are other ways to get it.

Have you investigated backscatter analysis? It's used to see ongoing and current Internet scope DDoS attacks. I just had a look on that and thought that its only be able to detect some of the attacks. You might not detect large state of the art reflection and amplification attacks with that method. But i think it is useful for some sort of attacks like SYN flood. Do you agree?

Inferring Internet Denial of Service Activity https://cseweb.ucsd.edu/~savage/papers/UsenixSec01.pdf

Analyzing Large DDoS Attacks Using Multiple Data Sources https://www.cs.utah.edu/~kobus/docs/ddos.lsad.pdf

ISP Security - Real World Techniques https://www.nanog.org/meetings/nanog23/presentations/greene.ppt

A Summary of DoS/DDoS Prevention, Monitoring and Mitigation Techniques in a Service Provider Environment https://www.sans.org/reading-room/whitepapers/intrusion/summary-dos-ddos-pre...

Maybe you have access to some public IPs, then you can do this data collection yourself.

Sure, I will definitely think about hat. Thanks again for your reply and for providing the links. Greetings, Martin

Regards,

Tyler

On 2016-04-28 02:31 AM, Martin Bacher wrote:

...

Literally the only people who were interested in it at the time was one of the spec's co-authors. :-)

That’s how it usually starts. ;)

Given that I may be the guilty one here, I thought it might be worth chiming in. Inter-AS FlowSpec largely met the same fate as inter-AS source-based RTBH, where upstreams would only want to permit you to block sources destined for your address blocks (i.e.,. not others, so you would have to specify extended drop rule sets -- at least source and destination). As a result, with inter-AS FlowSpec, to appropriately scope things ingress policy specification is more complicated and hardware support was pretty limited at the time as well. Additionally, there was also only one vendor implementation at the time but now there are many and the IETF's IDR working group is continuing to enhance the capabilities and options available with FlowSpec. There are a large number of intra-AS and multi-AS single administrative domains that use FlowSpec today (my $dayjob included, for an array of things, not just DDoS mitigation). And as you point out Martin, it's simply another option available in the toolkit. One of the nicest things about it is that unlike destination-based RTBH, where you effectively completed the attack, if you can identify the primitives, namely at the network and transport layer, you can squelch a large number of attack vectors in an automated manner with minimal action required by the operator. We use it effectively in a layered model where "Principle of Minimal Intervention" applies, allowing attack mitigation and traffic diversion in the most optimal place (e.g., at network ingress), and only scrubbing or diverting traffic when necessary. Just like destination and source-based RTBH, FlowSpec is simply another evolution of automating forwarding path configuration, where NFV/SDN are the newest incarnation and can allows badness such as DDoS to be dropped implicitly rather than explicitly, even... -danny

Am 02.05.2016 um 15:03 schrieb Alexander Maassen <outsider@scarynet.org>:

On Mon, May 2, 2016 2:30 pm, Danny McPherson wrote:

...

We use it effectively in a layered model where "Principle of Minimal Intervention" applies, allowing attack mitigation and traffic diversion in the most optimal place (e.g., at network ingress), and only scrubbing or diverting traffic when necessary.

Sorry to say, but the most optimal place for ddos mitigation is at network egress of origin. What comes in mind regarding that is the ability for target ASN telling source ASN to stop sending packets from a specific (let's say /29) in the case of a DDoS (with appropiate security measures in place off course).

Because, let's face it, why would a target of a ddos need to nullroute itself?

Well, I think ingress filtering at the Internet edge (see BCP38 and BCP84) would be the best approach. But we as Internet community are clearly failing in that area. And origin ASes of amplification and reflection attacks are most probably not able to detect DNS ANY queries or NTP monlist queries at a low rate without DPI. The networks used for reflection and amplification may be able to detect an ongoing attack and they will then hopefully fix their implementations and not deploy egress filters. So the question is how to get rid of source IP address spoofing at all? I don’t see any chance by now to push ASes, which are not filtering properly, to implement ingress filtering. What could help is to add session handling to UDP based protocols as proposed by Christian Rossow and implemented by Google in Quic. But that’s again just a workaround and may create new problems because of backwards compatibility issues. So filtering as precise as possible and as close as possible to the attack source is maybe the best option we have at the moment.

Am 02.05.2016 um 14:30 schrieb Danny McPherson <danny@tcb.net>:

On 2016-04-28 02:31 AM, Martin Bacher wrote:

...

...

Literally the only people who were interested in it at the time was one of the spec's co-authors. :-) That’s how it usually starts. ;)

Given that I may be the guilty one here, I thought it might be worth chiming in.

Inter-AS FlowSpec largely met the same fate as inter-AS source-based RTBH, where upstreams would only want to permit you to block sources destined for your address blocks (i.e.,. not others, so you would have to specify extended drop rule sets -- at least source and destination). I mainly agree on that. However, I have not found evidence of inter-AS S-RTBH deployments as of now. This would really require, at least in my understanding, a lot of hacks in order to implement it properly and avoid blackholing of the wrong traffic. BGP-FS is clearly doing a better job in that area. However, Tier 1s and most probably also some of the Tier 2s may not want to offer it to customers because they are loosing money if less traffic is sent downstream on IP-Transit links. What I would expect to see more often is iBGP deployments in order to protect the own infrastructure by remarking suspicious traffic to worst than best effort automatically. That’s again an example why BGP-FS in inter-AS setups may not be deployed on a large scale and things may stay worse for the Internet edge (and I still hope that I am wrong with that assumption).

As a result, with inter-AS FlowSpec, to appropriately scope things ingress policy specification is more complicated and hardware support was pretty limited at the time as well. Additionally, there was also only one vendor implementation at the time but now there are many and the IETF's IDR working group is continuing to enhance the capabilities and options available with FlowSpec. Yes, I have also noticed that. And the hardware support and inter-AS verification options are much better compared to the very beginning.

There are a large number of intra-AS and multi-AS single administrative domains that use FlowSpec today (my $dayjob included, for an array of things, not just DDoS mitigation). And as you point out Martin, it's simply another option available in the toolkit.

I personally think that it will really become standard for single administrative domains in the future as it is with L2VPNs and L3VPNs. Most of the AS will just deploy it and use it for DDoS mitigation and other applications. You just mentioned that you also used for other things. May I ask you about your use-cases?

One of the nicest things about it is that unlike destination-based RTBH, where you effectively completed the attack, if you can identify the primitives, namely at the network and transport layer, you can squelch a large number of attack vectors in an automated manner with minimal action required by the operator.

Could not agree more.

We use it effectively in a layered model where "Principle of Minimal Intervention" applies, allowing attack mitigation and traffic diversion in the most optimal place (e.g., at network ingress), and only scrubbing or diverting traffic when necessary. Just like destination and source-based RTBH, FlowSpec is simply another evolution of automating forwarding path configuration, where NFV/SDN are the newest incarnation and can allows badness such as DDoS to be dropped implicitly rather than explicitly, even…

Great. Thanks for sharing that. One must just make sure that the tools are used properly. High volume attacks can easily mitigated in many cases with BGP-FS while while other attacks like low bandwidth TCP attacks will have to be mitigated by scrubbing centers. @SDN/NFV: I am not so sure if this will really help or make things just more complicated. I have just been told that people are working on netconf/yang solutions for ACL deployments, which may again only work for intra-AS deployments. But your comment is going, at least in my understand, beyond ACL deployments, right? Could you please elaborate a bit further on that. Many thanks. Martin

-danny

On 2 May 2016, at 20:16, Martin Bacher wrote:

However, Tier 1s and most probably also some of the Tier 2s may not want to offer it to customers because they are loosing money if less traffic is sent downstream on IP-Transit links.

I will go a step further than Danny's comments and state that this is categorically and demonstrably untrue. Many of the quite large 'Tier-1' and 'Tier-2' (using the old terminology) operators on this list offer commercial DDoS mitigation services making use of technologies like D/RTBH, S/RTBH, IDMS, et. al. due to customer demand. They need these capabilities in order to defend their own properties and assets, and they are also offering them to end-customers who want and need them. In point of fact, it's becoming difficult to find one which *doesn't* offer this type of service. There were a couple of situations in the first half of the first decade of this millennium where operators took this attitude. But they changed their tunes pretty rapidly once they themselves were impacted, and once they started losing customers because they couldn't and wouldn't protect them. And as Danny notes, these technologies are all tools in the toolbox. NFV and 'SDN' have tremendous potential to make it a lot easier to bring mitigation resources to bear in a dynamic and optimal fashion within single spans of administrative control; and there are standards-based efforts underway to provide for a higher degree of automation, increased rapidity of response, and interoperability in both inter- and intra-network DDoS mitigation scenarios. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 3 May 2016, at 4:51, jim deleskie wrote:

I was going to avoid this thread because I've never been a huge fan of Flowspec for my own reasons.

Flowspec is an extremely useful tool, IMHO - not only for direct, layer-4-granular mitigation leveraging linecard ASICs, but for more granular and selective diversion into mitigation centers, as well. And its value is growing with increased platform support. It isn't perfect (nothing is), and operators must be aware of its performance/scalability envelope on a given platform, but it's a great tool to have in the toolbox.

I can say I, nor any of my peers ( in any sense of that word) that I have known, have wanted to keep "bad " traffic on our networks so we can bill for it.

+1! I ran into this situation precisely twice early in the 'oughts ("Let the packets come!" was the quote which stood out in my mind); those espousing it pretty quickly changed their tunes once their networks had been knocked flat a couple of times. ;> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 3 May 2016, at 5:38, Martin Bacher wrote:

Let the packets come is not the message.

That was *precisely* the message which was spoken to me directly by a large regional CONUS ISP in mid-2003 or thereabouts. I know this; I was there. And it was the wrong message, as that particular ISP found out a couple of weeks later when their network was knocked flat and they lost customers because of it. A bit of schadenfreude might not have been out of place, for the less-charitably inclined.

or remark and/or rate-limit the particular flows with nearly, of course not for the customer under attack, the same result.

This is almost always a Bad Idea, because the programmatically-generated attack traffic ends up 'crowding out' the legitimate traffic. For some attacks which are obviously out-of-profile with regards to the attack targets, this isn't as much of a concern; some large network operators are doing this with regards to common UDP reflection/amplification traffic (but they're being careful about it). And that still doesn't address the issue of high-volume traffic choking peering/transit links, of course.

But that does not imply that all upstream ISPs are filtering out attacks by default for customers which are not paying for that.

Nobody here has said that. But some beneficiary collateral effects of this nature do show up, from time to time.

This is at least my interpretation from reading the various available DDoS reports and research papers.

You should probably be aware that you are likely conversing directly with the authors of/contributors to some of those very reports and research papers in this thread (depending on which reports and papers you mean), and that the people with whom you are interacting routinely mitigate DDoS attacks on the public Internet as part of their normal work routine - and have done so for many years. For many of us, this is not a theoretical discussion; and it would probably be a good idea to keep in mind that our contributions to this thread aren't based upon reading various reports and research papers, but rather upon our actions which generate the data and experiential observations upon which such reports and research papers are based. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

Am 02.05.2016 um 23:38 schrieb Roland Dobbins <rdobbins@arbor.net>:

On 2 May 2016, at 20:16, Martin Bacher wrote:

...

However, Tier 1s and most probably also some of the Tier 2s may not want to offer it to customers because they are loosing money if less traffic is sent downstream on IP-Transit links.

I will go a step further than Danny's comments and state that this is categorically and demonstrably untrue.

Many of the quite large 'Tier-1' and 'Tier-2' (using the old terminology) operators on this list offer commercial DDoS mitigation services making use of technologies like D/RTBH, S/RTBH, IDMS, et. al. due to customer demand. They need these capabilities in order to defend their own properties and assets, and they are also offering them to end-customers who want and need them.

In point of fact, it's becoming difficult to find one which *doesn't* offer this type of service. It was not meant to be a general statement that they are not offering anti DDoS services in whatever flavor. But you usually just get what you pay for. Furthermore, my statement was related to inter-AS BGP-FS and that providers may not offer it to customers but use in instead for traffic remarking to something like worse than best effort and still forwarding it to a customer under attack if he is not paying extra fees for DDoS mitigation. That does not mean that the ISP does not help on request or deploys countermeasures if its own infrastructure or other customers are suffering from that attack. But he may not perform any mitigation (except for the own protection) by default.

There were a couple of situations in the first half of the first decade of this millennium where operators took this attitude. But they changed their tunes pretty rapidly once they themselves were impacted, and once they started losing customers because they couldn't and wouldn't protect them.

And as Danny notes, these technologies are all tools in the toolbox. NFV and 'SDN' have tremendous potential to make it a lot easier to bring mitigation resources to bear in a dynamic and optimal fashion within single spans of administrative control; and there are standards-based efforts underway to provide for a higher degree of automation, increased rapidity of response, and interoperability in both inter- and intra-network DDoS mitigation scenarios.

Sounds nice. Looking forward to see that implemented on a large scale in inter-AS setups. But I am not sure if this will really happen.

----------------------------------- Roland Dobbins <rdobbins@arbor.net>