At 18:03 30/06/2015 +0900, Randy Bush wrote:

be nice if some technical details were included

Your prefix: xx.104.150.0/24: Prefix Description: xxxx Update time: 2015-06-30 07:39 (UTC) Detected by #peers: 8 Detected prefix: xx.104.150.0/24 Announced by: AS58587 (FIBERATHOME-BD Fiber @ Home Limited,BD) Upstream AS: AS6939 (HURRICANE - Hurricane Electric, Inc.,US) ASpath: 25152 6939 58587 and another: On Tuesday June 30th 2015 at 7:37 UTC we detected a Origin AS Change event for your prefix (23.44.244.0/22 Akamai) The detected prefix: 23.44.244.0/22, was announced by AS58587 (FIBERATHOME-BD Fiber @ Home Limited,BD) Alert description: Origin AS Change Detected Prefix: 23.44.244.0/22 Detected Origin AS: 58587 Expected Origin AS: 1680 Same Aspath of 6939 58587 -Hank

Randy Bush <randy@psg.com> wrote

...

A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that.

7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp.

I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general? ----- Matsuzaki Yoshinobu <maz@iij.ad.jp> - IIJ/AS2497 INOC-DBA: 2497*629

On 30/06/2015 14:29, Mark Tinka wrote:

- Get your downstreams to create route objects before you turn them up. - Get your provisioning teams to validate the prefixes being provided by your downstreams. - Use both prefix- and AS_PATH-based filters for your downstreams. - Use BGP communities (as you've stated). - No exceptions.

plus: - fully automate ingress prefix management - use maxprefixes with manual reenable on all ebgp sessions I've been caught with fully automated IRR based per-session prefix filtering where the customer put the IXP AS macro into their AS macro. When the customer did a 7007 on this, we accepted everything that they announced back to us, oy vey. So you need both. Nick

On Tue, Jun 30, 2015 at 10:22:38PM +0900, Matsuzaki Yoshinobu wrote:

Randy Bush <randy@psg.com> wrote

...

...

A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that.

7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp.

I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general?

In addition to the BGP community scheme, outbound as-path filters could help. Most network's list of transit providers is fairly static, it would be easiy with as-path filters to prevent announcing upstream routes to other upstreams or peering partners. Kind regards, Job

On Tue, Jun 30, 2015 at 09:44:12AM -0400, Joe Abley wrote:

On 30 Jun 2015, at 9:41, Job Snijders wrote:

...

In addition to the BGP community scheme, outbound as-path filters could help.

I agree, but possibly not in the case of a redistribution loop.

(We don't know that's what happened, exactly, but I thought it was worth mentioning.)

Joe, you are right. In this specific situation, for a small to medium sized network, it might be prudent to apply an outbound prefix-filter on all transit & peering sessions and thus only allowing prefixes which actually belong to downstream customers and the network itself. One could generate that prefix-list based on the network's AS-SET. I wouldn't deploy /only/ outbound prefix-filters. This method should be viewed as complementary to other methods such as the already mentioned a BGP community scheme. Kind regards, Job

On Tue, Jun 30, 2015 at 04:38:48PM +0200, Mark Tinka wrote:

On 30/Jun/15 16:24, Job Snijders wrote:

...

In this specific situation, for a small to medium sized network, it might be prudent to apply an outbound prefix-filter on all transit & peering sessions and thus only allowing prefixes which actually belong to downstream customers and the network itself.

I say that regardless of size, deploy the ultimate solution as the network is only bound to grow.

It's harder for folk to undo old habits as they become more entrenched.

Nothing is ever regardless of anything :-) I would forsee issues if i'd try to add an eleven megabyte prefix-list on all devices in the network. Kind regards, Job

On Tue, 30 Jun 2015, Matsuzaki Yoshinobu wrote:

Randy Bush <randy@psg.com> wrote

...

...

A friend in AS58587 confirmed that this was caused by a configuration error - it seems like related to redistribution, and they already fixed that.

7007 all over again. do not redistribute bgp into igp. do not redistribute igp into bgp.

I also suggested them to implement BGP community based route filtering in their outbound policy. Any other suggestions or thoughts to prevent such incidents in general?

At a minimum, AS-PATH filtering of outgoing routes to just your ASN(s) and your downstream customer ASNs. Whether this is done manually, built using AS-SETs from your route registry of choice, or through some other automated means is another story. jms

On Tue, Jun 30, 2015 at 10:53:45AM -0400, Sandra Murphy wrote:

That sort of AS_PATH filtering would not have helped in this case. The AS originated the routes, it did not propagate an upstream route.

So an AS_PATH filter to just its own AS would have passed these routes.

You would need origin validation on your outbound routes. Job suggested prefix filters on outbound routes. (If you are doing prefix filters on your inbound customer links, it might be excessive caution to also prefix filter customers prefixes on outbound links? Or is it: you can never be too careful, belt-and-suspenders, measure twice, etc?)

I wouldn't consider it to be excessive caution to bring more safeguards to the game, you never know when diarrhea will strike. If you were the network causing a leak of this type, prefix filters on inbound facing your customers might not have prevented this. If you are a network providing transit to the leak originator mentioned in the above paragraph, I believe a prefix based filter could have made a big difference. Kind regards, Job

On 30/Jun/15 17:09, Job Snijders wrote:

If you are a network providing transit to the leak originator mentioned in the above paragraph, I believe a prefix based filter could have made a big difference.

And therein lies the secret sauce. Given that we've had an incident like this twice in the past month, I'm seriously concerned about the network operations of "top-tier" providers. Mark.

On 1/Jul/15 16:12, Jared Mauch wrote:

I would like to see others participate in the dialog with vendors so we don't seem to be quite an outlier with "wow, you have really large configs". The vendors haven't quite kept pace with the increase in density proportional to the number of configuration lines and it sure feels like we are the only people pushing them to improve.

I'm happy to help beat up the vendors (it's a favorite pass time of mine), but I'll be honest, we don't particularly have this problem in our network because - well, besides being a reasonably young operation - we do the opposite where we request customers to provide their prefix data, and we upload that into the network, together with strict AS_PATH lists (and max-prefix to boot). I found RPSL complicated a few years ago, and sort of put that on the back-burner. I have looked at it less in recent times because I'm putting quite a bit of work into RPKI (vendor implementation issues have been slowing me down, though, but we're looking better compared to just twelve months ago). So perhaps if there are other operators on this list using RPSL to build reasonably large configuration files that are testing the limits of router code, I echo Jared's plea to beat up your vendors and make this a priority, before we all get taken offline for the 3rd time in one year by the next boo-boo. And for anyone running Brocade, extra points if we can get them to implement RPKI as well :-)... Mark.

On 1/Jul/15 16:54, Nick Hilliard wrote:

you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations.

Honestly, I'm ambivalent about using the IRR data for prefix-list generation (even without RPSL), also because of how much junk there is in there, and also how redundant some of it really is, e.g., someone creating a /32 (IPv4) route object and yet we only accept up to a /24 (IPv4) on the actual eBGP session, e.t.c. What I'm more focused is how we can continue to scale our current system, which is much more strict, focuses on deploying customer aggregates + le 24 & le 128, instead of enumerating all possible de-aggregates that have been registered in the IRR (helps keep the configuration file small and manageable, without sacrificing reachability). And then see how we can add RPKI into the mix to make things even simpler, if at all. Mark.

That they do. Thanks for a great system, BTW! ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Nick Hilliard" <nick@foobar.org> To: "Mark Tinka" <mark.tinka@seacom.mu>, "Jared Mauch" <jared@puck.Nether.net> Cc: "North American Network Operators' Group" <nanog@nanog.org> Sent: Wednesday, July 1, 2015 10:11:13 AM Subject: Re: Route leak in Bangladesh On 01/07/2015 16:02, Mark Tinka wrote:

Honestly, I'm ambivalent about using the IRR data for prefix-list generation (even without RPSL), also because of how much junk there is in there, and also how redundant some of it really is, e.g., someone creating a /32 (IPv4) route object and yet we only accept up to a /24 (IPv4) on the actual eBGP session, e.t.c.

We went through this a couple of years ago at INEX and ended up with a provisioning system which allows the operator to only allow specific IRRDB source: entries, customisable per customer. You're right that it would be foolish to accept any IRRDB source because a lot of them are complete trash. Otherwise, it works incredibly well for us and has stopped innumerable prefix leaks and other silly misconfigs. The source code is available on github.com/inex. Lots of IXPs use it in production. Nick

On Wed 2015-Jul-01 17:02:13 +0200, Mark Tinka <mark.tinka@seacom.mu> wrote:

On 1/Jul/15 16:54, Nick Hilliard wrote:

...

you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations.

Honestly, I'm ambivalent about using the IRR data for prefix-list generation (even without RPSL), also because of how much junk there is in there, and also how redundant some of it really is, e.g., someone creating a /32 (IPv4) route object and yet we only accept up to a /24 (IPv4) on the actual eBGP session, e.t.c.

What I'm more focused is how we can continue to scale our current system, which is much more strict, focuses on deploying customer aggregates + le 24 & le 128, instead of enumerating all possible de-aggregates that have been registered in the IRR (helps keep the configuration file small and manageable, without sacrificing reachability). And then see how we can add RPKI into the mix to make things even simpler, if at all.

Orphaned/crufty data and braindead moves (e.g. someone including a large upstream's AS-SET in their own or something) aside, the deagg issue should be handled by the tools, no? We do automated prefix gen from IRR data, and I know IRRPT will aggregate the route records before spitting out the prefix list. I would have assumed that other tools do the same? Options are available to define your max prefix length and then build your filters off of that, e.g. <aggregated route object> upto /<max prefix len>. You still face issues with people people registering discontiguous blocks (e.g. network has a.b.0.0/22 but creates records for a.b.0.0/23 and a.b.3.0/24, but leaves out a.b.2.0/24), but there isn't much we can do about that without human interaction to verify the actual allocation. Also:

focuses on deploying customer aggregates + le 24 & le 128...

Jeebuz; you accept /128s? Perhaps "le 24 & le 48"?

Mark.

-- Hugo

On Wed, Jul 01, 2015 at 03:54:16PM +0100, Nick Hilliard wrote:

On 01/07/2015 15:51, Mark Tinka wrote:

...

I found RPSL complicated a few years ago, and sort of put that on the back-burner.

you probably want to ignore more rpsl constructs and depend solely on as-sets, aut-nums and route/route6 objects. RPSL is not going to live up to your expectations.

Yes, like any technology there are lots of knobs that one could use but are not recommended. You just need these few objects and life will be simple. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.

On 01/07/2015 17:03, Joe Abley wrote:

The idea of configuring this stuff from the IRR is great in terms of distributing the ops cycles in the right places, but it doesn't help with verifying that the end result isn't insane, as I think you and Mike have described on this list over the past couple of days.

that doesn't invalidate it as being part of a critical mechanism for filtering ebgp. Implemented well, it will catch 99% of problems. maxprefixes with no autorecover catches 75% of the rest. Between these two mechanisms, that's pretty good. Nick

On 1/Jul/15 16:52, Nick Hilliard wrote:

This is a strange sort of thing really. There's no reason that a compiled prefix list of 250k entries should take up much RAM in a trie structure; there's no reason that a competently written parser shouldn't be able to handle 20 megs of prefix lists / sets in a trivial amount of time and there's no reason that writing a 20 meg configuration file should take long to write to disk / flash / etc.

BIRD handles this in ultraquick time. Even recent versions of Quagga can now suck + parse 10 megs of prefix filters in a second or two and write them out in less.

But Junos / IOS / XR puke horribly. What gives?

Nick, I think the concerns are two-fold: 1. The time it takes to process the trie. 2. How much physical space there is to support the configuration. Remember some high-end Cisco routers only have 2MB of NVRAM. This could get tested with a large prefix-list configuration. Junos may not have much of a space issue since the configuration is stored on the compact flash or HDD. Trie compilation or process will be very OS-dependent, and how the vendor has chosen to optimize that operation. Mark.

On 1/Jul/15 17:04, Nick Hilliard wrote:

Naah, trie compilation is simple, particularly with a line oriented configuration like IOS (one of the worse offenders). Once the config is syntax-checked, a regexp will split it out trivially and the binary form of the data can be compiled. Even on Junos, that sort of config will be handled by lex/yacc, which is highly optimised.

Insertion / deletion of data in a patricia trie is ridiculously fast and there are a couple of bsd licensed implementations out there.

My experience around this was mostly when Cisco began introducing Turbo ACL's. There seemed to be a few problems around that time, but it was such a long time ago that I barely remember the details. That said, I'm not quite sure if there are specific issues Jared and others are facing around this in their networks. From my side, none that we have witnessed. But then again, our configurations are significantly smaller because we do not take data from the IRR. Mark.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 30/Jun/15 16:53, Sandra Murphy wrote:

That sort of AS_PATH filtering would not have helped in this case.

The AS originated the routes, it did not propagate an upstream route.

So an AS_PATH filter to just its own AS would have passed these routes.

You would need origin validation on your outbound routes. Job

suggested prefix filters on outbound routes. (If you are doing prefix filters on your inbound customer links, it might be excessive caution to also prefix filter customers prefixes on outbound links? Or is it: you can never be too careful, belt-and-suspenders, measure twice, etc?) Assuming you're running the same hardware/software across your backbone, correct prefix filters on inbound sessions to downstreams should be just fine. If those break, it's likely whatever broke them would also break them on egress to your upstreams and peers. The problem with egress prefix filters to upstreams and peers is that it's simply not scalable. Assuming you have a uniform routing policy where neighbors are all treated as eBGP sessions, then there is no real difference between upstreams, peers and other customers. Imagine having to build outbound prefix filters across your entire backbone for a uniform eBGP routing policy. Mark. -----BEGIN PGP SIGNATURE----- iQIcBAEBCAAGBQJVk4b7AAoJEGcZuYTeKm+GjPkP/1vEnL7mh0alWw+p6xCScUyH NxTYOYg1eBYUWQnGIWc+UTfZzKyr/LYbNyBF2Msf1aeNBOEb6kIY2geHUIGhOAZv DYIzggbvwWvd3X92aV76m3Nm8+z6nkDxnhYWgfefcXMofNTgHhQNKgsFp0efdDhA Mru60Cwi87apBLwY9wKYGqDtIgncKjLj92GfggimD7iwidvHZBXpKLIvPBE8sg9p aA/P9QqV2ZpVwoTtZy1Kb7B0FBogQFhPJX9RbWQUm0WwCuqMc8C7SibQMoF6hN0k rTuex7F4iPxTdvAcex/rRzIrQnDArIrMGkdOq3liQ8RG5d93Rara4NA9BgT6+ja/ idQ88lXjlBwzEEBh6k44Kg9Q686K503PK+hR8WrvETfojN8C4uD4WhUuqh3m2EPW UwJiZ8YD8oWQhLYpjdq/Rl7ozwu2ogi/ko69XuImi7f8OWscHD6QURoC0hONgLqF Rq7UgNcnOekUbTA+eP7ANFwKXNO+o9tomZ1tpmZqhNF5LLvazQFETcpEO2huQiON 2apxUiLWp3o8qCYKlvfUvREeF7fXaosgjXviWkjbdZc0v6hNjpd+M2uFPTz9CDgx PF9R+MzCu9C+gcfZRv4veY/ZFMxNxTNhOxppx69uyTG9+XCRXb5evjoV3VZPi/Qx RPUZQ1Ekzl0gAE7D4US6 =VZEA -----END PGP SIGNATURE-----