Greetings, We are aware of this issue and as is usual we filter customers based on their registered routes. This creates some unique challenges that we have been speaking about publicly and privately with various groups. I have started the process (yay telco-speak) to fix this. It would be helpful if networks would take a look at what routes they have registered in the various IRRs as well as if their AS-SETs expand out to something quite large. We have seen many customers import objects that then import their other upstream networks. We have found the IRR Explorer tool helpful to look at who has registered our IP space and to police these registrations with the various IRRs out there. http://irrexplorer.nlnog.net/ http://irrexplorer.nlnog.net/prefix/184.105.213.86 The stability of the routing ecosystem is something that I personally care a lot about and have privately given Mike and others my cell number to allow them to follow-up. As is often operators end up chasing problems after the fact, and this appears to be no exception. *sigh* - Jared

On Jun 29, 2015, at 5:18 PM, Mike Leber <mleber@he.net> wrote:

NTT's customer Sofia Connect leaked our routes to NTT. NTT accepted these routes instead of properly filtering their customer announcements. As a network of non-trivial size, announcing over 75,000 customer routes which is nearly 15% of the IPv4 routing table, we'd expect the common courtesy of having our ASN included in their customer facing AS-PATH filters, as we extend this same courtesy to other networks of this size (such as AS2914).

Mike.

On 6/29/15 2:04 PM, Jim Popovitch wrote:

...

Hello,

I haven't seen anything to explain this, so I'm asking a larger audience. Did anyone notice any unusual NTT or HE routing this AM?

Here's what I saw:

2.|-- xe-0-1-0-17.r04.atlnga05.us.bb.gin.ntt.net 0.0% 20 0.8 0.7 0.6 0.9 0.1 3.|-- ae-2.r20.atlnga05.us.bb.gin.ntt.net 0.0% 20 4.6 6.2 0.5 13.6 4.8 4.|-- ae-4.r22.asbnva02.us.bb.gin.ntt.net 0.0% 20 15.3 15.0 13.9 15.8 0.7 5.|-- ae-4.r20.frnkge04.de.bb.gin.ntt.net 0.0% 20 127.3 106.7 98.5 127.3 11.1 6.|-- ae-2.r02.frnkge04.de.bb.gin.ntt.net 0.0% 20 126.8 126.0 125.7 126.8 0.2 7.|-- ae-1.r00.sofibu01.bg.bb.gin.ntt.net 0.0% 20 131.1 130.0 128.7 131.4 1.2 8.|-- 83.217.227.42 80.0% 20 148.5 146.0 144.2 148.5 2.0 9.|-- ip-48-93.sofia-connect.net 90.0% 20 184.5 163.8 143.1 184.5 29.3 10.|-- ??? 100.0 20 0.0 0.0 0.0 0.0 0.0 11.|-- 10ge5-4.core1.vie1.he.net 75.0% 20 160.7 150.4 143.9 160.7 6.3 12.|-- 10ge1-4.core1.prg1.he.net 80.0% 20 158.4 159.5 157.9 161.1 1.6 13.|-- 10ge10-12.core1.fra1.he.net 75.0% 20 154.5 159.2 145.9 174.4 10.7 14.|-- 100ge5-2.core1.par2.he.net 75.0% 20 187.9 172.9 157.1 187.9 11.1 15.|-- 100ge7-1.core1.nyc4.he.net 78.9% 19 147.2 146.2 144.6 147.5 1.4 16.|-- 100ge7-2.core1.chi1.he.net 78.9% 19 165.6 172.1 165.6 183.5 8.0 17.|-- 10ge15-2.core1.den1.he.net 89.5% 19 201.3 204.7 201.3 208.1 4.8

-Jim P.

On Tue, Jun 30, 2015 at 03:24:02AM +0000, Faisal Imtiaz wrote:

Hi Jared,

This is neat !, for someone who recently started working the IRR's, I can tell you that it has been very difficult finding all info in one location.

What you shared is pretty neat !, and I would like to clean up the records associated with our prefixes.

Can you suggest some practical tips on getting older 'stale' records cleaned up from the different registries ? (i.e. records created for us by others, in a former time-frame).

I've not found a great method as it usually involves a lot of either happy or grumpy sounding emails to addresses that may bounce quite a lot. We have had good luck getting RADB to clean out older entries. I recently helped someone announce some IP blocks from the NTT network and there are many crusty IRR objects that seem impossible to clean up because the IRR is feels like first-in-never-out input. http://irrexplorer.nlnog.net/prefix/203.10.78.0/24 - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.

I was thinking that when I posted yesterday. These were announcements from a peer, not customer routes. We are lowering our max prefix limits on many peers as a result of this. We are also going towards more prefix filtering on peers beyond bogons and martians. Mike. On 6/30/15 2:19 AM, Randy Bush wrote:

...

NTT's customer Sofia Connect leaked our routes to NTT. NTT accepted these routes instead of properly filtering their customer announcements. As a network of non-trivial size, announcing over 75,000 customer routes which is nearly 15% of the IPv4 routing table, we'd expect the common courtesy of having our ASN included in their customer facing AS-PATH filters, as we extend this same courtesy to other networks of this size (such as AS2914). sometimes the goddesses have a sense of humor

At Tue, 30 Jun 2015 10:27:21 +0200, Grzegorz Janoszka wrote:

...

We have just received alert from bgpmon that AS58587 Fiber @ Home Limited has hijacked most of our (AS43996) prefixes and Hurricane Electric gladly accepted them. randy

On 6/30/15 3:02 PM, Tore Anderson wrote:

* Mike Leber

...

I was thinking that when I posted yesterday.

These were announcements from a peer, not customer routes.

We are lowering our max prefix limits on many peers as a result of this.

We are also going towards more prefix filtering on peers beyond bogons and martians. Hi Mike,

You're not mentioning RPKI here. Any particular reason why not?

If I understand correctly, in today's leak the origin AS was changed/reset, so RPKI ought to have saved the day. (At least Grzegorz' day, considering that 33 of AS43996's prefixes are covered by ROAs.)

Yes, we will incorporate RPKI into how we build our prefix filters for peers as we improve our tools. Currently this will involve some amount of prefix list compression due to the limits of current hardware and the need to still have BGP converge. As Job Snijders said, "I would forsee issues if i'd try to add an eleven megabyte prefix-list on all devices in the network.". Mike.

On Tue, Jun 30, 2015 at 03:32:42PM -0700, Ca By wrote:

It is NTT that would have mitigated this issue if they deployed and enforcer rpki, right?

No, NTT deploying RPKI would not have helped in yesterday's issue. But, RPKI could've made a difference in today's Bangladesh leak, even if RPKI validation was not implemented by the direct upstream with propagated the leak. Kind regards, Job

On Tue, Jun 30, 2015 at 05:40:03PM -0500, Jared Mauch wrote:

We have been pushing large configurations to devices. You can check my slides from the London IEPG meeting.

These are the slides: http://iepg.org/2014-03-02-ietf89/ietf89_iepg_jmauch.pdf

When 96% of your config is prefix filters we are sure trying.

I ask others to encourage your vendors to make this a priority as we have faced a number of issues in this area and have been waiting quite some time for vendor resolution.

I'd like to add, that our average router config, since then has grown with an extra 100k lines compared to those 2013 statistics. Kind regards, Job

On Wed, Jul 01, 2015 at 12:02:40AM +0200, Tore Anderson wrote:

...

I was thinking that when I posted yesterday.

These were announcements from a peer, not customer routes.

We are lowering our max prefix limits on many peers as a result of this.

We are also going towards more prefix filtering on peers beyond bogons and martians.

You're not mentioning RPKI here. Any particular reason why not?

If I understand correctly, in today's leak the origin AS was changed/reset, so RPKI ought to have saved the day. (At least Grzegorz' day, considering that 33 of AS43996's prefixes are covered by ROAs.)

This assessment is correct, however there might be some constraints in play with regard to RPKI, which are not really related to RPKI itself, which prohibit meaningful deployment. I've seen a few obstacles myself: - equipment might not support the RTR protocol to validate announcements against the cache validator - Legal obstacles in obtaining the anchors from all RIRs - when not using the RTR protocol but generating prefix-list filters based on RPKI data, the devices might not support sufficient entries. Would be good if other people share obstacles, and possibly, the methods they used to overcome those. I'll count "not using brocade" as a valid method. Kind regards, Job

On Wed, Jul 01, 2015 at 09:36:34AM +0900, Randy Bush wrote:

...

- when not using the RTR protocol but generating prefix-list filters based on RPKI data, the devices might not support sufficient entries.

because the rpki generated acls are bigger and heavier than those in the irr. and they have trans-fats.

I don't consider RPKI generated ACLs a 1 to 1 replacement for IRR based filters. They might be used as supplement to each other. Kind regards, Job

On 1/Jul/15 00:02, Tore Anderson wrote:

You're not mentioning RPKI here. Any particular reason why not?

If I understand correctly, in today's leak the origin AS was changed/reset, so RPKI ought to have saved the day. (At least Grzegorz' day, considering that 33 of AS43996's prefixes are covered by ROAs.)

It certainly would have. BGPmon was awash with alarms about Origin Validation violations for our prefixes that were originated by the offending network yesterday. If HE implemented Origin Validation, they'd have dropped these routes assuming that was their policy. Mark.