Thank you Nick. Graham Johnston Manager, Network Services Westman Communications Group 1906 Park Avenue | Brandon, MB | R7B 0R9 204-717-2829 |     johnstong@westmancom.com             -----Original Message----- From: Nick Hilliard <nick@foobar.org> Sent: March 27, 2019 4:50 PM To: Graham Johnston <johnstong@westmancom.com> Cc: nanog@nanog.org Subject: Re: Advertisement of Equinix Chicago IX Subnet CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe. Graham Johnston wrote on 27/03/2019 21:36:

What am I doing that isn't best practices that would have prevented this?

you're setting the next-hop of the prefixes learned at the IXP to be your own IP address from the IXP subnet (i.e. 208.115.136.0/23). When your routers learn this address from an external source, that is preferred to your internal OSPF route. Ergo your IX traffic is sent out via transit. There are two things you should do: 1. change the bgp distance for ebgp to be higher than all your IGPs. On a cisco router, you would use something like: router bgp xxx address-family ipv4 distance bgp 200 200 200 address-family ipv6 distance bgp 200 200 200 2. use next-hop-self on internal ibgp sessions to ensure that when you redistribute the eBGP routes learned from your IX towards the internals of your network, the next-hop address is set to be the loopback address of your peering router. I.e. you remove the requirement for your internal network to know anything about the IXP address range. Nick

I have a policy applied to my upstreams and peers to deny the IXP's LANs were connected to. I don't think of any reason to learn these routes from someone else's network. On Wed, Mar 27, 2019 at 7:44 PM Cummings, Chris <ccummings@coeur.com> wrote:

Not too sure about your topology, but I’ve had something similar bite me, so we typically put a prefix list inbound to deny receiving our internal prefixes from our peers. This probably doesn’t work as well if your network is less “eyeballish” than ours, however.

/chris

On Wed, Mar 27, 2019 at 4:37 PM -0500, "Graham Johnston" < johnstong@westmancom.com> wrote:

This afternoon at around 12:17 central time today we began learning the

...

subnet for the Equinix IX in Chicago via a transit provider; we are on the IX as well. The subnet in question is 208.115.136.0/23. Using stat.ripe.net <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net&c=E,1,HdSVqYeR7jgCV-Dur66y05aHEW-BSduVIIHYHrXZ1P6qOt3fa684wgoFR9CoVMgOpEaWMO0lwDjZkSR-n80nd7Rvcqp4MKodaGyrIDIjEhtPXiDie1SaYsyZJ9ed&typo=1> I can see that this subnet is also being learned by others, see the snip below. On our network this caused a nasty routing loop until we figured out what was wrong. My current best understanding is that because the route was learned via eBGP it trumped the OSPF learned route. As soon as I filtered the advertisement from my transit provider everything returned to normal. What am I doing that isn’t best practices that would have prevented this?

Thanks,

graham

RIPE Info

*1* RRCs see *1* peers announcing *208.115.136.0/23 <http://208.115.136.0/23>* originated by *AS32703* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2fAS32703&c=E,1,TOo4BxuZBilA6dEeEsyArFdQvYciFoXF4XjZNU4NqyzUFPawLd-3hzV5XwlwfBLIcVRBns_GfdJCxNBaU2dYqDWisxgCxwxRPMoTfXq-TRSDQa_BgAvqRg,,&typo=1>

· ▼RRC00 in *Amsterdam, Netherlands* sees *1* ASN orginating *208.115.136.0/23 <http://208.115.136.0/23>*.AS32703

o ▼*AS32703 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2fAS32703&c=E,1,LPjxozPn3-dGOA9bDJB081OscbzusnfrxssBxyMbOyunZUcNyeibk_RHV8UYO3Fw77TpLU9yRsywr6KjrmyXWgKk4DQ7XRSgr1_W1SNgkfA,&typo=1>* is seen as the origin by *1* peer.192.102.254.1

§ ▼*192.102.254.1 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2f192.102.254.1&c=E,1,fW5rffxlYLANo-g3GopSdMyHH2oIqoulMERJOjPrrdRL4Z8602v0WhaVuS6ignBPzPDgh4S05V55mLAGu_OFn1TzFyYcCpMMzTgH1ejtJmILMrcaDQDn&typo=1>* is announcing route *AS395152* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2fAS395152&c=E,1,I_iMCTImXK-T7Vj5VALSLMN6lo0N3-N2qYG7QlBHNK8oXNmPQnsp4zJy424NN2Y8z2WxSBIfaPSkLoibtnClWliVcGMhdMDsIewEnAgiZaRITyPjKA,,&typo=1> *AS63297* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2fAS63297&c=E,1,V7oySywzIc8rSc64KXotimJVgetH1G5VqJoedNuNjm9JbOYDh8qrdMlVKD12tKJtJ4STBfu9kLFuBXInbfko44ryiCz5Gy2CztDGyYXF4HJW6Jm3uPvJgOUAfTc,&typo=1> *AS6327* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2fAS6327&c=E,1,4wIITl8037dr3SSHzQmbAIwgiFe3X75-DkFAlERAGWEFjFROhFPMC2c3IGy_vChkNN-YI2OoobMvhOUKjiV9mt69N8kXl_RTvv22nZHKLJkYc59V&typo=1> *AS36280* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2fAS36280&c=E,1,_jAHKYzgyGwMDV4H1HRk1FK3bV5j_t6dSn2YfYhnhLBYub5v33-ryduZ34KVZYUy19lhSRThf8TUnUT_6V35nTMLw6SCXqY0S8bggDBKvYUg&typo=1> *AS32703* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstat.ripe.net%2fAS32703&c=E,1,RAfxFbCQUejEFosUxg2dek9Ke5qatnE5GGjP6p2ovv1XL6hN77GlayI0Nm5jA_jRLCxzzaZQUdABGyy7HlA7bi93SIbytUbKx_49kJPC168,&typo=1> .

§ Origin: IGP

§ Next Hop: 192.102.254.1

§ Peer: 192.102.254.1

§ Community: 63297:1000

§ AS Path: 395152 63297 6327 36280 32703

§ Last Updated: 2019-03-27T17:17:19

Route-views

route-views.chicago.routeviews.org <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2froute-views.chicago.routeviews.org&c=E,1,E0igNv77g9AAa2d6Uaxl8p-e1C0XIX7IzMRDUURg85DkFqIFTzckgumVyHoZqhybvGEz7rGGqi_cSc8KzJW5xx3nxdSBkfe6z_hdXiip8re7qfTpyjS1o2wzcvLw&typo=1>> show ip bgp 208.115.136.0

BGP routing table entry for 208.115.136.0/23

Paths: (1 available, best #1, table Default-IP-Routing-Table)

Not advertised to any peer

32709 32703

208.115.136.134 from 208.115.136.134 (63.134.128.248)

Origin IGP, localpref 100, valid, external, best

AddPath ID: RX 0, TX 64414249

Last update: Wed Mar 27 17:16:09 2019

* christopher.morrell.nanog@gmail.com (Christopher Morrell) [Thu 28 Mar 2019, 14:35 CET]:

I've been bit by this in the past at two different exchanges. I too have a policy applied to deny IXP LANs from upstreams and peers. It would be nice if there was a list of all IXP LANs somewhere that we could generically add to all upstream and peers.

I like Nick Hilliard's posted solution much better than creating static bogon lists that people will eventually forget about. -- Niels.