RFC 9234 route leak prevention in the wild!
Dear all, I'd like to share an update on RFC 9234 deployment. RFC 9234 titled "BGP Open Policy" aka the "Only-To-Customer" (OTC) BGP Path Attribute is an anti-route-leak mechanism which is *NOT* based on RPKI! (yes ... routing security is more than just RPKI! :-) The basic idea of 9234 is that BGP routers (based on their role in the Gao-Rexford inter-domain routing model) attach a special BGP Path attribute, or take action based on the presence and contents of this BGP Path attribute - with the intention to constrain a route's propagation radius to just the downstream customer cone of the neighboring ASN. Most operators will intuitively understand that any route propagating through multiple IX route servers operated by different IXPs is a route leak: ``` IXP_1 IXP_2 / \ / \ / \ / \ ISP_A ISP_B ISP_C (receives) (leaker) (originates) ``` (figure 1. propagation from right to left; leak scenario) In the above example, ISP_A originates a route towards IXP_1's route servers, IXP_1 propagates the route to ISP_B (so far so good); but for one reason or another ISP_B subsequently continues propagation of the route towards IXP_2's route servers, who in turn propagate it to ISP_C. ISP_B is forwarding IP traffic between ISP_A and ISP_C for zero revenue. ISP_A and ISP_C are probably not expecting ISP_B to be in the middle. This situation can happen as a result of a misconfiguration in ISP_B's equipment, even when all participants use IRR & RPKI ROV to attempt to mitigate the worst routing incidents. What does it matter / impact ============================ Calgary-based YYCIX deployed RFC9234 support in late 2022/early 2023 using OpenBGPD; and FranceIX deployed support using BIRD in Q2 2024. Both IXPs configured their route servers to reject BGP routes that have an OTC attribute attached, and to attach an OTC attribute when propagating routes to the Route Server's peers. As it happens to be, currently (Mon Sep 2 12:26:50 UTC 2024) a small route leak is happening involving both YYCIX and FranceIX Route Servers via an ISP connected to both IXP's Route Servers; with the leak being stopped at YYCIX thanks to RFC 9234! Appendix A contains a table of leaked IPv4 routes. Let's zoom in on 1 entry: ``` $ bgpctl show rib 157.185.154.0/24 detail BGP routing table entry for 157.185.154.0/24 6939 38040 54994 Nexthop 206.126.225.20 (via 206.126.225.20) Neighbor 206.126.225.20 (216.218.252.194) Origin IGP, metric 1911, localpref 100, weight 0, ovs not-found, avs unknown, external, otc leak Last update: 11:58:08 ago Communities: 0:2906 0:16265 0:16276 0:18638 0:41690 0:48641 0:49029 Ext. Communities: ovs not-found Large Communities: 53339:11:1 53339:11:3 Aggregator: 54994 [163.171.131.254] OTC: 51706 ``` (figure 2. inspecting an leaked route using OpenBGPD's CLI) In figure 2. one can see the route is marked as 'otc leak', this was made possible because FranceIX's route server's attached the OTC attribute with the ASN value set to their Route Server's ASN (51706). ``` YYCIX FranceIX . x <adds OTC> \ . \ / \ ISP_A 6939_38040 54994 ``` (figure 3. right to left: real world example of blocked leak) In figure 3 AS 54994 originates 157.185.154.0/24 and propagates a route towards the FranceIX route servers. FranceIX accepts this route (probably because an IRR route object exists) and propagates it onward with the "Only-To-Customer" attribute set to 51706. The route is received by AS 38040, who appear to propagate the route to their upstream 6939. << An AS 38040 router is likely misconfigured! >> Then 6939 sends its customer's routes to the YYCIX route server, but the YYCIX route server recognizes that the route already passed through a 'valley' and thus considers this a leak; and blocks further propagation towards ISP_A. Impact with partial deployment ============================== RFC 9234 is an easy mechanism to configure and debug for both small & large network operators and IXP route server operators. In the above scenario YYCIX and FranceIX are likely the only 2 entities in the entire AS path which support RFC 9234; but we're already seeing leaks being blocked despite partial deployment! It's not hard to imagine that many IX-to-IX route leaks can be blocked with only the IXP operators themselves enabling RFC 9234 support. The world's most populair RS implementations (BIRD and OpenBGPD) already support RFC 9234! Tens of thousands of IX customers would enjoy the benefits of a few hundred IXPs taking action. RFC 9234 has no dependency on the RPKI, this means tha What can you do? ================ Just ask your vendors (your hardware routers and IXPs) to implement and deploy RFC 9234! :-) The more people ask, the more it'll bubble to the top of the priority list. The cost of implementing & deploying RFC 9234 is excellent bang for buck. Closing words ============= Shout out to our friends at FranceIX and MSK-IX for being amongst the first IXPs to deploy RFC 9234! Your effort helped reduce the potential impact of today's route leak! Kind regards, Job (YYCIX volunteer) Appendix A: ----------- Vantage point: YYCIX Route Server 1 (rs1.yycix.ca) Timestamp: Mon Sep 2 12:31:50 UTC 2024 Destination Prefix AS_PATH 102.164.129.0/24 6939 37613 6758 37649 i 102.164.139.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 102.164.140.0/24 6939 37613 6758 37649 i 102.164.141.0/24 6939 37613 6758 37649 i 102.164.182.0/24 6939 37613 6758 37649 i 154.65.33.0/24 6939 37613 6758 37649 i 154.65.34.0/24 6939 37613 6758 37649 i 154.65.35.0/24 6939 37613 6758 37649 i 154.65.36.0/24 6939 37613 6758 37649 i 154.65.37.0/24 6939 37613 6758 37649 i 154.65.38.0/24 6939 37613 6758 37649 i 154.65.39.0/24 6939 37613 6758 37649 i 154.72.35.0/24 6939 37613 37100 37027 37063 327721 i 157.185.151.0/24 6939 38040 54994 i 157.185.154.0/24 6939 38040 54994 i 163.171.164.0/24 6939 38040 54994 i 192.150.250.0/23 6939 4651 4618 4618 63529 4621 3836 i 196.50.8.0/24 6939 37613 6758 37649 i 196.50.9.0/24 6939 37613 6758 37649 i 196.50.11.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.13.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.14.0/24 6939 37613 6758 37649 i 2001:67c:15f4::/48 6939 41103 i 2401:c500:fd08::/48 6939 4637 38040 54994 i 2a01:53c0:ff03::/48 6939 4637 38040 54994 i 2a01:53c0:ff0f::/48 6939 4637 38040 54994 i 2a01:58c0::/32 6939 16347 42487 i 2a03:8920::/32 6939 41103 i 2a03:d602::/31 6939 16347 42487 i 2a03:d606::/31 6939 16347 42487 i 2a0d:ee00::/32 6939 16347 42487 i 2a0e:5b80::/29 6939 16347 42487 i 2a0e:e080::/32 6939 16347 42487 i 2a0f:c540::/29 6939 16347 42487 i
On 2024-09-02 08:33, Job Snijders via NANOG wrote:
I'd like to share an update on RFC 9234 deployment. RFC 9234 titled "BGP Open Policy" aka the "Only-To-Customer" (OTC) BGP Path Attribute is an anti-route-leak mechanism which is *NOT* based on RPKI! (yes ... routing security is more than just RPKI! :-)
Calgary-based YYCIX deployed RFC9234 support in late 2022/early 2023 using OpenBGPD; and FranceIX deployed support using BIRD in Q2 2024. Both IXPs configured their route servers to reject BGP routes that have an OTC attribute attached, and to attach an OTC attribute when propagating routes to the Route Server's peers.
Do they have configuration snippets available anywhere? With BIRD, it looks like a global "local role rs_server;" might be all that is needed. -- Richard
I don’t know whether to say it’s ironic or not, but reading this, I was thinking “IX route servers may not be involved in the propagation, but I know from past experience that HE intentionally propagates some peer routes to other peers.” And then your example leak has HE in the path. This may well be intentional and not a “leak” for some definitions of the word. Sent from my iPhone
On Sep 2, 2024, at 9:34 AM, Job Snijders via NANOG <nanog@nanog.org> wrote:
Dear all,
I'd like to share an update on RFC 9234 deployment. RFC 9234 titled "BGP Open Policy" aka the "Only-To-Customer" (OTC) BGP Path Attribute is an anti-route-leak mechanism which is *NOT* based on RPKI! (yes ... routing security is more than just RPKI! :-)
The basic idea of 9234 is that BGP routers (based on their role in the Gao-Rexford inter-domain routing model) attach a special BGP Path attribute, or take action based on the presence and contents of this BGP Path attribute - with the intention to constrain a route's propagation radius to just the downstream customer cone of the neighboring ASN.
Most operators will intuitively understand that any route propagating through multiple IX route servers operated by different IXPs is a route leak:
``` IXP_1 IXP_2 / \ / \ / \ / \ ISP_A ISP_B ISP_C (receives) (leaker) (originates) ``` (figure 1. propagation from right to left; leak scenario)
In the above example, ISP_A originates a route towards IXP_1's route servers, IXP_1 propagates the route to ISP_B (so far so good); but for one reason or another ISP_B subsequently continues propagation of the route towards IXP_2's route servers, who in turn propagate it to ISP_C. ISP_B is forwarding IP traffic between ISP_A and ISP_C for zero revenue. ISP_A and ISP_C are probably not expecting ISP_B to be in the middle. This situation can happen as a result of a misconfiguration in ISP_B's equipment, even when all participants use IRR & RPKI ROV to attempt to mitigate the worst routing incidents.
What does it matter / impact ============================
Calgary-based YYCIX deployed RFC9234 support in late 2022/early 2023 using OpenBGPD; and FranceIX deployed support using BIRD in Q2 2024. Both IXPs configured their route servers to reject BGP routes that have an OTC attribute attached, and to attach an OTC attribute when propagating routes to the Route Server's peers.
As it happens to be, currently (Mon Sep 2 12:26:50 UTC 2024) a small route leak is happening involving both YYCIX and FranceIX Route Servers via an ISP connected to both IXP's Route Servers; with the leak being stopped at YYCIX thanks to RFC 9234! Appendix A contains a table of leaked IPv4 routes.
Let's zoom in on 1 entry:
``` $ bgpctl show rib 157.185.154.0/24 detail
BGP routing table entry for 157.185.154.0/24 6939 38040 54994 Nexthop 206.126.225.20 (via 206.126.225.20) Neighbor 206.126.225.20 (216.218.252.194) Origin IGP, metric 1911, localpref 100, weight 0, ovs not-found, avs unknown, external, otc leak Last update: 11:58:08 ago Communities: 0:2906 0:16265 0:16276 0:18638 0:41690 0:48641 0:49029 Ext. Communities: ovs not-found Large Communities: 53339:11:1 53339:11:3 Aggregator: 54994 [163.171.131.254] OTC: 51706 ``` (figure 2. inspecting an leaked route using OpenBGPD's CLI)
In figure 2. one can see the route is marked as 'otc leak', this was made possible because FranceIX's route server's attached the OTC attribute with the ASN value set to their Route Server's ASN (51706).
``` YYCIX FranceIX . x <adds OTC> \ . \ / \ ISP_A 6939_38040 54994 ``` (figure 3. right to left: real world example of blocked leak)
In figure 3 AS 54994 originates 157.185.154.0/24 and propagates a route towards the FranceIX route servers. FranceIX accepts this route (probably because an IRR route object exists) and propagates it onward with the "Only-To-Customer" attribute set to 51706. The route is received by AS 38040, who appear to propagate the route to their upstream 6939. << An AS 38040 router is likely misconfigured! >> Then 6939 sends its customer's routes to the YYCIX route server, but the YYCIX route server recognizes that the route already passed through a 'valley' and thus considers this a leak; and blocks further propagation towards ISP_A.
Impact with partial deployment ==============================
RFC 9234 is an easy mechanism to configure and debug for both small & large network operators and IXP route server operators. In the above scenario YYCIX and FranceIX are likely the only 2 entities in the entire AS path which support RFC 9234; but we're already seeing leaks being blocked despite partial deployment!
It's not hard to imagine that many IX-to-IX route leaks can be blocked with only the IXP operators themselves enabling RFC 9234 support. The world's most populair RS implementations (BIRD and OpenBGPD) already support RFC 9234! Tens of thousands of IX customers would enjoy the benefits of a few hundred IXPs taking action. RFC 9234 has no dependency on the RPKI, this means tha
What can you do? ================
Just ask your vendors (your hardware routers and IXPs) to implement and deploy RFC 9234! :-) The more people ask, the more it'll bubble to the top of the priority list. The cost of implementing & deploying RFC 9234 is excellent bang for buck.
Closing words =============
Shout out to our friends at FranceIX and MSK-IX for being amongst the first IXPs to deploy RFC 9234! Your effort helped reduce the potential impact of today's route leak!
Kind regards,
Job (YYCIX volunteer)
Appendix A: -----------
Vantage point: YYCIX Route Server 1 (rs1.yycix.ca) Timestamp: Mon Sep 2 12:31:50 UTC 2024
Destination Prefix AS_PATH 102.164.129.0/24 6939 37613 6758 37649 i 102.164.139.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 102.164.140.0/24 6939 37613 6758 37649 i 102.164.141.0/24 6939 37613 6758 37649 i 102.164.182.0/24 6939 37613 6758 37649 i 154.65.33.0/24 6939 37613 6758 37649 i 154.65.34.0/24 6939 37613 6758 37649 i 154.65.35.0/24 6939 37613 6758 37649 i 154.65.36.0/24 6939 37613 6758 37649 i 154.65.37.0/24 6939 37613 6758 37649 i 154.65.38.0/24 6939 37613 6758 37649 i 154.65.39.0/24 6939 37613 6758 37649 i 154.72.35.0/24 6939 37613 37100 37027 37063 327721 i 157.185.151.0/24 6939 38040 54994 i 157.185.154.0/24 6939 38040 54994 i 163.171.164.0/24 6939 38040 54994 i 192.150.250.0/23 6939 4651 4618 4618 63529 4621 3836 i 196.50.8.0/24 6939 37613 6758 37649 i 196.50.9.0/24 6939 37613 6758 37649 i 196.50.11.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.13.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.14.0/24 6939 37613 6758 37649 i 2001:67c:15f4::/48 6939 41103 i 2401:c500:fd08::/48 6939 4637 38040 54994 i 2a01:53c0:ff03::/48 6939 4637 38040 54994 i 2a01:53c0:ff0f::/48 6939 4637 38040 54994 i 2a01:58c0::/32 6939 16347 42487 i 2a03:8920::/32 6939 41103 i 2a03:d602::/31 6939 16347 42487 i 2a03:d606::/31 6939 16347 42487 i 2a0d:ee00::/32 6939 16347 42487 i 2a0e:5b80::/29 6939 16347 42487 i 2a0e:e080::/32 6939 16347 42487 i 2a0f:c540::/29 6939 16347 42487 i
Separate from the goodness that is RFC 9234, Regarding the comment "HE intentionally propagates some peer routes to other peers", in all such cases involving Hurricane Electric, one side or the other is configured as a customer and is receiving some form of IP Transit from Hurricane Electric. In no case do we tag a route server as a customer, so if Hurricane Electric is announcing route to a route server, then that route was learned via a customer IP Transit session and tagged as a customer route. Similarly if we pass a route learned from a route server to another network, that network is receiving IP Transit from Hurricane Electric. Related to this topic is Hurricane Electric just implemented "Never via route servers" (the peeringdb flag) as part of our algorithm for prefix filtering. I'll send an email with a fresh subject about that, to avoid hijacking Job's announcement. Mike. On 9/5/24 7:28 AM, Jon Lewis wrote:
I don’t know whether to say it’s ironic or not, but reading this, I was thinking “IX route servers may not be involved in the propagation, but I know from past experience that HE intentionally propagates some peer routes to other peers.”
And then your example leak has HE in the path. This may well be intentional and not a “leak” for some definitions of the word.
Sent from my iPhone
On Sep 2, 2024, at 9:34 AM, Job Snijders via NANOG<nanog@nanog.org> wrote:
Dear all,
I'd like to share an update on RFC 9234 deployment. RFC 9234 titled "BGP Open Policy" aka the "Only-To-Customer" (OTC) BGP Path Attribute is an anti-route-leak mechanism which is *NOT* based on RPKI! (yes ... routing security is more than just RPKI! :-)
The basic idea of 9234 is that BGP routers (based on their role in the Gao-Rexford inter-domain routing model) attach a special BGP Path attribute, or take action based on the presence and contents of this BGP Path attribute - with the intention to constrain a route's propagation radius to just the downstream customer cone of the neighboring ASN.
Most operators will intuitively understand that any route propagating through multiple IX route servers operated by different IXPs is a route leak:
``` IXP_1 IXP_2 / \ / \ / \ / \ ISP_A ISP_B ISP_C (receives) (leaker) (originates) ``` (figure 1. propagation from right to left; leak scenario)
In the above example, ISP_A originates a route towards IXP_1's route servers, IXP_1 propagates the route to ISP_B (so far so good); but for one reason or another ISP_B subsequently continues propagation of the route towards IXP_2's route servers, who in turn propagate it to ISP_C. ISP_B is forwarding IP traffic between ISP_A and ISP_C for zero revenue. ISP_A and ISP_C are probably not expecting ISP_B to be in the middle. This situation can happen as a result of a misconfiguration in ISP_B's equipment, even when all participants use IRR & RPKI ROV to attempt to mitigate the worst routing incidents.
What does it matter / impact ============================
Calgary-based YYCIX deployed RFC9234 support in late 2022/early 2023 using OpenBGPD; and FranceIX deployed support using BIRD in Q2 2024. Both IXPs configured their route servers to reject BGP routes that have an OTC attribute attached, and to attach an OTC attribute when propagating routes to the Route Server's peers.
As it happens to be, currently (Mon Sep 2 12:26:50 UTC 2024) a small route leak is happening involving both YYCIX and FranceIX Route Servers via an ISP connected to both IXP's Route Servers; with the leak being stopped at YYCIX thanks to RFC 9234! Appendix A contains a table of leaked IPv4 routes.
Let's zoom in on 1 entry:
``` $ bgpctl show rib 157.185.154.0/24 detail
BGP routing table entry for 157.185.154.0/24 6939 38040 54994 Nexthop 206.126.225.20 (via 206.126.225.20) Neighbor 206.126.225.20 (216.218.252.194) Origin IGP, metric 1911, localpref 100, weight 0, ovs not-found, avs unknown, external, otc leak Last update: 11:58:08 ago Communities: 0:2906 0:16265 0:16276 0:18638 0:41690 0:48641 0:49029 Ext. Communities: ovs not-found Large Communities: 53339:11:1 53339:11:3 Aggregator: 54994 [163.171.131.254] OTC: 51706 ``` (figure 2. inspecting an leaked route using OpenBGPD's CLI)
In figure 2. one can see the route is marked as 'otc leak', this was made possible because FranceIX's route server's attached the OTC attribute with the ASN value set to their Route Server's ASN (51706).
``` YYCIX FranceIX . x <adds OTC> \ . \ / \ ISP_A 6939_38040 54994 ``` (figure 3. right to left: real world example of blocked leak)
In figure 3 AS 54994 originates 157.185.154.0/24 and propagates a route towards the FranceIX route servers. FranceIX accepts this route (probably because an IRR route object exists) and propagates it onward with the "Only-To-Customer" attribute set to 51706. The route is received by AS 38040, who appear to propagate the route to their upstream 6939. << An AS 38040 router is likely misconfigured! >> Then 6939 sends its customer's routes to the YYCIX route server, but the YYCIX route server recognizes that the route already passed through a 'valley' and thus considers this a leak; and blocks further propagation towards ISP_A.
Impact with partial deployment ==============================
RFC 9234 is an easy mechanism to configure and debug for both small & large network operators and IXP route server operators. In the above scenario YYCIX and FranceIX are likely the only 2 entities in the entire AS path which support RFC 9234; but we're already seeing leaks being blocked despite partial deployment!
It's not hard to imagine that many IX-to-IX route leaks can be blocked with only the IXP operators themselves enabling RFC 9234 support. The world's most populair RS implementations (BIRD and OpenBGPD) already support RFC 9234! Tens of thousands of IX customers would enjoy the benefits of a few hundred IXPs taking action. RFC 9234 has no dependency on the RPKI, this means tha
What can you do? ================
Just ask your vendors (your hardware routers and IXPs) to implement and deploy RFC 9234! :-) The more people ask, the more it'll bubble to the top of the priority list. The cost of implementing & deploying RFC 9234 is excellent bang for buck.
Closing words =============
Shout out to our friends at FranceIX and MSK-IX for being amongst the first IXPs to deploy RFC 9234! Your effort helped reduce the potential impact of today's route leak!
Kind regards,
Job (YYCIX volunteer)
Appendix A: -----------
Vantage point: YYCIX Route Server 1 (rs1.yycix.ca) Timestamp: Mon Sep 2 12:31:50 UTC 2024
Destination Prefix AS_PATH 102.164.129.0/24 6939 37613 6758 37649 i 102.164.139.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 102.164.140.0/24 6939 37613 6758 37649 i 102.164.141.0/24 6939 37613 6758 37649 i 102.164.182.0/24 6939 37613 6758 37649 i 154.65.33.0/24 6939 37613 6758 37649 i 154.65.34.0/24 6939 37613 6758 37649 i 154.65.35.0/24 6939 37613 6758 37649 i 154.65.36.0/24 6939 37613 6758 37649 i 154.65.37.0/24 6939 37613 6758 37649 i 154.65.38.0/24 6939 37613 6758 37649 i 154.65.39.0/24 6939 37613 6758 37649 i 154.72.35.0/24 6939 37613 37100 37027 37063 327721 i 157.185.151.0/24 6939 38040 54994 i 157.185.154.0/24 6939 38040 54994 i 163.171.164.0/24 6939 38040 54994 i 192.150.250.0/23 6939 4651 4618 4618 63529 4621 3836 i 196.50.8.0/24 6939 37613 6758 37649 i 196.50.9.0/24 6939 37613 6758 37649 i 196.50.11.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.13.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.14.0/24 6939 37613 6758 37649 i 2001:67c:15f4::/48 6939 41103 i 2401:c500:fd08::/48 6939 4637 38040 54994 i 2a01:53c0:ff03::/48 6939 4637 38040 54994 i 2a01:53c0:ff0f::/48 6939 4637 38040 54994 i 2a01:58c0::/32 6939 16347 42487 i 2a03:8920::/32 6939 41103 i 2a03:d602::/31 6939 16347 42487 i 2a03:d606::/31 6939 16347 42487 i 2a0d:ee00::/32 6939 16347 42487 i 2a0e:5b80::/29 6939 16347 42487 i 2a0e:e080::/32 6939 16347 42487 i 2a0f:c540::/29 6939 16347 42487 i
Em qui., 5 de set. de 2024 às 14:35, Jon Lewis <jlewis@lewis.org> escreveu:
This may well be intentional
In addition to their own intention, they should ask if the other side agrees with that. I don't remember which research paper compiled this. But the top 5 ASNs targeted by the no_export_to<asn> BGP communities at IXP Route-Servers say a lot about who mostly causes this type of problem. -- Douglas Fernando Fischer Engº de Controle e Automação
participants (5)
-
Douglas Fischer -
Job Snijders -
Jon Lewis -
Mike Leber -
Richard Laager