AT&T/as7018 now drops invalid prefixes from peers
FYI: The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers. We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time. Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet. Thanks! Jay B.
On Mon, Feb 11, 2019 at 6:55 AM Jay Borkenhagen <jayb@braeburn.org> wrote:
FYI:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time.
Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet.
Thanks!
Jay B.
Good move AT&T , thanks for taking this on
A round of applause to AT&T for leading the way! Best regards, Martijn On 2/11/19 3:53 PM, Jay Borkenhagen wrote:
FYI:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time.
Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet.
Thanks!
Jay B.
Dear Jay, AT&T, On Mon, Feb 11, 2019 at 09:53:45AM -0500, Jay Borkenhagen wrote:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
Thanks for filtering us! :-) AT&T doing origin validation combined with the peerlock-style AS_PATH filters this makes for a pretty strongly protected path between you and others.
We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time.
I think this is a sensible strategy.
Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet.
Thank you for paving the way! If you can share more about the experience in terms of load on the support tiers in your organisation, or questions from peering partners, that could perhaps be helpful information for others in their preparations. Kind regards, Job
Job Snijders writes:
Dear Jay, AT&T,
On Mon, Feb 11, 2019 at 09:53:45AM -0500, Jay Borkenhagen wrote:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
Thanks for filtering us! :-)
Any time! :-)
If you can share more about the experience in terms of load on the support tiers in your organisation, or questions from peering partners, that could perhaps be helpful information for others in their preparations.
A few reports of resulting connectivity loss have come in through various channels: on Jared's outages mailing list, on IRC, through our customer care ticket system, etc. Thusfar I have been very pleased with the reactions folks have had when we described how our policy change caused us to lose their affected route announcement. Everyone so far has understood the purpose of the RPKI, they understood why the affected route announcements were deemed invalid and thus were dropped, and best of all -- they understood what they needed to do to fix things. We got some very good advice watching this video from your most recent NLNOG day: https://www.youtube.com/watch?v=vrzl__yGqLE ... but there is one place where I disagree with Niels. He advised against lowering the local-pref of invalid routes. I agree that this should not be anyone's target policy, but it is a useful step along the way. To set invalid routes a lower local-pref, one needs to establish RTR sessions from routers to relying party servers, and to configure a policy that takes validation state into account. The policy here can also set community based on the validation state, which can help with flow-based traffic analysis. Then, when you are comfortable operating RPs that talk RTR to your routers and you're ready to implement a meaningful policy, it's a simple matter of changing from: if validation-state = invalid then local-pref = $LOWER community = foo fi to: if validation-state = invalid then drop fi In short: C'mon in! The water's fine! :-) Thanks. Jay B.
On 12 Feb 2019, at 01:52, Jay Borkenhagen <jayb@braeburn.org> wrote:
We got some very good advice watching this video from your most recent NLNOG day:
https://www.youtube.com/watch?v=vrzl__yGqLE
... but there is one place where I disagree with Niels.
You’re of course welcome to do so :-)
He advised against lowering the local-pref of invalid routes. I agree that this should not be anyone's target policy, but it is a useful step along the way. To set invalid routes a lower local-pref, one needs to establish RTR sessions from routers to relying party servers, and to configure a policy that takes validation state into account.
I agree that this is a good approach for taking first steps into the RPKI world and I would not discourage a lower local preference as a first stage. As long as we’re on the same page about invalid == reject being the intended end result.
In short: C'mon in! The water's fine! :-)
As a competitive swimmer I couldn’t agree more! -- Niels Raijer niels@fusix.nl
On Tue, 12 Feb 2019, 01:52 Jay Borkenhagen <jayb@braeburn.org wrote:
... but there is one place where I disagree with Niels. He advised against lowering the local-pref of invalid routes. I agree that this should not be anyone's target policy, but it is a useful step along the way.
For initial deployment, this can seem attractive, but remember that one of the benefits an ROA gives is specifying the maximum prefix length. This means that someone can't hijack a /23 with a /24. Lowering local pref on invalid means you're no longer protected (just generating alerts) because longer prefix length always beats local preference. Once you are confident that you're not dropping anything valuable, the local preference rule should move to dropping the route (not the traffic!) from being installed. M
Matthew Walster wrote on 12/02/2019 14:50:
For initial deployment, this can seem attractive, but remember that one of the benefits an ROA gives is specifying the maximum prefix length. This means that someone can't hijack a /23 with a /24.
they can if they forge the source ASN. RPKI helps against misconfigs rather than intentional hijackings. Nick
On Tue, Feb 12, 2019 at 03:05:28PM +0000, Nick Hilliard wrote:
Matthew Walster wrote on 12/02/2019 14:50:
For initial deployment, this can seem attractive, but remember that one of the benefits an ROA gives is specifying the maximum prefix length. This means that someone can't hijack a /23 with a /24.
they can if they forge the source ASN. RPKI helps against misconfigs rather than intentional hijackings.
Only if you specify a a minlen of /23 and a maxlen of /24 and you only announce a /23. Which you should not.
On Tue, Feb 12, 2019 at 3:06 PM Nick Hilliard <nick@foobar.org> wrote:
Matthew Walster wrote on 12/02/2019 14:50:
For initial deployment, this can seem attractive, but remember that one of the benefits an ROA gives is specifying the maximum prefix length. This means that someone can't hijack a /23 with a /24.
they can if they forge the source ASN. RPKI helps against misconfigs rather than intentional hijackings.
Some networks have AS_PATH filters in place that prevent accepting a spoofed ASN behind an EBGP session that is not authorized to announce the spoofed ASN. Secondly, there also is a group of networks that assign the same local preference for all routes received via peering - meaning that the use of a spoofed ASN will make the AS_PATH one hop longer. In other words: everyone should peer directly with the destination networks that matter to them. This is not news of course. :-) I agree some attacks in some cases may still get through, but I've come to think that ASN spoofing is far less of an issue than I originally thought it would be. Kind regards, Job
On Tue, 12 Feb 2019 at 16:05, Nick Hilliard <nick@foobar.org> wrote:
Matthew Walster wrote on 12/02/2019 14:50:
For initial deployment, this can seem attractive, but remember that one of the benefits an ROA gives is specifying the maximum prefix length. This means that someone can't hijack a /23 with a /24.
they can if they forge the source ASN. RPKI helps against misconfigs rather than intentional hijackings.
As it stands today, RPKI is only useful to prevent fat-fingering of ebgp routing policies, where routes are leaked from a point of "legitimate" re-origination such as a web filter service. It's entirely unsuitable (at present) for anything where someone is re-originating the prefix somewhere else with the same origin ASN present (i.e. maliciously) and it doesn't protect against someone accidentally sending you a prefix they heard elsewhere that is valid. My understanding is that part of that problem can be solved with https://tools.ietf.org/html/draft-ietf-sidrops-rpki-tree-validation-03 but once again it's still only preventing fat-fingering and not malicious intent. RPKI provides an additional layer of security, but it stands independent of the need for prefix list and AS_PATH filter generation (likely derived from IRR data). I'm actually of the opinion that the whole "PKI" part of RPKI is the bit that really needs to die. PKI certificates brought no real benefit to the solution. Embedding RFC 3779 (X.509 Extensions for IP Addresses and AS Identifiers) makes the data so much less accessible and difficult to process for all but the most technically competent system/network administrators, especially considering most existing tools and libraries for X.509 certificate operations just don't support doing anything reasonable with them in a way that doesn't involve several hours afterwards in a candle-lit bath with a Pink Floyd CD playing in the background. Personally, I'd like to see BMP (RFC 7854, unidirectional flow of information) bolted on to an extended version of the RTR protocol (RFC8210) or a stripped down unidirectional BGP session, and have the whole policy engine offloaded from the border router to some kind of daemon running, potentially on a distributed controller of sorts, that: - Pulled in all this raw data and processed it, using the compute power available in servers after the invention of the Cyrix 200. - Asserted the veracity of the data it has received (real-time updates of prefix-lists, paths, trustworthiness, advertised "directionality" akin to peerlock etc) without having to periodically push new configuration to your routers. - Applied policy decisions, including mapping traffic for that destination into a FEC / MPLS tunnel as appropriate. - Possibly even applying aggregation at the FIB install level if appropriate to reduce total FIB entry size and programming time. - Distributed a final RIB (+FIB if needed) out to each of the BGP routers in the AS, depending on the view they needed. I think I'd have a hard time convincing people of that though, and the RIR policy folk would have a small heart attack at the level of complexity required. M
Le 2019-02-12 20:56, Nick Hilliard a écrit :
Matthew Walster wrote on 12/02/2019 19:27:
I'm actually of the opinion that the whole "PKI" part of RPKI is the bit that really needs to die.
I'll claim a de-facto godwin if anyone mentions the word "blockchain".
Nick
On Tue, Feb 12, 2019 at 7:30 PM Matthew Walster <matthew@walster.org> wrote:
On Tue, 12 Feb 2019 at 16:05, Nick Hilliard <nick@foobar.org> wrote:
Matthew Walster wrote on 12/02/2019 14:50:
For initial deployment, this can seem attractive, but remember that one of the benefits an ROA gives is specifying the maximum prefix length. This means that someone can't hijack a /23 with a /24.
they can if they forge the source ASN. RPKI helps against misconfigs rather than intentional hijackings.
As it stands today, RPKI is only useful to prevent fat-fingering of ebgp routing policies, where routes are leaked from a point of "legitimate" re-origination such as a web filter service.
This simply isn't true. I'm willing to argue a bit about to what degree and in what circumstances RPKI Origin Validation technology is useful or not, but the use of the word "only" in this context is incorrect.
It's entirely unsuitable (at present) for anything where someone is re-originating the prefix somewhere else with the same origin ASN present (i.e. maliciously) and it doesn't protect against someone accidentally sending you a prefix they heard elsewhere that is valid.
This also is not true. It is not as black and white as you make it out to be. 1/ For instance AT&T does not accept BGP UPDATES with 2914 anywhere in the AS_PATH except on the direct EBGP sessions between 7018 and 2914. This means that you can craft BGP UPDATES with 2914 all you want, but 7018 won't accept them. You can't inject yourself between AT&T and NTT using spoofing. 2/ Many networks give all their peering partners the same LOCAL_PREFERENCE, so you'll have to not only spoof the BGP Origin ASN but also compete & win for the shortest path in order for your hijack to arrive at the intended location. We as industry essentially already have path validation for paths of length 1. This may not seem much, but since people in this industry tend to peer directly with networks that matter to them. The majority of Internet traffic flows over paths that have an AS_PATH length of 1.
My understanding is that part of that problem can be solved with https://tools.ietf.org/html/draft-ietf-sidrops-rpki-tree-validation-03 but once again it's still only preventing fat-fingering and not malicious intent.
Uh... that draft is about something else entirely. :-)
I think I'd have a hard time convincing people of that though, and the RIR policy folk would have a small heart attack at the level of complexity required.
you are probably right about that, I'd prefer to stick to a technology that actually exists and is deployable! :-) Kind regards, Job
On Wed, 13 Feb 2019 at 00:24, Job Snijders <job@instituut.net> wrote:
On Tue, Feb 12, 2019 at 7:30 PM Matthew Walster <matthew@walster.org> wrote:
As it stands today, RPKI is only useful to prevent fat-fingering of ebgp routing policies, where routes are leaked from a point of "legitimate" re-origination such as a web filter service.
This simply isn't true. I'm willing to argue a bit about to what degree and in what circumstances RPKI Origin Validation technology is useful or not, but the use of the word "only" in this context is incorrect.
I have massively oversimplified the situation, true, but essentially the vast majority of the pie chart of "why a prefix would be originated from another ASN and/or of a different prefix length" is: - Leaking from within the originator's borders where the border acts as a point of aggregation (atomic or otherwise). Mostly harmless. - Someone else leaking out the prefix when they generate it to assist with traffic engineering, including through a proxy/filter (Pakistan/YouTube) - Someone maliciously acting, so as to steal that traffic (predominantly Bitcoin mining pools, but could be ACME TLS etc) Have I missed anything (that has substantial real-life relevance to any incident seen so far) or would you say that was a far summary?
It's entirely unsuitable (at present) for anything where someone is re-originating the prefix somewhere else with the same origin ASN present (i.e. maliciously) and it doesn't protect against someone accidentally sending you a prefix they heard elsewhere that is valid.
This also is not true. It is not as black and white as you make it out to be.
1/ For instance AT&T does not accept BGP UPDATES with 2914 anywhere in the AS_PATH except on the direct EBGP sessions between 7018 and 2914. This means that you can craft BGP UPDATES with 2914 all you want, but 7018 won't accept them. You can't inject yourself between AT&T and NTT using spoofing.
Not relevant to RPKI as it currently stands, which is unrelated to the current mechanism being used: origin validation. Path filtering and IRR prefix list filtering are much more important tools that should be used first -- RPKI can only supplement those tools and should not be used in isolation, unlike the others which do a "good enough" job for most non-transit networks.
2/ Many networks give all their peering partners the same LOCAL_PREFERENCE, so you'll have to not only spoof the BGP Origin ASN but also compete & win for the shortest path in order for your hijack to arrive at the intended location.
We as industry essentially already have path validation for paths of length 1. This may not seem much, but since people in this industry tend to peer directly with networks that matter to them. The majority of Internet traffic flows over paths that have an AS_PATH length of 1.
We do? I beg to differ. Much of Europe does, national networks in North America generally do, but those in the LACNIC / AfriNIC / APNIC regions? Of course there are exceptions (HKIX et al) but your average path length is going to be generally longer in those regions, and all it takes is for someone at a local IX to forge a couple of path attributes, and suddenly a bunch of traffic shifts.
My understanding is that part of that problem can be solved with https://tools.ietf.org/html/draft-ietf-sidrops-rpki-tree-validation-03 but once again it's still only preventing fat-fingering and not malicious intent.
Uh... that draft is about something else entirely. :-)
Oh whooops! Copied the URL from the wrong tab! I of course meant the draft you and Massimiliano Stucchi have at https://tools.ietf.org/html/draft-ss-grow-rpki-as-cones-00 -- my bad!!
I think I'd have a hard time convincing people of that though, and the RIR policy folk would have a small heart attack at the level of complexity required.
you are probably right about that, I'd prefer to stick to a technology that actually exists and is deployable! :-)
My idea is deployable today (for some values of today). Mark all the received NLRIs as not to be installed in the FIB, replicate the data via BMP or BGP with add-paths to a collector, process it, and ship out the results via a BGP session that sets the correct next-hops. I wrote a simple BGP speaker that had 4 byte ASN capability advertisement and graceful restart, literally in a day. Sure, it only supported IPv4 unicast, but with the wide array of open source BGP engines out there, it wouldn't be so hard to put something together that offloaded the entire policy phase from the router to a control plane. You could easily add modules in (just a chain of OOP interfaces) so that you could compare across prefixes, apply aggregation and damping, near real-time mirroring of IRR data to use as filter sets, maybe even read interface stats via SNMP / streaming telemetry / sFlow to start moving traffic around when ports get hot. Oh dear, I've re-invented SDN haven't I? :S However, just because you *could* deploy it today, doesn't mean you should =) M
1/ For instance AT&T does not accept BGP UPDATES with 2914 anywhere in the AS_PATH except on the direct EBGP sessions between 7018 and 2914. This means that you can craft BGP UPDATES with 2914 all you want, but 7018 won't accept them. You can't inject yourself between AT&T and NTT using spoofing.
Sure, but RPKI plays no role in this.
2/ Many networks give all their peering partners the same LOCAL_PREFERENCE, so you'll have to not only spoof the BGP Origin ASN but also compete & win for the shortest path in order for your hijack to arrive at the intended location.
Also utterly and completely unrelated to ROAs.
We as industry essentially already have path validation for paths of length 1. This may not seem much, but since people in this industry tend to peer directly with networks that matter to them. The majority of Internet traffic flows over paths that have an AS_PATH length of 1.
I would buy this argument with length 1-3, but I’m not completely convinced of “1”. Owen
Jay & everyone AT&T: I just want to say thank you. Kudos to your team for implementing and management for having the intestinal fortitude to do so. -- TTFN, patrick
On Feb 11, 2019, at 09:53, Jay Borkenhagen <jayb@braeburn.org> wrote:
FYI:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time.
Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet.
Thanks!
Jay B.
That's great! Do you guys have plans to publish ROAs for your own netblocks? If so, can you please share info on your process (tools, pitfalls, etc.)? Thanks! On 2/11/19, 7:55 AM, "NANOG on behalf of Jay Borkenhagen" <nanog-bounces@nanog.org on behalf of jayb@braeburn.org> wrote: FYI: The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers. We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time. Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet. Thanks! Jay B. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Compton, Rich A writes:
That's great! Do you guys have plans to publish ROAs for your own netblocks? If so, can you please share info on your process (tools, pitfalls, etc.)? Thanks!
Hi Rich, We do have ROAs published for a not insignificant fraction of our address space. For example (and cherry-picking the representation most favorable to us) we're listed at #6 in the "25 Autonomous Systems with the most Address Space VALID by RPKI" at this NIST RPKI tracker: https://rpki-monitor.antd.nist.gov/#rpki_adopters We will publish more ROAs over time. Thusfar we have been utilizing ARIN's hosted model, but down the road ARIN's delegated model will be in our future. https://www.arin.net/resources/rpki/using_rpki.html Thanks. Jay B.
Congrats Jay, this is awesome news!
On 12 Feb 2019, at 01:01, Jay Borkenhagen <jayb@braeburn.org> wrote:
Compton, Rich A writes:
That's great! Do you guys have plans to publish ROAs for your own netblocks? If so, can you please share info on your process (tools, pitfalls, etc.)? Thanks!
Hi Rich,
We do have ROAs published for a not insignificant fraction of our address space. For example (and cherry-picking the representation most favorable to us) we're listed at #6 in the "25 Autonomous Systems with the most Address Space VALID by RPKI" at this NIST RPKI tracker:
I’m interested to hear what is preventing you from creating ROAs for all of your announcements.
We will publish more ROAs over time. Thusfar we have been utilizing ARIN's hosted model, but down the road ARIN's delegated model will be in our future.
What are your main drivers for wanting to move to the delegated model? Cheers! -Alex
Congrats Jay, this is awesome news!
Thanks, Alex!
I’m interested to hear what is preventing you from creating ROAs for all of your announcements.
We will publish more ROAs over time. Thusfar we have been utilizing ARIN's hosted model, but down the road ARIN's delegated model will be in our future.
What are your main drivers for wanting to move to the delegated model?
We can publish ROAs immediately for aggregate address blocks that we have been allocated if all routes are originated only by our network. But for our address allocations within which we have further assigned sub-blocks to our customers as PA space where we allow multihoming (e.g. within 12.0.0.0/8), we need to offer our downstream customers the ability to publish ROAs for their specific portions first before we publish a ROA for the aggregate, or else we'll make their announcements become invalid. Setting up that ability for our customers to publish ROAs for the PA space they receive from us will require tight integration with our customer software support systems, and perhaps also with our own certificate authority -- thus the delegated model. BTW: Alex, do you know where one might be able to get RPKI CA software? :-) Thanks. Jay B.
This is the best news today! Great job!! Cheers, Melchior On Mon, Feb 11, 2019 at 3:56 PM Jay Borkenhagen <jayb@braeburn.org> wrote:
FYI:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time.
Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet.
Thanks!
Jay B.
valdis.kletnieks@vt.edu writes:
On Mon, 11 Feb 2019 09:53:45 -0500, Jay Borkenhagen said:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
Congrats!
Thanks!
Are you able to comment on what amount of routes are getting dropped?
In round numbers, we dropped about 5000 invalid prefixes total between ipv4 and ipv6. Roughly half of those prefixes were covered by less-specific non-invalid routes, so connectivity should not have been affected for those prefixes (assuming an announcement yields reachability to all destinations within it). Flow analysis was showing just a couple Gbps of traffic to all invalid routes all across the country, and much less than that with those invalids having no covering less-specifics. Jay B.
On 11/Feb/19 16:53, Jay Borkenhagen wrote:
FYI:
The AT&T/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers.
We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time.
Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet.
Well done! Mark.
participants (17)
-
Alex Band -
Ca By -
Compton, Rich A -
Denis Fondras -
i3D.net - Martijn Schmidt -
Jay Borkenhagen -
Job Snijders -
Job Snijders -
Mark Tinka -
Matthew Walster -
Melchior Aelmans -
Michael Hallgren -
Nick Hilliard -
Niels Raijer -
Owen DeLong -
Patrick W. Gilmore -
valdis.kletnieks＠vt.edu