Heya NANOG, I thought this email conversation might be of interest to the group: https://mailarchive.ietf.org/arch/msg/sidrops/RdbccLbXEHUrmmdIS5K9GOdJFXA/ Kind regards, Job ----- Forwarded message from Job Snijders <job@fastly.com> ----- Date: Fri, 19 May 2023 20:54:26 +0200 From: Job Snijders <job@fastly.com> To: sidrops@ietf.org Subject: Re: [Sidrops] Estimating timeline for ASPA Deployment Dear Tony, On Fri, May 19, 2023 at 10:18:58AM -0400, Tony Tauber wrote:
I wonder what people think of likely soonest target dates for adoption of ASPA?
... Where do psychics go to dance? The crystal ball ;-) ... Disclaimer: The following is based on unsubstantiated information and/or my gutfeeling, I can't vouch for accuracy. I'd like to share some information categorized as 'here work needs to be done', a timeline, and a conclusion referencing earlier RPKI experiences. What needs to happen where? --------------------------- There are a few categories of 'moving parts' in the global Internet routing system that need upgrades in order for ASPA to see widespread adoption. Category | (Non-exhaustive) list of implementations --------------------------+------------------------------------------ Relaying Party packages: OpenBSD rpki-client, NLnet Labs Routinator, LACNIC/NicMX FORT, Cloudflare octorpki, Mikhail Puzanov's rpki-prover, ZDNS RPSTIR2. Signer implementations: NLnet Labs Krill, Dragon Labs rpki.net, and a number of Regional Internet Registries (RIRs) have their own in-house Signer solution. BGP implementations: OpenBSD OpenBGPD, NIC.CZ BIRD, FRRouting, Nokia SR-OS, Cisco IOS XR, Juniper Junos, and many others, see the 'BGP speakers' list at: http://largebgpcommunities.net/implementations/ Standalone RTR servers: StayRTR, NLnet Labs RTRTR, GoRTR, rpkirtr Integrated RTR servers: FORT, Routinator In order for there to be any ASPA deployment, at least one ASPA-capable implementation in all of the RP, Signer, and BGP categories is needed. Without a Signer there isn't anything for the RP to validate, without an RP there isn't any for the BGP speaker to verify BGP UPDATEs against. This is a great example of why the IETF process is so valuable: this process has brought together many stakeholders from different corners of the ecosystem each with different roles to jointly develop a solution for BGP route leaks. Timeline ======== 2023 - "the early wave": ASPA support arrives in select BGP, RP, RTR, and Signer implementations. At the moment of writing OpenBSD rpki-client & OpenBGPD; NLnet Labs Routinator, Krill & RTRTR, StayRTR, rpki-prover, and RIPE NCC have either already released ASPA-capable software or are in an advanced stage to do so. These 'early wave' implementations are incredible achievements as there weren't any examples for the developers to look at, they were building across unchartered territory. One Internet Exchange Point managed to deploy fully ASPA-capable Route Server stack using bleeding edge software, however this effort should be considered an outliner from a timeline perspective: https://mailman.nanog.org/pipermail/nanog/2023-February/221471.html 2024 - The IETF ratifies ASPA by publishing a series of RFC documents detailing the specification. While SIDROPS (this working group) at this moment in time is in the late stages of specifying the ASPA standard, the Internet Engineering Steering Group (IESG) and RFC Editor still need to review the draft documents; all in all this might take another 6-10 months (from now). BGP monitoring software such as BGPAlerter and BGP.Tools might take an interest and start using ASPA data to enhance their alerting capabilities: a growing amount of ASPA data (which can be used to identify route leaks) is becoming available through the 'delegated' (permissionless) RPKI distribution channels. 2025 - Having access to both published RFCs and multiple battle-tested RP implementations (which originated in the 'early wave'), more and more RIRs will feel comfortable scheduling and committing development time to productize an ASPA-offering for their respective RPKI dashboard/web portal/api services. Why 2025? RIRs oftentimes are somewhat conservative and like to see which way the wind blows before offering new services to their constitutes. This is very understandable as for an RIR to offer ASPA the work involved is significantly more than just developing the technical Signer software. RIRs need to design (web-based) user interfaces, which is a big hurdle because for the first time ever RIRs will move away from an 'ROA-centric' user interface to 'the RPKI offers multiple applications' (such as ASPA). As often is the case, going from 0 to 1 is extremely hard, from 1 to 2 still hard, and from 3 onwards it's smoother sailing. RIRs also need time to develop training courses for their internal staff, for external outreach such as capacity building. An RIR can't offer a new feature without explaining to their constitutes what new technology does, how to use it and how to troubleshoot it. It'll take time before RIRs feel comfortable answering questions about ASPA anyone might have. 2026 - General availability in commercial-off-the-shelf BGP speaker implementations (I expect most open-source implementations to be part of the 'early wave' or soon thereafter). Similar to how the RIRs need to develop training and documentation accessible for newcomers to ASPA, the COTS vendors need to develop ASPA-capable software, update procedures in Technical Assistance Centers, update and translate documentation to many languages, train pre-sales engineers. The larger the COTS vendor the more work incorporating a technology like ASPA as part of the product offering is. 2027 - Large nationwide and international ISPs deploy ASPA verification. By now many people will have flown around the world explaining the benefits of ASPA to ISPs both large and small, urging everyone to deploy ASPA verification and drop ASPA-invalid routes. Standing on the shoulders of giants: the published RFCs (by now 3 years old), the early wave RP/RTR implementations, the RIRs supporting ASPA in their hosted offerings since 2025, and now the recently gained access to commercially supported COTS BGP software in hand all contributed towards this moment. The large ISPs can deploy ASPA in their quality assurance environments, assess the revenue impact of deploying "if ASPA-Invalid then drop" EBGP routing policies, and schedule ASPA deployment through the 'once-a-year-sw-upgrade'-window applicable to most of their critical equipment. In conclusion: Of course - come 2027 - many people on social media might be inquiring their ISPs "why haven't you deployed ASPA-verification yet?!?!11" - without realizing that deploying something like ASPA at global scale depends on a complicated long supply chain (complicated both in the technical realm expressed as software, and in the realm of human-to-human conversation such as training and outreach). There are lessons I've learned from the global deployment of RPKI-ROV, a few of which I'd like to share publicly: There was an unfortunate long gap between people being able to create ROAs (without consequence, as for years virtually no organisation was performing RPKI-ROV), and large ISPs truly being able to deploy well-tested stable software to do Route Origin Validation at their EBGP edge; that by the time 2019/2020 rolled around this 'gap' had resulted in many BGP routes being considered ROV-invalid due to (by then inconsequential and forgotten) ROA misconfigurations. This latter aspect in turn made large ISPs extremely hesitant to deploy RPKI-ROV "invalid = drop" EBGP routing policies as a potential for customer outages (aka revenue impact) was perceived. It took building additional software (extensions to traffic analyzers such as pmacct.net and Kentik.com) to be able gauge exactly what the operational and revenue impact of deploying RPKI-ROV would be, before large ISP senior management buy-in happened. Additionally, in the global RPKI-ROV deployment the Internet routing community went from 'zero to one' in context of using *anything* RPKI-based. Much auxiliary infrastructure had to be build indirectly related to the production and consumption of ROAs: for example RIRs had to learn how to operate key materials stored in HSMs, issuance of Root CA certificates, other aspects of PKI. All in all I'm optimistic: with the collective 'first RPKI' experience under our belts, a positive outlook on ASPA-capable open source BGP software suitable for Internet Exchange Point deployments, and a solid diverse set of participants in 'the early wave'; universal deployment of ASPA-verification will be quite the journey - but hopefully smoother because of our prior experiences. この調子でがんばれ！ Job ----- End forwarded message -----