Le 15/12/2025 à 06:48, Hank Nussbacher via NANOG a écrit :
On 15/12/2025 6:55, John Curran via NANOG wrote:
A masterclass in owning a mistake and handling it properly.
Hi Hank, Fully agreed! brother, it's where the difference stands: PmR -- Post-mortem Report! Incidents are common things; PmR is BCOP (Best Operational Practice)...it's still rare :'-( ...and this one gives us an opportunity to ameliorate the BCOP template [*]; imho. __ [*]: dns-post-mortem <https://archive.psg.com/240614.dns-post-mortem> ...i humbly suggest to John and team, to accept to update the structure of their PmR, so that, in its next version, it takes into account all the feedback/suggestions received on topic. What could be mine? That file should be version-controlled & must follow a staging step prior to be committed... ...imho. It's practically what they decided until they are able to fully integrate it into normal pipelines. ...what to do during the staging: cross-check using some API endpoints such as: * Related prefixes <https://stat.ripe.net/docs/data-api/api-endpoints/related-prefixes/> * Prefix overview <https://stat.ripe.net/docs/data-api/api-endpoints/prefix-overview/> * RIS first last seen <https://stat.ripe.net/docs/data-api/api-endpoints/ris-first-last-seen/> * Routing status <https://stat.ripe.net/docs/data-api/api-endpoints/routing-status/> * RPKI history <https://stat.ripe.net/docs/data-api/api-endpoints/rpki-history/> * Routing history <https://stat.ripe.net/docs/data-api/api-endpoints/routing-history/> * Prefix routing consistency <https://stat.ripe.net/docs/data-api/api-endpoints/prefix-routing-consistency/> * RPKI validation <https://stat.ripe.net/docs/data-api/api-endpoints/rpki-validation/> * Reverse DNS consistency <https://stat.ripe.net/docs/data-api/api-endpoints/reverse-dns-consistency/> * <https://stat.ripe.net/docs/data-api/api-endpoints/related-prefixes/>Looking glass <https://stat.ripe.net/docs/data-api/api-endpoints/looking-glass/> * AS routing consistency <https://stat.ripe.net/docs/data-api/api-endpoints/as-routing-consistency/> * * Easy BGP Monitoring with BGPalerter | RIPE Labs <https://labs.ripe.net/author/massimo_candela/easy-bgp-monitoring-with-bgpalerter/> * An Introduction to IRR Explorer | RIPE Labs <https://labs.ripe.net/author/teun/an-introduction-to-irr-explorer/> * BGPAlerter | Loud Networking <https://loudnetworking.org/2021/07/28/bgpalerter/> * . Shalom, --sb.
Regards, Hank
On Dec 14, 2025, at 6:54 PM, Jon Lewis via NANOG <nanog@lists.nanog.org> wrote:
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/ This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function). Jon –
I agree completely. This was a failure of ARIN in the performance of its core mission, and one that resulted in customer impact. The community is entitled to full transparency in understanding how this occurred and the steps we have taken to prevent any similar incident in the future.
Analysts have performed this particular allocation process thousands of times previously without issue, but in this case a resource analyst made an error that resulted in the reallocation of a previously assigned NRPM 4.10 address block. Clearly, while that was the trigger for this incident, the real fault here is that ARIN should not have any processes that are predicated on perfect human performance. Prior to this incident, it was my belief that this was already the case and that assigned resource blocks could not be impacted by analyst error.
That is not the case for the manual processes used in the management of NRPM 4.10 address blocks. As a result, we have corrected the process to require a second set of eyes before any change is committed. Longer term, I prefer to fully automate this process, but until that can be implemented we will continue with the manual process, as amended with a mandatory supervisor confirmation step, as a reasonable and appropriate mitigation.
I have experience running several major ISPs and am fully aware that operators rely on ARIN for flawless performance. Even a single customer impact is not acceptable, which is why we issued the report to the community detailing the incident and its resolution. To the extent that there is any need for additional clarity, please don’t hesitate to ask – either here on the list (or to me directly as you prefer.)
Thanks, /John
John Curran President and CEO American Registry for Internet Numbers
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/2F2TMDVE...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/U33VXD6K...
-- Best Regards ! baya.sylvain [AT cmNOG DOT cm] | cmNOG's Structure <https://www.cmnog.cm/dokuwiki/Structure> | CAMIX's Website <https://web.archive.org/web/20240706030230/http://www.camix.cm/> | Douala-IX's Looking Glass <https://tools.std.douala-ix.net/lg> | cmNOG's Surveys <https://survey2.cmnog.cm/> | Subscribe to cmNOG's Mailing List <https://lists.cmnog.cm/mailman/listinfo/cmnog> | __ #LASAINTEBIBLE|#Ephésiens3:17,14-19«17 en sorte que CHRIST habite dans vos cœurs par la foi; afin qu'étant enracinés et fondés dans L'Amour,»#AMEN,#Maranatha,#MerciJÉSUS! #MaPrière est que tu naisses de nouveau.#Chrétiennement