On 15/12/2025 6:55, John Curran via NANOG wrote: A masterclass in owning a mistake and handling it properly. Regards, Hank
On Dec 14, 2025, at 6:54 PM, Jon Lewis via NANOG <nanog@lists.nanog.org> wrote:
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/ This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function). Jon –
I agree completely. This was a failure of ARIN in the performance of its core mission, and one that resulted in customer impact. The community is entitled to full transparency in understanding how this occurred and the steps we have taken to prevent any similar incident in the future.
Analysts have performed this particular allocation process thousands of times previously without issue, but in this case a resource analyst made an error that resulted in the reallocation of a previously assigned NRPM 4.10 address block. Clearly, while that was the trigger for this incident, the real fault here is that ARIN should not have any processes that are predicated on perfect human performance. Prior to this incident, it was my belief that this was already the case and that assigned resource blocks could not be impacted by analyst error.
That is not the case for the manual processes used in the management of NRPM 4.10 address blocks. As a result, we have corrected the process to require a second set of eyes before any change is committed. Longer term, I prefer to fully automate this process, but until that can be implemented we will continue with the manual process, as amended with a mandatory supervisor confirmation step, as a reasonable and appropriate mitigation.
I have experience running several major ISPs and am fully aware that operators rely on ARIN for flawless performance. Even a single customer impact is not acceptable, which is why we issued the report to the community detailing the incident and its resolution. To the extent that there is any need for additional clarity, please don’t hesitate to ask – either here on the list (or to me directly as you prefer.)
Thanks, /John
John Curran President and CEO American Registry for Internet Numbers
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/2F2TMDVE...