Accidental ARIN Reallocation
Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately. They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly. I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier. Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes. Best, Chase Lauer GalaxyGate, AS397031 https://galaxygate.net
"accidentally reallocated" is a new one to me... You would think that there would be checks and balances in place to prevent this from happening, such as looking for any valid ROAs or route objects in ARIN's IRR database, looking for any existing Whois objects or BGP routes, etc. to make sure the route is not being used or delegated prior to redelegation. Regards, Christopher Hawker ________________________________ From: Chase via NANOG <nanog@lists.nanog.org> Sent: Wednesday, 10 December 2025 5:19 AM To: nanog <nanog@lists.nanog.org> Cc: Chase <chase@galaxygate.net> Subject: Accidental ARIN Reallocation Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately. They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly. I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier. Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes. Best, Chase Lauer GalaxyGate, AS397031 https://galaxygate.net _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5MCMSACQ...
Operations teams everywhere, start writing that playbook for this. On Tue, Dec 9, 2025 at 3:10 PM Christopher Hawker via NANOG <nanog@lists.nanog.org> wrote:
"accidentally reallocated" is a new one to me... You would think that there would be checks and balances in place to prevent this from happening, such as looking for any valid ROAs or route objects in ARIN's IRR database, looking for any existing Whois objects or BGP routes, etc. to make sure the route is not being used or delegated prior to redelegation.
Regards, Christopher Hawker ________________________________ From: Chase via NANOG <nanog@lists.nanog.org> Sent: Wednesday, 10 December 2025 5:19 AM To: nanog <nanog@lists.nanog.org> Cc: Chase <chase@galaxygate.net> Subject: Accidental ARIN Reallocation
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
Best,
Chase Lauer
GalaxyGate, AS397031
https://galaxygate.net _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5MCMSACQ... _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/XDJIYKX5...
-- - Andrew "lathama" Latham -
Chase (and the NANOG operator community) – Apologies for not replying earlier, but I wanted to make sure we understood exactly what went wrong and had that written up as an incident report (i.e., rather than responding piecemeal and without full clarity). Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/ This came down to some remaining manual handling around NRPM 4.10 space, which uses a sparse allocation model and wasn’t yet fully integrated into our automated inventory. That gap allowed your already in-use IPv4 /24 block to be mistaken as available and reissued to another customer. When it was removed and reissued, your associated ROA was removed as well, along with reverse DNS services, etc. We had plans to automate our NRPM 4.10 inventory management (largely for efficiency reasons), but this incident and subsequent review showed that the remaining manual steps pose more risk than is reasonable. As a result, we’ve moved that work well up the priority list for development. In the meantime, and as detailed in the incident report, we’ve put additional controls in place – including a mandatory second review on any resource deletion from an organization – to prevent this from happening again. I will also be reviewing our number resource inventory management practices internally (and with the ARIN Board of Trustees) to ensure there are not any other similar situations that might pose such a risk. My deepest apologies for this incident; we are acutely aware that the integrity of Internet number resources is essential to network operators, and thus it must be inherent to ARIN’s performance at all times. Sincerely, /John John Curran President and CEO American Registry for Internet Numbers On Dec 9, 2025, at 1:19 PM, Chase via NANOG <nanog@lists.nanog.org> wrote: Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately. They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly. I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier. Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes. Best, Chase Lauer GalaxyGate, AS397031 https://galaxygate.net _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5MCMSACQ...
An unfortunate occurrence. Respect to ARIN for owning it completely, and being transparent enough to share it with us. Aaron
On Dec 12, 2025, at 11:28 AM, John Curran via NANOG <nanog@lists.nanog.org> wrote:
Chase (and the NANOG operator community) –
Apologies for not replying earlier, but I wanted to make sure we understood exactly what went wrong and had that written up as an incident report (i.e., rather than responding piecemeal and without full clarity).
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This came down to some remaining manual handling around NRPM 4.10 space, which uses a sparse allocation model and wasn’t yet fully integrated into our automated inventory. That gap allowed your already in-use IPv4 /24 block to be mistaken as available and reissued to another customer. When it was removed and reissued, your associated ROA was removed as well, along with reverse DNS services, etc.
We had plans to automate our NRPM 4.10 inventory management (largely for efficiency reasons), but this incident and subsequent review showed that the remaining manual steps pose more risk than is reasonable. As a result, we’ve moved that work well up the priority list for development. In the meantime, and as detailed in the incident report, we’ve put additional controls in place – including a mandatory second review on any resource deletion from an organization – to prevent this from happening again.
I will also be reviewing our number resource inventory management practices internally (and with the ARIN Board of Trustees) to ensure there are not any other similar situations that might pose such a risk. My deepest apologies for this incident; we are acutely aware that the integrity of Internet number resources is essential to network operators, and thus it must be inherent to ARIN’s performance at all times.
Sincerely, /John
John Curran President and CEO American Registry for Internet Numbers
On Dec 9, 2025, at 1:19 PM, Chase via NANOG <nanog@lists.nanog.org> wrote:
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
Best,
Chase Lauer
GalaxyGate, AS397031
https://galaxygate.net _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5MCMSACQ...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/FY3SDD72...
I would agree with Aaron1 that it's gratifying to see a public and relatively quick and thorough report. However, the write-up <https://www.arin.net/announcements/20251212/> raises questions for me: - What is "NPRM 4.10 space"? - I looked it up and think I somewhat understand that it's NRPM 4.10 <https://www.arin.net/participate/policy/nrpm/#4-10-dedicated-ipv4-allocation-to-facilitate-ipv6-deployment> :Dedicated IPv4 Allocation to Facilitate IPv6 Deployment ARIN shall allocate a contiguous /10 from its last /8 IPv4 allocation from IANA. This IPv4 allocation will be set aside and dedicated to facilitate IPv6 deployment. Allocations and assignments from this block must be justified by immediate IPv6 deployment requirements.... ---- - What exact space falls in this category? - How is it used? - How likely or not are the problems/fixes applicable to other things that ARIN is doing on behalf of its members? All in all, the write-up smacks a bit of "inside baseball <https://en.wikipedia.org/wiki/Inside_baseball_(metaphor)>" (from Wikipedia, linked: In North American slang, the term *inside baseball* refers to the minutiae and detailed inner workings of a system that are only interesting to, or appreciated by, experts, insiders, and aficionados.) Again, I appreciate the willingness and effort by the ARIN Staff and Leadership, just wanted to give a little more feedback. Thanks, Tony On Fri, Dec 12, 2025 at 2:21 PM Aaron1 via NANOG <nanog@lists.nanog.org> wrote:
An unfortunate occurrence. Respect to ARIN for owning it completely, and being transparent enough to share it with us.
Aaron
On Dec 12, 2025, at 11:28 AM, John Curran via NANOG < nanog@lists.nanog.org> wrote:
Chase (and the NANOG operator community) –
Apologies for not replying earlier, but I wanted to make sure we understood exactly what went wrong and had that written up as an incident report (i.e., rather than responding piecemeal and without full clarity).
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This came down to some remaining manual handling around NRPM 4.10 space, which uses a sparse allocation model and wasn’t yet fully integrated into our automated inventory. That gap allowed your already in-use IPv4 /24 block to be mistaken as available and reissued to another customer. When it was removed and reissued, your associated ROA was removed as well, along with reverse DNS services, etc.
We had plans to automate our NRPM 4.10 inventory management (largely for efficiency reasons), but this incident and subsequent review showed that the remaining manual steps pose more risk than is reasonable. As a result, we’ve moved that work well up the priority list for development. In the meantime, and as detailed in the incident report, we’ve put additional controls in place – including a mandatory second review on any resource deletion from an organization – to prevent this from happening again.
I will also be reviewing our number resource inventory management practices internally (and with the ARIN Board of Trustees) to ensure there are not any other similar situations that might pose such a risk. My deepest apologies for this incident; we are acutely aware that the integrity of Internet number resources is essential to network operators, and thus it must be inherent to ARIN’s performance at all times.
Sincerely, /John
John Curran President and CEO American Registry for Internet Numbers
On Dec 9, 2025, at 1:19 PM, Chase via NANOG <nanog@lists.nanog.org> wrote:
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets ( 23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
Best,
Chase Lauer
GalaxyGate, AS397031
https://galaxygate.net _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5MCMSACQ...
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/FY3SDD72...
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/JJ5IYY6I...
Huge respect for admitting that ARIN uses Excel as an IPAM solution. You are braver than I... I'll see myself out. On Fri, Dec 12, 2025 at 12:29 PM John Curran via NANOG < nanog@lists.nanog.org> wrote:
Chase (and the NANOG operator community) –
Apologies for not replying earlier, but I wanted to make sure we understood exactly what went wrong and had that written up as an incident report (i.e., rather than responding piecemeal and without full clarity).
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This came down to some remaining manual handling around NRPM 4.10 space, which uses a sparse allocation model and wasn’t yet fully integrated into our automated inventory. That gap allowed your already in-use IPv4 /24 block to be mistaken as available and reissued to another customer. When it was removed and reissued, your associated ROA was removed as well, along with reverse DNS services, etc.
We had plans to automate our NRPM 4.10 inventory management (largely for efficiency reasons), but this incident and subsequent review showed that the remaining manual steps pose more risk than is reasonable. As a result, we’ve moved that work well up the priority list for development. In the meantime, and as detailed in the incident report, we’ve put additional controls in place – including a mandatory second review on any resource deletion from an organization – to prevent this from happening again.
I will also be reviewing our number resource inventory management practices internally (and with the ARIN Board of Trustees) to ensure there are not any other similar situations that might pose such a risk. My deepest apologies for this incident; we are acutely aware that the integrity of Internet number resources is essential to network operators, and thus it must be inherent to ARIN’s performance at all times.
Sincerely, /John
John Curran President and CEO American Registry for Internet Numbers
On Dec 9, 2025, at 1:19 PM, Chase via NANOG <nanog@lists.nanog.org> wrote:
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets ( 23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
Best,
Chase Lauer
GalaxyGate, AS397031
https://galaxygate.net _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5MCMSACQ...
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/FY3SDD72...
On Fri, Dec 12, 2025 at 11:49 AM Tony Tauber via NANOG <nanog@lists.nanog.org> wrote:
I would agree with Aaron1 that it's gratifying to see a public and relatively quick and thorough report. However, the write-up <https://www.arin.net/announcements/20251212/> raises questions for me:
- What is "NPRM 4.10 space"? - I looked it up and think I somewhat understand that it's NRPM 4.10 <https://www.arin.net/participate/policy/nrpm/#4-10-dedicated-ipv4-allocation-to-facilitate-ipv6-deployment> :Dedicated IPv4 Allocation to Facilitate IPv6 Deployment
What exact space falls in this category?
How is it used?
Hi Tony, 464xlat and similar technologies. Basically, ARIN has a set-aside for folks who have IPv6 devices on an IPv6 network that have a need to also talk to IPv4 devices on the Internet via one of the transition technologies. For that specific use case, there's still a pool of unallocated addresses available. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Responses in line below. From: Tony Tauber via NANOG <nanog@lists.nanog.org> Date: Friday, December 12, 2025 at 2:50 PM To: North American Network Operators Group <nanog@lists.nanog.org> Cc: Chase <chase@galaxygate.net>, John Curran <jcurran@arin.net>, Tony Tauber <ttauber@1-4-5.net> Subject: Re: Accidental ARIN Reallocation I would agree with Aaron1 that it's gratifying to see a public and relatively quick and thorough report. However, the write-up <https://www.arin.net/announcements/20251212/> raises questions for me: - What is "NPRM 4.10 space"? - I looked it up and think I somewhat understand that it's NRPM 4.10 <https://www.arin.net/participate/policy/nrpm/#4-10-dedicated-ipv4-allocation-to-facilitate-ipv6-deployment> :Dedicated IPv4 Allocation to Facilitate IPv6 Deployment ARIN shall allocate a contiguous /10 from its last /8 IPv4 allocation from IANA. This IPv4 allocation will be set aside and dedicated to facilitate IPv6 deployment. Allocations and assignments from this block must be justified by immediate IPv6 deployment requirements.... ---- - [ARIN] This is a reserved IPv4 block set aside to facilitate IPv6 deployment under NRPM 4.10. Allocations and assignments from this space must be justified by an immediate IPv6 deployment need. Only one /24 can be requested per organization every 6 months if all policy conditions are met. What exact space falls in this category? -[ARIN] The community set aside 23.128.0.0/10 as the NRPM 4.10 IPv4 reserve for IPv6 transition. This /10 represents 0.3% of our total IPv4 inventory. Total starting pool: 23.128.0.0/10 (16,384 /24s) Remaining in the pool: 13,167 /24s Average Issuance: ~55 /24s a month Customers with 4.10 space: 2,386 unique Org IDs, 2,116 (89%) of which have only a single 4.10 /24 How is it used? - [ARIN] NRPM 4.10 space is issued to support organizations' IPv6 transition and translation services. Of the 16,384 /24s in the reserve, 3,217 (about 20%) have been issued to 2,386 customers. These 4.10 allocations represent roughly 0.05% of ARIN's total allocated IPv4 space. How likely or not are the problems/fixes applicable to other things that ARIN is doing on behalf of its members? - [ARIN] The issues and fixes we identified are mostly specific to the way NRPM 4.10 space is issued. However, the same delete function is used in a small number of other workflows, such as Inter-RIR transfers, the IPv4 Waiting List, cleanup of outdated SWIPs, and certain Ask ARIN requests to merge or split networks. This delete function is used infrequently and is only authorized for designated senior analysts and management. ARIN has strengthened the related review controls as part of our corrective actions.- All in all, the write-up smacks a bit of "inside baseball <https://en.wikipedia.org/wiki/Inside_baseball_(metaphor)>" (from Wikipedia, linked: In North American slang, the term *inside baseball* refers to the minutiae and detailed inner workings of a system that are only interesting to, or appreciated by, experts, insiders, and aficionados.) Again, I appreciate the willingness and effort by the ARIN Staff and Leadership, just wanted to give a little more feedback. Thanks, Tony On Fri, Dec 12, 2025 at 2:21 PM Aaron1 via NANOG <nanog@lists.nanog.org> wrote:
An unfortunate occurrence. Respect to ARIN for owning it completely, and being transparent enough to share it with us.
Aaron
On Dec 12, 2025, at 11:28 AM, John Curran via NANOG < nanog@lists.nanog.org> wrote:
Chase (and the NANOG operator community) –
Apologies for not replying earlier, but I wanted to make sure we understood exactly what went wrong and had that written up as an incident report (i.e., rather than responding piecemeal and without full clarity).
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This came down to some remaining manual handling around NRPM 4.10 space, which uses a sparse allocation model and wasn’t yet fully integrated into our automated inventory. That gap allowed your already in-use IPv4 /24 block to be mistaken as available and reissued to another customer. When it was removed and reissued, your associated ROA was removed as well, along with reverse DNS services, etc.
We had plans to automate our NRPM 4.10 inventory management (largely for efficiency reasons), but this incident and subsequent review showed that the remaining manual steps pose more risk than is reasonable. As a result, we’ve moved that work well up the priority list for development. In the meantime, and as detailed in the incident report, we’ve put additional controls in place – including a mandatory second review on any resource deletion from an organization – to prevent this from happening again.
I will also be reviewing our number resource inventory management practices internally (and with the ARIN Board of Trustees) to ensure there are not any other similar situations that might pose such a risk. My deepest apologies for this incident; we are acutely aware that the integrity of Internet number resources is essential to network operators, and thus it must be inherent to ARIN’s performance at all times.
Sincerely, /John
John Curran President and CEO American Registry for Internet Numbers
On Dec 9, 2025, at 1:19 PM, Chase via NANOG <nanog@lists.nanog.org> wrote:
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets ( 23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
Best,
Chase Lauer
GalaxyGate, AS397031
https://galaxygate.net _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5MCMSACQ...
On 12/12/25 15:53, William Herrin via NANOG wrote:
464xlat and similar technologies. Basically, ARIN has a set-aside for folks who have IPv6 devices on an IPv6 network that have a need to also talk to IPv4 devices on the Internet via one of the transition technologies. For that specific use case, there's still a pool of unallocated addresses available.
In addition to address translation targets, they can (per policy) be used for other critical functions that require dual-stacking such as DNS servers (both recursive and authoritative noting that there are still a lot of "target" networks that are IPv6-enabled but don't have dual-stack authoritative nameservers) and things like mail servers and potentially even web servers that require "some rando on the Internet" who may not have IPv6 at all to be able to hit them. The (somewhat unwritten) intent appears to be to allow a newly started entity to still obtain a minimal IPv4 presence so as to facilitate an "IPv6 first" network deployment without having to wait on the waiting list or go to the open market via specified transfer. You can get a 4.10 /24 allocation essentially immediately after having an IPv6 allocation provided you use it for the stated purposes. A /24 is cramped but largely adequate for this purpose for most newly-started network. If you can justify need for more space such as because your NAT overload ratio is becoming untenable, you can request expansion, and ARIN has policies in place to try to make it so that your expansion space will aggregate with your original space without requiring you to re-number.
I also hope the other RIRs are reviewing their internal processes to see if they have similar exposures that need to be addressed (no pun intended). Thank you jms On Fri, Dec 12, 2025 at 9:49 PM Brandon Martin via NANOG < nanog@lists.nanog.org> wrote:
On 12/12/25 15:53, William Herrin via NANOG wrote:
464xlat and similar technologies. Basically, ARIN has a set-aside for folks who have IPv6 devices on an IPv6 network that have a need to also talk to IPv4 devices on the Internet via one of the transition technologies. For that specific use case, there's still a pool of unallocated addresses available.
In addition to address translation targets, they can (per policy) be used for other critical functions that require dual-stacking such as DNS servers (both recursive and authoritative noting that there are still a lot of "target" networks that are IPv6-enabled but don't have dual-stack authoritative nameservers) and things like mail servers and potentially even web servers that require "some rando on the Internet" who may not have IPv6 at all to be able to hit them.
The (somewhat unwritten) intent appears to be to allow a newly started entity to still obtain a minimal IPv4 presence so as to facilitate an "IPv6 first" network deployment without having to wait on the waiting list or go to the open market via specified transfer. You can get a 4.10 /24 allocation essentially immediately after having an IPv6 allocation provided you use it for the stated purposes.
A /24 is cramped but largely adequate for this purpose for most newly-started network. If you can justify need for more space such as because your NAT overload ratio is becoming untenable, you can request expansion, and ARIN has policies in place to try to make it so that your expansion space will aggregate with your original space without requiring you to re-number. _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5HZMKGRO...
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function). I assume 23.128.0.0/10 records are maintained partially in the "off line Excel file" because your primary system lacks necessary features to differentiate free space / sparse allocations in 23.128.0.0/10 from "regular" free space? I assume the sparse allocation strategy here is to allow a member to come back for more 4.10 space and ideally just extend an original /24 to a /23, perhaps getting the next /24 eventually, and finally swap the adjacent /23 and /24 for the covering /22? Things I'm curious about that are not mentioned in the incident report: 1) When the analyst looked in the e-black-book and selected 23.150.164.0 (23.150.164.0/22, I presume) to be used to satisfy the request they were working on, did they fail to see that this /22 was already in use for a sparse assignment and 23.150.164.0/24 was already assigned, or when 23.150.164.0/24 was originally assigned to AS397031, was the e-black-book not properly updated to reflect the sparse assignment of 23.150.164.0/22? 2) When assigning IP space, is it customary for the analyst to check current/recent snapshots of the global BGP tables to see if the space selected for assignment is currently or has recently been advertised? It's not guaranteed that assigned space will be in the DFZ, but if it is, that's a pretty good indicator that something is going wrong with the current assignment of the space. 3) Did the block split automatically delete any/all child subnets of 23.150.164.0/22, or did the analyst manually delete 23.150.164.0/24 from whois (and that automatically deleted the ROA and rDNS delegation)? ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
See inline (ARIN) John S. From: Jon Lewis via NANOG <nanog@lists.nanog.org> Date: Sunday, December 14, 2025 at 6:55 PM To: John Curran via NANOG <nanog@lists.nanog.org> Cc: John Curran <jcurran@arin.net>, Jon Lewis <jlewis@lewis.org> Subject: Re: Accidental ARIN Reallocation On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function). I assume 23.128.0.0/10 records are maintained partially in the "off line Excel file" because your primary system lacks necessary features to differentiate free space / sparse allocations in 23.128.0.0/10 from "regular" free space? I assume the sparse allocation strategy here is to allow a member to come back for more 4.10 space and ideally just extend an original /24 to a /23, perhaps getting the next /24 eventually, and finally swap the adjacent /23 and /24 for the covering /22? (ARIN) correct, and the community developed policy requires sparse allocation. From the policy "A /24 will be allocated. ARIN should use sparse allocation when possible within that /10 block. “ Things I'm curious about that are not mentioned in the incident report: 1) When the analyst looked in the e-black-book and selected 23.150.164.0 (23.150.164.0/22, I presume) to be used to satisfy the request they were working on, did they fail to see that this /22 was already in use for a sparse assignment and 23.150.164.0/24 was already assigned, or when 23.150.164.0/24 was originally assigned to AS397031, was the e-black-book not properly updated to reflect the sparse assignment of 23.150.164.0/22? (ARIN) The e-black-book (spreadsheet) was not properly updated which led to the analyst selecting that block. 2) When assigning IP space, is it customary for the analyst to check current/recent snapshots of the global BGP tables to see if the space selected for assignment is currently or has recently been advertised? It's not guaranteed that assigned space will be in the DFZ, but if it is, that's a pretty good indicator that something is going wrong with the current assignment of the space. (ARIN) Yes the steps call for the analyst to check for routing, this step was missed. 3) Did the block split automatically delete any/all child subnets of 23.150.164.0/22, or did the analyst manually delete 23.150.164.0/24 from whois (and that automatically deleted the ROA and rDNS delegation)? (ARIN) The split, in this case, returned 23.150.164.0/24, 23.150.165.0/24 and 23.150.166.0/23 so the analyst selected 23.150.164.0/24 for issuance. The next step is to delete it from the ARIN OrgID in order to have it available to allocate to the customer. The fact that it was assigned to OrgID GL-700 versus OrgID ARIN was missed. (ARIN) We have inserted a supervisor mandatory check and approval in the process prior to the delete action being initiated. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/X3WPNTJZ...
Sounds like you’re treating this as a simple error when the root cause is a lack of change management practices. Having a supervisor approve something is meaningless if there is no actual cross-validation of the change and associated data sources. Frankly I agree with others that you are likely not treating this seriously enough. I’ll reiterate what others have said - your whole literal purpose is to manage these assignments. You failed at your fundamental mission. I would recommend an independent audit and review of practices, as well as seeking inspiration from documentation-driven aerospace maintenance practices. - bri On Sun, Dec 14, 2025 at 16:08 John Sweeting via NANOG <nanog@lists.nanog.org> wrote:
See inline (ARIN)
John S.
From: Jon Lewis via NANOG <nanog@lists.nanog.org> Date: Sunday, December 14, 2025 at 6:55 PM To: John Curran via NANOG <nanog@lists.nanog.org> Cc: John Curran <jcurran@arin.net>, Jon Lewis <jlewis@lewis.org> Subject: Re: Accidental ARIN Reallocation
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function).
I assume 23.128.0.0/10 records are maintained partially in the "off line Excel file" because your primary system lacks necessary features to differentiate free space / sparse allocations in 23.128.0.0/10 from "regular" free space? I assume the sparse allocation strategy here is to allow a member to come back for more 4.10 space and ideally just extend an original /24 to a /23, perhaps getting the next /24 eventually, and finally swap the adjacent /23 and /24 for the covering /22?
(ARIN) correct, and the community developed policy requires sparse allocation. From the policy "A /24 will be allocated. ARIN should use sparse allocation when possible within that /10 block. “
Things I'm curious about that are not mentioned in the incident report:
1) When the analyst looked in the e-black-book and selected 23.150.164.0 (23.150.164.0/22, I presume) to be used to satisfy the request they were working on, did they fail to see that this /22 was already in use for a sparse assignment and 23.150.164.0/24 was already assigned, or when 23.150.164.0/24 was originally assigned to AS397031, was the e-black-book not properly updated to reflect the sparse assignment of 23.150.164.0/22?
(ARIN) The e-black-book (spreadsheet) was not properly updated which led to the analyst selecting that block.
2) When assigning IP space, is it customary for the analyst to check current/recent snapshots of the global BGP tables to see if the space selected for assignment is currently or has recently been advertised? It's not guaranteed that assigned space will be in the DFZ, but if it is, that's a pretty good indicator that something is going wrong with the current assignment of the space.
(ARIN) Yes the steps call for the analyst to check for routing, this step was missed.
3) Did the block split automatically delete any/all child subnets of 23.150.164.0/22, or did the analyst manually delete 23.150.164.0/24 from whois (and that automatically deleted the ROA and rDNS delegation)?
(ARIN) The split, in this case, returned 23.150.164.0/24, 23.150.165.0/24 and 23.150.166.0/23 so the analyst selected 23.150.164.0/24 for issuance. The next step is to delete it from the ARIN OrgID in order to have it available to allocate to the customer. The fact that it was assigned to OrgID GL-700 versus OrgID ARIN was missed.
(ARIN) We have inserted a supervisor mandatory check and approval in the process prior to the delete action being initiated.
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/X3WPNTJZ... _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/6AC5KQD2...
On Dec 14, 2025, at 6:54 PM, Jon Lewis via NANOG <nanog@lists.nanog.org> wrote:
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function).
Jon – I agree completely. This was a failure of ARIN in the performance of its core mission, and one that resulted in customer impact. The community is entitled to full transparency in understanding how this occurred and the steps we have taken to prevent any similar incident in the future. Analysts have performed this particular allocation process thousands of times previously without issue, but in this case a resource analyst made an error that resulted in the reallocation of a previously assigned NRPM 4.10 address block. Clearly, while that was the trigger for this incident, the real fault here is that ARIN should not have any processes that are predicated on perfect human performance. Prior to this incident, it was my belief that this was already the case and that assigned resource blocks could not be impacted by analyst error. That is not the case for the manual processes used in the management of NRPM 4.10 address blocks. As a result, we have corrected the process to require a second set of eyes before any change is committed. Longer term, I prefer to fully automate this process, but until that can be implemented we will continue with the manual process, as amended with a mandatory supervisor confirmation step, as a reasonable and appropriate mitigation. I have experience running several major ISPs and am fully aware that operators rely on ARIN for flawless performance. Even a single customer impact is not acceptable, which is why we issued the report to the community detailing the incident and its resolution. To the extent that there is any need for additional clarity, please don’t hesitate to ask – either here on the list (or to me directly as you prefer.) Thanks, /John John Curran President and CEO American Registry for Internet Numbers
Brian – I actually just replied to Jon’s email with very much the same sentiment – this was not a “simple error”; it was an incident that revealed ARIN has operated with manual processes for management of 4.10 address space that lacked appropriate controls to catch human error. That’s a very serious problem, and we have promptly remedied it with appropriate review and cross-validation. We will be reviewing this incident and our broader registry change management practices with the ARIN Board of Trustees, and I will note your suggestions regarding an independent audit [1] and looking to more formal, documentation-driven models (i.e. aerospace) for consideration. Thanks, /John John Curran President and CEO American Registry for Internet Numbers [1] Note that ARIN does periodically engage an independent firm to conduct a full review and audit of the Registration Services Department’s processes and procedures <https://www.arin.net/about/corporate/rsd_audits/> but that is focused on ensuring that registry changes are made consistent with the policies described in the Number Resource Policy Manual (NRPM) – as opposed to your suggestion which implies an audit that is done against best practices in change management.
On Dec 14, 2025, at 11:12 PM, Brian Russo via NANOG <nanog@lists.nanog.org> wrote:
Sounds like you’re treating this as a simple error when the root cause is a lack of change management practices.
Having a supervisor approve something is meaningless if there is no actual cross-validation of the change and associated data sources.
Frankly I agree with others that you are likely not treating this seriously enough. I’ll reiterate what others have said - your whole literal purpose is to manage these assignments. You failed at your fundamental mission.
I would recommend an independent audit and review of practices, as well as seeking inspiration from documentation-driven aerospace maintenance practices.
- bri
On Sun, Dec 14, 2025 at 16:08 John Sweeting via NANOG <nanog@lists.nanog.org> wrote:
See inline (ARIN)
John S.
From: Jon Lewis via NANOG <nanog@lists.nanog.org> Date: Sunday, December 14, 2025 at 6:55 PM To: John Curran via NANOG <nanog@lists.nanog.org> Cc: John Curran <jcurran@arin.net>, Jon Lewis <jlewis@lewis.org> Subject: Re: Accidental ARIN Reallocation
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/
This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function).
I assume 23.128.0.0/10 records are maintained partially in the "off line Excel file" because your primary system lacks necessary features to differentiate free space / sparse allocations in 23.128.0.0/10 from "regular" free space? I assume the sparse allocation strategy here is to allow a member to come back for more 4.10 space and ideally just extend an original /24 to a /23, perhaps getting the next /24 eventually, and finally swap the adjacent /23 and /24 for the covering /22?
(ARIN) correct, and the community developed policy requires sparse allocation. From the policy "A /24 will be allocated. ARIN should use sparse allocation when possible within that /10 block. “
Things I'm curious about that are not mentioned in the incident report:
1) When the analyst looked in the e-black-book and selected 23.150.164.0 (23.150.164.0/22, I presume) to be used to satisfy the request they were working on, did they fail to see that this /22 was already in use for a sparse assignment and 23.150.164.0/24 was already assigned, or when 23.150.164.0/24 was originally assigned to AS397031, was the e-black-book not properly updated to reflect the sparse assignment of 23.150.164.0/22?
(ARIN) The e-black-book (spreadsheet) was not properly updated which led to the analyst selecting that block.
2) When assigning IP space, is it customary for the analyst to check current/recent snapshots of the global BGP tables to see if the space selected for assignment is currently or has recently been advertised? It's not guaranteed that assigned space will be in the DFZ, but if it is, that's a pretty good indicator that something is going wrong with the current assignment of the space.
(ARIN) Yes the steps call for the analyst to check for routing, this step was missed.
3) Did the block split automatically delete any/all child subnets of 23.150.164.0/22, or did the analyst manually delete 23.150.164.0/24 from whois (and that automatically deleted the ROA and rDNS delegation)?
(ARIN) The split, in this case, returned 23.150.164.0/24, 23.150.165.0/24 and 23.150.166.0/23 so the analyst selected 23.150.164.0/24 for issuance. The next step is to delete it from the ARIN OrgID in order to have it available to allocate to the customer. The fact that it was assigned to OrgID GL-700 versus OrgID ARIN was missed.
(ARIN) We have inserted a supervisor mandatory check and approval in the process prior to the delete action being initiated.
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/X3WPNTJZ... _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/6AC5KQD2...
NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/65OO7CNH...
On 15/12/2025 6:55, John Curran via NANOG wrote: A masterclass in owning a mistake and handling it properly. Regards, Hank
On Dec 14, 2025, at 6:54 PM, Jon Lewis via NANOG <nanog@lists.nanog.org> wrote:
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/ This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function). Jon –
I agree completely. This was a failure of ARIN in the performance of its core mission, and one that resulted in customer impact. The community is entitled to full transparency in understanding how this occurred and the steps we have taken to prevent any similar incident in the future.
Analysts have performed this particular allocation process thousands of times previously without issue, but in this case a resource analyst made an error that resulted in the reallocation of a previously assigned NRPM 4.10 address block. Clearly, while that was the trigger for this incident, the real fault here is that ARIN should not have any processes that are predicated on perfect human performance. Prior to this incident, it was my belief that this was already the case and that assigned resource blocks could not be impacted by analyst error.
That is not the case for the manual processes used in the management of NRPM 4.10 address blocks. As a result, we have corrected the process to require a second set of eyes before any change is committed. Longer term, I prefer to fully automate this process, but until that can be implemented we will continue with the manual process, as amended with a mandatory supervisor confirmation step, as a reasonable and appropriate mitigation.
I have experience running several major ISPs and am fully aware that operators rely on ARIN for flawless performance. Even a single customer impact is not acceptable, which is why we issued the report to the community detailing the incident and its resolution. To the extent that there is any need for additional clarity, please don’t hesitate to ask – either here on the list (or to me directly as you prefer.)
Thanks, /John
John Curran President and CEO American Registry for Internet Numbers
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/2F2TMDVE...
Hank, On Dec 14, 2025, at 9:48 PM, Hank Nussbacher via NANOG <nanog@lists.nanog.org> wrote:
A masterclass in owning a mistake and handling it properly.
“Owning”? Sure. "Handling it properly"? Time will tell. The issue here is that RIRs were created to have precisely one job, namely to ensure the allocation of unique resources. Everything else is secondary. And ARIN failed at that one job. It is, of course, true that mistakes (to put it politely) happen. People are fallible, bugs exist, systems crash, etc. What matters in the context of “core mission" is how much the organization's policies, processes, and priorities played in those mistakes. It appears updating systems to address “known weaknesses" was not prioritized, that internal processes were apparently not followed, and that policies were not in place to ensure ARIN could not fail in its core mission. Outside of the impact to the direct customers and a potential degradation in trust in ARIN’s service, there is a larger context: at a time when the RIR system as a whole is facing increased scrutiny due to governance concerns, changes in its operational role due to (and failures in) RPKI, threats from various actors, etc., this isn’t a good look. ARIN has made a number of promises and presumably over time, there will be information about how it is living up to those promises. Hopefully, that information will show ARIN is "handling it properly”. Regards, -drc
ARIN hasn’t exhibited a pattern of this error. They haven’t exhibited a pattern of similar errors. Every single one of us has, at some point made a decision to defer dealing with something because of resourcing, timing, frequency of potential mistake, etc. Anyone who says otherwise is full of shit. There’s no reason to get all verklempt over this. On Mon, Dec 15, 2025 at 16:06 David Conrad via NANOG <nanog@lists.nanog.org> wrote:
Hank,
On Dec 14, 2025, at 9:48 PM, Hank Nussbacher via NANOG < nanog@lists.nanog.org> wrote:
A masterclass in owning a mistake and handling it properly.
“Owning”? Sure. "Handling it properly"? Time will tell.
The issue here is that RIRs were created to have precisely one job, namely to ensure the allocation of unique resources. Everything else is secondary. And ARIN failed at that one job.
It is, of course, true that mistakes (to put it politely) happen. People are fallible, bugs exist, systems crash, etc. What matters in the context of “core mission" is how much the organization's policies, processes, and priorities played in those mistakes. It appears updating systems to address “known weaknesses" was not prioritized, that internal processes were apparently not followed, and that policies were not in place to ensure ARIN could not fail in its core mission. Outside of the impact to the direct customers and a potential degradation in trust in ARIN’s service, there is a larger context: at a time when the RIR system as a whole is facing increased scrutiny due to governance concerns, changes in its operational role due to (and failures in) RPKI, threats from various actors, etc., this isn’t a good look.
ARIN has made a number of promises and presumably over time, there will be information about how it is living up to those promises. Hopefully, that information will show ARIN is "handling it properly”.
Regards, -drc
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/BP4LWM6S...
If this is the first and only incident of a resource being inadvertently reassigned to a different organization, then I’d estimate that ARIN has an error rate that is much, *much* lower than, say, the error rate of DNS registrars handling domain transfers. The key qualifier is the “if” - this could be the only time this has happened, or just the first incident that has public awareness (at least in as long as I’ve been paying attention). While it’s entirely reasonable that ARIN would not report other incidents contemporaneously due to customer privacy concerns, I don’t think it’s inappropriate to ask if there have been other incidents like this, and if so, how many and how recently. If nothing else, I expect that this will be a topic of the next Policy Experience Report, either from the stage or from the mics. Probably both. -Chris
On Dec 15, 2025, at 15:08, Tom Beecher via NANOG <nanog@lists.nanog.org> wrote:
ARIN hasn’t exhibited a pattern of this error. They haven’t exhibited a pattern of similar errors.
Every single one of us has, at some point made a decision to defer dealing with something because of resourcing, timing, frequency of potential mistake, etc. Anyone who says otherwise is full of shit.
There’s no reason to get all verklempt over this.
On Mon, Dec 15, 2025 at 16:06 David Conrad via NANOG <nanog@lists.nanog.org> wrote:
Hank,
On Dec 14, 2025, at 9:48 PM, Hank Nussbacher via NANOG < nanog@lists.nanog.org> wrote:
A masterclass in owning a mistake and handling it properly.
“Owning”? Sure. "Handling it properly"? Time will tell.
The issue here is that RIRs were created to have precisely one job, namely to ensure the allocation of unique resources. Everything else is secondary. And ARIN failed at that one job.
It is, of course, true that mistakes (to put it politely) happen. People are fallible, bugs exist, systems crash, etc. What matters in the context of “core mission" is how much the organization's policies, processes, and priorities played in those mistakes. It appears updating systems to address “known weaknesses" was not prioritized, that internal processes were apparently not followed, and that policies were not in place to ensure ARIN could not fail in its core mission. Outside of the impact to the direct customers and a potential degradation in trust in ARIN’s service, there is a larger context: at a time when the RIR system as a whole is facing increased scrutiny due to governance concerns, changes in its operational role due to (and failures in) RPKI, threats from various actors, etc., this isn’t a good look.
ARIN has made a number of promises and presumably over time, there will be information about how it is living up to those promises. Hopefully, that information will show ARIN is "handling it properly”.
Regards, -drc
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/BP4LWM6S...
NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/T6BYEYDW...
Chris - There have not been other incidents – we publish incident reports for all customer-impacting events (even if just a single customer.) Thanks, /John John Curran President and CEO American Registry for Internet Numbers
On Dec 15, 2025, at 6:50 PM, Chris Woodfield via NANOG <nanog@lists.nanog.org> wrote:
If this is the first and only incident of a resource being inadvertently reassigned to a different organization, then I’d estimate that ARIN has an error rate that is much, *much* lower than, say, the error rate of DNS registrars handling domain transfers. The key qualifier is the “if” - this could be the only time this has happened, or just the first incident that has public awareness (at least in as long as I’ve been paying attention). While it’s entirely reasonable that ARIN would not report other incidents contemporaneously due to customer privacy concerns, I don’t think it’s inappropriate to ask if there have been other incidents like this, and if so, how many and how recently.
If nothing else, I expect that this will be a topic of the next Policy Experience Report, either from the stage or from the mics. Probably both.
-Chris
On Dec 15, 2025, at 15:08, Tom Beecher via NANOG <nanog@lists.nanog.org> wrote:
ARIN hasn’t exhibited a pattern of this error. They haven’t exhibited a pattern of similar errors.
Every single one of us has, at some point made a decision to defer dealing with something because of resourcing, timing, frequency of potential mistake, etc. Anyone who says otherwise is full of shit.
There’s no reason to get all verklempt over this.
On Mon, Dec 15, 2025 at 16:06 David Conrad via NANOG <nanog@lists.nanog.org> wrote:
Hank,
On Dec 14, 2025, at 9:48 PM, Hank Nussbacher via NANOG < nanog@lists.nanog.org> wrote:
A masterclass in owning a mistake and handling it properly.
“Owning”? Sure. "Handling it properly"? Time will tell.
The issue here is that RIRs were created to have precisely one job, namely to ensure the allocation of unique resources. Everything else is secondary. And ARIN failed at that one job.
It is, of course, true that mistakes (to put it politely) happen. People are fallible, bugs exist, systems crash, etc. What matters in the context of “core mission" is how much the organization's policies, processes, and priorities played in those mistakes. It appears updating systems to address “known weaknesses" was not prioritized, that internal processes were apparently not followed, and that policies were not in place to ensure ARIN could not fail in its core mission. Outside of the impact to the direct customers and a potential degradation in trust in ARIN’s service, there is a larger context: at a time when the RIR system as a whole is facing increased scrutiny due to governance concerns, changes in its operational role due to (and failures in) RPKI, threats from various actors, etc., this isn’t a good look.
ARIN has made a number of promises and presumably over time, there will be information about how it is living up to those promises. Hopefully, that information will show ARIN is "handling it properly”.
Regards, -drc
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/BP4LWM6S...
NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/T6BYEYDW...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/XV5MLLVV...
On 15/12/2025 23:06, David Conrad wrote:
there is a larger context: at a time when the RIR system as a whole is facing increased scrutiny due to governance concerns, changes in its operational role due to (and failures in) RPKI, threats from various actors, etc., this isn’t a good look.
ARIN has made a number of promises and presumably over time, there will be information about how it is living up to those promises. Hopefully, that information will show ARIN is "handling it properly”.
Regards, -drc
I'll take ARIN's track record over the past decade then lets say, Cloudflare, AWS, Azure, or name any other major Internet entity. -Hank
It seems like there are lots of folks that use it for direct downstream customer-facing allocations and are not even utilizing them for dual-stack services as intended. I have seen a number of “low end” web hosting companies (folks that advertise on websites like Low End Box) that do just that, as well as some smaller start up ISPs (including one right in my backyard that doesn’t even bother announcing IPv6). Perhaps ARIN is OK with it — but there is probably better use for the space if it’s just being freely given out to those who aren’t willing to use it as intended.
On Dec 12, 2025, at 8:49 PM, Brandon Martin via NANOG <nanog@lists.nanog.org> wrote:
On 12/12/25 15:53, William Herrin via NANOG wrote:
464xlat and similar technologies. Basically, ARIN has a set-aside for folks who have IPv6 devices on an IPv6 network that have a need to also talk to IPv4 devices on the Internet via one of the transition technologies. For that specific use case, there's still a pool of unallocated addresses available.
In addition to address translation targets, they can (per policy) be used for other critical functions that require dual-stacking such as DNS servers (both recursive and authoritative noting that there are still a lot of "target" networks that are IPv6-enabled but don't have dual-stack authoritative nameservers) and things like mail servers and potentially even web servers that require "some rando on the Internet" who may not have IPv6 at all to be able to hit them.
The (somewhat unwritten) intent appears to be to allow a newly started entity to still obtain a minimal IPv4 presence so as to facilitate an "IPv6 first" network deployment without having to wait on the waiting list or go to the open market via specified transfer. You can get a 4.10 /24 allocation essentially immediately after having an IPv6 allocation provided you use it for the stated purposes.
A /24 is cramped but largely adequate for this purpose for most newly-started network. If you can justify need for more space such as because your NAT overload ratio is becoming untenable, you can request expansion, and ARIN has policies in place to try to make it so that your expansion space will aggregate with your original space without requiring you to re-number. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/5HZMKGRO...
On Tue, Dec 16, 2025 at 7:32 PM Tim Burke via NANOG <nanog@lists.nanog.org> wrote:
It seems like there are lots of folks that use it for direct downstream customer-facing allocations and are not even utilizing them for dual-stack services as intended. I have seen a number of “low end” web hosting companies (folks that advertise on websites like Low End Box) that do just that, as well as some smaller start up ISPs (including one right in my backyard that doesn’t even bother announcing IPv6).
Hi Tim, If you know it's 4.10 space (not address space allocated under a different policy section) and you know they're using it for plain IPv4 or generic dual stack, please file a report at https://account.arin.net/public/fraud ARIN takes fraud seriously but they don't have eyes everywhere. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Le 15/12/2025 à 06:48, Hank Nussbacher via NANOG a écrit :
On 15/12/2025 6:55, John Curran via NANOG wrote:
A masterclass in owning a mistake and handling it properly.
Hi Hank, Fully agreed! brother, it's where the difference stands: PmR -- Post-mortem Report! Incidents are common things; PmR is BCOP (Best Operational Practice)...it's still rare :'-( ...and this one gives us an opportunity to ameliorate the BCOP template [*]; imho. __ [*]: dns-post-mortem <https://archive.psg.com/240614.dns-post-mortem> ...i humbly suggest to John and team, to accept to update the structure of their PmR, so that, in its next version, it takes into account all the feedback/suggestions received on topic. What could be mine? That file should be version-controlled & must follow a staging step prior to be committed... ...imho. It's practically what they decided until they are able to fully integrate it into normal pipelines. ...what to do during the staging: cross-check using some API endpoints such as: * Related prefixes <https://stat.ripe.net/docs/data-api/api-endpoints/related-prefixes/> * Prefix overview <https://stat.ripe.net/docs/data-api/api-endpoints/prefix-overview/> * RIS first last seen <https://stat.ripe.net/docs/data-api/api-endpoints/ris-first-last-seen/> * Routing status <https://stat.ripe.net/docs/data-api/api-endpoints/routing-status/> * RPKI history <https://stat.ripe.net/docs/data-api/api-endpoints/rpki-history/> * Routing history <https://stat.ripe.net/docs/data-api/api-endpoints/routing-history/> * Prefix routing consistency <https://stat.ripe.net/docs/data-api/api-endpoints/prefix-routing-consistency/> * RPKI validation <https://stat.ripe.net/docs/data-api/api-endpoints/rpki-validation/> * Reverse DNS consistency <https://stat.ripe.net/docs/data-api/api-endpoints/reverse-dns-consistency/> * <https://stat.ripe.net/docs/data-api/api-endpoints/related-prefixes/>Looking glass <https://stat.ripe.net/docs/data-api/api-endpoints/looking-glass/> * AS routing consistency <https://stat.ripe.net/docs/data-api/api-endpoints/as-routing-consistency/> * * Easy BGP Monitoring with BGPalerter | RIPE Labs <https://labs.ripe.net/author/massimo_candela/easy-bgp-monitoring-with-bgpalerter/> * An Introduction to IRR Explorer | RIPE Labs <https://labs.ripe.net/author/teun/an-introduction-to-irr-explorer/> * BGPAlerter | Loud Networking <https://loudnetworking.org/2021/07/28/bgpalerter/> * . Shalom, --sb.
Regards, Hank
On Dec 14, 2025, at 6:54 PM, Jon Lewis via NANOG <nanog@lists.nanog.org> wrote:
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a public incident report that lays out what happened, the impact, and what we’re changing: https://www.arin.net/announcements/20251212/ This is a pretty epic failure considering ARIN's purpose is the assignment of unique Internet numbers (and the necessary record keeping to facilitate that function). Jon –
I agree completely. This was a failure of ARIN in the performance of its core mission, and one that resulted in customer impact. The community is entitled to full transparency in understanding how this occurred and the steps we have taken to prevent any similar incident in the future.
Analysts have performed this particular allocation process thousands of times previously without issue, but in this case a resource analyst made an error that resulted in the reallocation of a previously assigned NRPM 4.10 address block. Clearly, while that was the trigger for this incident, the real fault here is that ARIN should not have any processes that are predicated on perfect human performance. Prior to this incident, it was my belief that this was already the case and that assigned resource blocks could not be impacted by analyst error.
That is not the case for the manual processes used in the management of NRPM 4.10 address blocks. As a result, we have corrected the process to require a second set of eyes before any change is committed. Longer term, I prefer to fully automate this process, but until that can be implemented we will continue with the manual process, as amended with a mandatory supervisor confirmation step, as a reasonable and appropriate mitigation.
I have experience running several major ISPs and am fully aware that operators rely on ARIN for flawless performance. Even a single customer impact is not acceptable, which is why we issued the report to the community detailing the incident and its resolution. To the extent that there is any need for additional clarity, please don’t hesitate to ask – either here on the list (or to me directly as you prefer.)
Thanks, /John
John Curran President and CEO American Registry for Internet Numbers
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/2F2TMDVE...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/U33VXD6K...
-- Best Regards ! baya.sylvain [AT cmNOG DOT cm] | cmNOG's Structure <https://www.cmnog.cm/dokuwiki/Structure> | CAMIX's Website <https://web.archive.org/web/20240706030230/http://www.camix.cm/> | Douala-IX's Looking Glass <https://tools.std.douala-ix.net/lg> | cmNOG's Surveys <https://survey2.cmnog.cm/> | Subscribe to cmNOG's Mailing List <https://lists.cmnog.cm/mailman/listinfo/cmnog> | __ #LASAINTEBIBLE|#Ephésiens3:17,14-19«17 en sorte que CHRIST habite dans vos cœurs par la foi; afin qu'étant enracinés et fondés dans L'Amour,»#AMEN,#Maranatha,#MerciJÉSUS! #MaPrière est que tu naisses de nouveau.#Chrétiennement
On Wed, Dec 17, 2025 at 12:06 AM William Herrin via NANOG <nanog@lists.nanog.org> wrote:
On Tue, Dec 16, 2025 at 7:32 PM Tim Burke via NANOG <nanog@lists.nanog.org> wrote:
It seems like there are lots of folks that use it for direct downstream customer-facing allocations and are not even utilizing them for dual-stack services as intended. I have seen a number of “low end” web hosting companies (folks that advertise on websites like Low End Box) that do just that, as well as some smaller start up ISPs (including one right in my backyard that doesn’t even bother announcing IPv6).
I don't think, really, there was ever any REAL hope that 100.64 was going to be used for anything except 'more rfc1918'. I'm sure in our heart of hearts we HOPED this would be a bridge element to get to more v6 and less v4, and that MIGHT even be the case sometimes, but.... it's non-globally-unique and people will do with that as they may.
Hi Tim,
If you know it's 4.10 space (not address space allocated under a different policy section) and you know they're using it for plain IPv4 or generic dual stack, please file a report at https://account.arin.net/public/fraud
Is it abuse/fraud if the LIR is treating this as RFC1918 / private space?
ARIN takes fraud seriously but they don't have eyes everywhere.
yes.
On Wed, Dec 17, 2025 at 11:08 AM Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Dec 17, 2025 at 12:06 AM William Herrin via NANOG <nanog@lists.nanog.org> wrote: I don't think, really, there was ever any REAL hope that 100.64 was
If you know it's 4.10 space (not address space allocated under a different policy section) and you know they're using it for plain IPv4 or generic dual stack, please file a report at https://account.arin.net/public/fraud
Is it abuse/fraud if the LIR is treating this as RFC1918 / private space?
Hi Chris, Did I miss the part where we pivoted to talking about RFC 6598 space? I thought we were talking about ARIN NRPM 4.10 space. An ISP can employ 100.64.0.0/10 (RFC 6598 space) any way they choose. ARIN is not a party to that process. It's usually not a good idea to use 100.64.0.0/10 for anything other than numbering downstream customers in conjunction with CGNAT, but that's an operational choice for the ISP. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
As I remember it, the rationale for RFC6596 was to reserve a private address space that specifically was not RFC1918, so that cable providers and other ISPs could have a separate private range to NAT behind that wouldn’t conflict with their customers' 10/8, 192.168/24, etc home networks. This is tangential to any discussion of 4.10 space, which is intended as a IPv4 bridge for IPv6-only networks to NAT into. -Chris
On Dec 17, 2025, at 11:48, Randy Bush via NANOG <nanog@lists.nanog.org> wrote:
I don't think, really, there was ever any REAL hope that 100.64 was going to be used for anything except 'more rfc1918'.
my memory is that was the actual plan and justification. specifically, i think it was the cable folk who wanted it; but i am less sure of that part.
randy _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/ZRREP5QV...
On Wed, Dec 17, 2025 at 2:48 PM Randy Bush <randy@psg.com> wrote:
I don't think, really, there was ever any REAL hope that 100.64 was going to be used for anything except 'more rfc1918'.
my memory is that was the actual plan and justification. specifically, i think it was the cable folk who wanted it; but i am less sure of that part.
yup! basically: "Hey, everyone of our customers uses like 10/8 and 192.168/16 and.. can't we have an answer which isn't their internal space? for our internal space?" great times.
participants (21)
-
Aaron1 -
Andrew Kirch -
Andrew Latham -
Brandon Martin -
Brian Russo -
Chase -
Chris Woodfield -
Christopher Hawker -
Christopher Morrow -
David Conrad -
Hank Nussbacher -
John Curran -
John Sweeting -
Jon Lewis -
Justin Streiner -
Randy Bush -
Sylvain BAYA -
Tim Burke -
Tom Beecher -
Tony Tauber -
William Herrin