Dear all, I analysed the alert, here is my assessment. If I recall correctly, Packetvis uses multiple data sources (different versions of validator implementations) and alerts on anomalies spotted by more than a single data source. Most RPKI Validator implementations limit the maximum allowable file size of RPKI Signed Objects. It appears one particular Manifest in ARIN's "Hosted CA" system exceeded a threshold known to exist in older implementations, rendering all subordinate ROAs on that manifest invalid for those instances. Timeline (based on rpkiviews.org): * On Tue 28 Jan 2025 14:02:07 +0000, one Large CA's keypair signed a Manifest with FileAndHash 50,157 entries. This manifest was nominally valid until Thu 30 Jan 2025 10:00:00 +0000 and 3,964,618 bytes in size. Note that this is very large compared to other Manifest objects: it is 174% larger than the second largest Manifest, and 456% larger than the third largest Manifest. * On Tue 28 Jan 2025 17:54:07 +0000 this Large CA's keypair signed a Manifest with FileAndHash 51,014 entries. That particular issuance was 4,032,321 bytes in size and exceeded a threshold (4M bytes). Following the "Failed Fetch" mechanism described in RFC 9286 Section 6.6, affected instances continued to use the older "Tue 28 Jan 2025 14:02:07" manifest, until it expired (which happened today at Thu 30 Jan 2025 10:00:00). It is interesting that the 'trigger event' happened two days ago, but it is only just now that it became quite tangible! It seems this anomaly could've been alerted for earlier on. I noted in my "RPKI's 2024 Year In Review" report: """ "Efficiency" in this context arises from validators spending the computational cost of validating a single EE certificate and yielding more than 1 ROAIPAddress. Under the RIPE NCC TAL one yields 6.5 prefixes per ROA, while in the ARIN region this is number is 1.1 prefixes per ROA. As stewards of this technology, we need to keep an eye on the overall efficiency of the RPKI to ensure things don't get out of hand. """ source: https://mailman.nanog.org/pipermail/nanog/2025-January/227166.html When a resource holder creates many ROAs (tens of thousands), it'll result in many Manifest FileAndHash entries (again, tens of thousands), which increases the file size of the Manifest (to the point that some validators may consider such a Manifest object invalid). When this happens, the validator marks ROAs as invalid, in turn BGP routers will considered covered routes 'not-found'. Another downside of systems signing over only a single prefix per ROA is that each individual ROA object comes with 1500~2100 bytes 'overhead' regardless of how many prefixes are encoded inside of it (due to the embedded X.509 End-Entity certificate and signature). Expressed as a percentage, this overhead is ~ 98% in the case of single prefixes. While Manifests grow linearly, the per-ROA overhead makes for a somewhat steep curve in turn directly impact size of RRDP snapshots (in which all Manifests + ROAs are bundled together). Publication point operators are recommended to keep in mind that RRDP snapshot size is another limit that can be tripped. Stakeholders operating certification services should keep in mind that validator implementations might restrict the file size of individual objects, and the number of objects, but also impose limits on the size of the RRDP snapshot, the duration of the synchronization task, etc. Kind regards, Job On Thu, Jan 30, 2025 at 10:43:52AM +0000, Christopher Hawker wrote:
Hello folks,
Has anyone received any similar event notifications (from PacketVis or other)? Trying to work out if it's a false-positive.
Regards, Christopher Hawker ________________________________ From: PacketVis <notifications@packetvis.com> Sent: Thursday, January 30, 2025 9:40 PM To: Christopher Hawker <chris@thesysadmin.au> Subject: bgp ta-malfunction - low severity - PacketVis
Possible TA malfunction: 29.17% of the ROAs disappeared from ARIN.
Type: ta-malfunction Severity: low Monitored: ASarin When: 2025-01-30 10:40 UTC
See more details about the event: https://packetvis.com/bgp/event/554c1692012ae202467afe3a38ae5075-dca21ae6-ff...