Questions for Level3 & Choopa about their Enabling of IP Hijacking (RPKI Invalid)
Is there anyone on this list at Level3 or Choopa who can respond to why: 1. Level3/Centurylink/Lumen: How come RPKI invalid prefixes were allowed to be announced and considered valid? 2. Level3/Centurylink/Lumen: Is there any actual NOC hidden behind the numbers where someone can reasonably be of assistance? Why are your SOC, Support and DDoS team saying you can't blackhole a hijacked prefix because of "GOVERNMENT regulations"? 3. CHOOPA: Why does your Vultr brand allow anyone who can successfully insert an IRR record (eg, in RaDB) to be imported (even when RPKI is invalid) and permit hijacking? 4. CHOOPA: Why does your Network Team need 3 days to check an RPKI valid or invalid? It is not rocket science. Please feel free to contact me off the list with these answers. If you are interested in the long story of the whole ordeal of being hijacked for 3 days, it is below. IP hijacking ordeal we went through: We are extremely disappointed with Choopa/Vultr and Level3. On 2020-12-07 at exactly 23:48:10, we notified Choopa and Vultr via email, ticket and contact form that one of our IPv4 prefixes was being hijacked by one of their customers. Our prefixes are RPKI signed, and the ASN announcing the prefix was not in the RPKI sign. We continued to follow up on the request, and sent more requests in to Vultr/Choopa's system. When we phoned Choopa, we were told by the individual on the phone that they see the ticket, and they'll bump it up on the network engineering ticket list. Frustrated after three days of an ongoing hijack, and consistently no assistance from Vultr or it's parent Choopa, we reached out to the only Tier-1 Choopa has in London that was NOT filtering according to RPKI, Level3. We sent an email to the noc@level3carrier.com, which was listed as a point of contact on the ASN & PeeringDB pages. The email had no response. We then called the number (1-833-453-8353), and spoke to Technical Support, who transferred us the first time to the SOC, the second time to their DDoS department. The person who responded in the SOC said that they can filter it "very fast" if we validate ownership of the prefix (despite being RPKI signed). So we complied, we were told to email "abuse@centurylinkservices.net" with a message saying the prefix is being hijacked, the ASN of the hijacker and the direct upstream (Vultr/Choopa). We sent that email, he said we "should get" an automatic reply (none arrived, confirmed with mx it was delivered). We waited half an hour, which they agreed was a "reasonable time" to wait for it to be filtered. After half an hour, we followed up, and this time landed on the DDoS department (I have no clue how they thought this through). At the DDoS department, they said that they can't help, and I should "keep emailing" abuse@centurylinkservices.net. He offered a phone number for me to call as a "direct line to abuse", upon hanging up and dialing, I got to the generic prompts for *customer services*. They were no help either. Fast forward four hours, we have no point of contact at Level3 or Choopa, we have had no communication from either. We finally get a message from Choopa, reading: "Greetings, This ticket has been forwarded to our networking team so they can examine your situation, check the infrastructure configuration, and apply any relevant changes. Please allow for significant additional time while we review this ticket." "Significant additional time" to check an RPKI valid? That seems incredibly odd. Fast forward to another hour, we receive the next of Choopa messages, this time saying: "Thanks for the update on this. We have validated the removal request and have removed the prefix from our network. Please allow additional time for this to update to the providers.". We finally thought we were in the clear. Almost two hours go by, and the prefix is still not filtered what-so-ever. We follow up once more, and are told "We have removed the prefix and is not announced by Vultr anymore. Please allow 24 to 48 hours for the internet providers to update their routing database." Finally, an hour later, nlnog & other lg's are seeing Level3 no longer announcing the prefix. What we found out in the process is Vultr ignores RPKI invalid (despite having a table on their system which shows RPKI Invalid/Invalid ASN), as long as at one point in time an IRR record existed (or, is created). Once Vultr gets their hands on it, they make the IRR records at RaDB, and keep updating them even when they're not valid. With the run around of Level3, and Choopa/Vultr, they're practically inviting IP hijackers to play. Insanity!
participants (1)
-
listsīŧ geeksolutions.ca