Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records? And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that. ~Seth
Perhaps the provider only had a single person maintaining the tooling they used to interact with the IRR records, that person left/was laid off, and it broke. Perhaps they don't have anyone else that can make it work again, and they don't want to hire someone else, so they fell back to paper. Perhaps they have a legal reason to require a paper trail and not rely on IRR records. Plenty of possibilities, all plausible. On Mon, Feb 26, 2024 at 1:58 PM Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
I can’t speak for all providers but when it comes to some downstream networks we will usually request an LOA as additional proof that the customer is authorized to announce the prefixes, in addition to the IRR objects and (where possible) RPKI ROAs. Mainly only a thing where RPKI is not possible and the only route object available is in a non-auth database such as RADB. Overall it helps keep a paper trail (as Tom said) in case someone comes knocking. Kind regards, Peter On Mon, Feb 26, 2024 at 14:13 Tom Beecher <beecher@beecher.cc> wrote:
Perhaps the provider only had a single person maintaining the tooling they used to interact with the IRR records, that person left/was laid off, and it broke. Perhaps they don't have anyone else that can make it work again, and they don't want to hire someone else, so they fell back to paper.
Perhaps they have a legal reason to require a paper trail and not rely on IRR records.
Plenty of possibilities, all plausible.
On Mon, Feb 26, 2024 at 1:58 PM Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
On Mon, 26 Feb 2024 10:57:05 -0800 Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
For routing, some have been proposing that the RPKI. There was some discussion here a few months ago: <https://mailman.nanog.org/pipermail/nanog/2023-November/224035.html> Shortly thereafter this blog post appeared: <https://mailman.nanog.org/pipermail/nanog/2023-November/224035.html>
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
Did you ask them why or can you name the provider? John
A paper LOA is a legally binding document, an IRR record is an IRR record. Falsifying an LOA that is transmitted digitally is wire fraud and can basically be handed right over to a DA for injunction and prosecution. Falsifying IRR records on the other hand leaves more work for the ISP's lawyers to walk a judge (and jury) through the entire purpose and use of that system, as opposed to "here's a super important sheet of paper that they lied on case closed". -Matt On Mon, Feb 26, 2024 at 11:57 AM Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
-- Matt Erculiani
Authentication by letterhead? Paper LOAs are unauthenticated documents, not worth the paper they are written on. Usually FAXed, which is even less authenticatable (is that a word?). Prosecutors are capable of using digital documents. Do it all the time with echecks, credit cards, ecommerce orders and ACH payments. But LOAs are typically civil disputes, not criminal, when someone mistypes an IP address. They should verifiy the information in the paper LOA with a registry anyway. Since LOAs have no intrinsic value, wouldn't be worth the prosecutors time. Usually a salesperson or order entry clerk thinks its required because they've always required it. But no one in the legal department actually knows what to do with a LOA or how to authenticate them. Because carriers never authenticate LOAs. On Mon, 26 Feb 2024, Matt Erculiani wrote:
A paper LOA is a legally binding document, an IRR record is an IRR record. Falsifying an LOA that is transmitted digitally is wire fraud and can basically be handed right over to a DA for injunction and prosecution.
Falsifying IRR records on the other hand leaves more work for the ISP's lawyers to walk a judge (and jury) through the entire purpose and use of that system, as opposed to "here's a super important sheet of paper that they lied on case closed".
-Matt
On Mon, Feb 26, 2024 at 11:57 AM Seth Mattinen via NANOG <nanog@nanog.org> wrote: Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
-- Matt Erculiani
Most important parts on the LOA are the explicit ASN, the name to be found in the cross-connect order portal and local contact data. Contractors need that. Global networks rarely have a contact appropriate for provisioning in a public facing database. On Mon, Feb 26, 2024 at 14:50 Sean Donelan <sean@donelan.com> wrote:
Authentication by letterhead?
Paper LOAs are unauthenticated documents, not worth the paper they are written on. Usually FAXed, which is even less authenticatable (is that a word?).
Prosecutors are capable of using digital documents. Do it all the time with echecks, credit cards, ecommerce orders and ACH payments. But LOAs are typically civil disputes, not criminal, when someone mistypes an IP address.
They should verifiy the information in the paper LOA with a registry anyway. Since LOAs have no intrinsic value, wouldn't be worth the prosecutors time.
Usually a salesperson or order entry clerk thinks its required because they've always required it. But no one in the legal department actually knows what to do with a LOA or how to authenticate them.
Because carriers never authenticate LOAs.
On Mon, 26 Feb 2024, Matt Erculiani wrote:
A paper LOA is a legally binding document, an IRR record is an IRR record. Falsifying an LOA that is transmitted digitally is wire fraud and can basically be handed right over to a DA for injunction and prosecution.
Falsifying IRR records on the other hand leaves more work for the ISP's lawyers to walk a judge (and jury) through the entire purpose and use of that system, as opposed to "here's a super important sheet of paper that they lied on case closed".
-Matt
On Mon, Feb 26, 2024 at 11:57 AM Seth Mattinen via NANOG < nanog@nanog.org> wrote: Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
-- Matt Erculiani
Also known as an cross-connect order form. Why FAX a piece of paper? Nobody cross-checks it, until after it goes wrong. On Mon, 26 Feb 2024, Ren Provo wrote:
Most important parts on the LOA are the explicit ASN, the name to be found in the cross-connect order portal and local contact data. Contractors need that.
Global networks rarely have a contact appropriate for provisioning in a public facing database.
On Mon, Feb 26, 2024 at 14:50 Sean Donelan <sean@donelan.com> wrote: Authentication by letterhead?
Paper LOAs are unauthenticated documents, not worth the paper they are written on. Usually FAXed, which is even less authenticatable (is that a word?).
Prosecutors are capable of using digital documents. Do it all the time with echecks, credit cards, ecommerce orders and ACH payments. But LOAs are typically civil disputes, not criminal, when someone mistypes an IP address.
They should verifiy the information in the paper LOA with a registry anyway. Since LOAs have no intrinsic value, wouldn't be worth the prosecutors time.
Usually a salesperson or order entry clerk thinks its required because they've always required it. But no one in the legal department actually knows what to do with a LOA or how to authenticate them.
Because carriers never authenticate LOAs.
On Mon, 26 Feb 2024, Matt Erculiani wrote: > A paper LOA is a legally binding document, an IRR record is an IRR record. > Falsifying an LOA that is transmitted digitally is wire fraud and can > basically be handed right over to a DA for injunction and prosecution. > > Falsifying IRR records on the other hand leaves more work for the ISP's > lawyers to walk a judge (and jury) through the entire purpose and use of > that system, as opposed to "here's a super important sheet of paper that > they lied on case closed". > > -Matt > > On Mon, Feb 26, 2024 at 11:57 AM Seth Mattinen via NANOG <nanog@nanog.org> > wrote: > Why do companies still insist on, or deploy new systems that > rely on > paper LOA for IP and ASN resources? How can this be considered > more > trustworthy than RIR based IRR records? > > And I'm not even talking about old companies, I have a situation > right > now where a VPS provider I'm using will no longer use IRR and > only > accepts new paper LOAs. In the year 2024. I don't understand how > anyone > can go backwards like that. > > ~Seth > > > > -- > Matt Erculiani > >
One thing that I recently read on this mailing list, is that at least in the US, a transmitting a fraudulent LOA is a federal crime - wire fraud. [0] Being able to hopefully charge and convict someone performing fraud is a useful deterrent. -joe [0] - https://pc.nanog.org/static/published/meetings/NANOG77/2108/20191028_Elverso..., page 13. On 2/26/2024 at 12:58 PM, "Seth Mattinen via NANOG" <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
On Mon, Feb 26, 2024 at 1:20 PM Joe via NANOG <nanog@nanog.org> wrote:
One thing that I recently read on this mailing list, is that at least in the US, a transmitting a fraudulent LOA is a federal crime - wire fraud. [0] Being able to hopefully charge and convict someone performing fraud is a useful deterrent.
This would be just as true of an Emailed declaration signed with the sender's name or other digital representation of a signature. If there is a fraudulent scheme, then deliberately providing a false emailed declaration of authorization just as criminal. My suggestion would be that a LOA should only ever be used as a Supportive document, it could be used for that, and Verifying the data using IRR or RPKI after would still be necessary. An LOA on its own should never be enough. An LOA can still be Incorrect or Wrong due to a Typo'd ASN or IP number, but Not fraudulent. And even if the information is deliberately wrong it might not meet the conditions for fraud. It is also possible the sender of the LOA can send an erroneous document and have No legal responsibility for the results of incorrectly including some IP or AS number on the form. Surely a network service provider must have some level of duty to verify the authenticity of information furnished on the LOAs and confirm that the IP numbers are Not incorrectly entered, for example clerical errors in processing the document.
-joe -- -J
Highly anecdotal, but we’ve always refused to provide them, and they’ve always set it up without an LOA. YMMV since we negotiate larger contracts, but we’ve only ever been asked maybe twice? Both times they admitted they had no idea why they asked for it, so it just seems like some process they forgot to get rid of. -Dan
On Feb 26, 2024, at 13:59, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
There is one purpose: to facilitate IP fraud, and maintain currently fraudulently routed IPs. Anyone can dummy up a LOA. And there is still quite a lot of unrouted IP space. VPS providers know this, and know their customers are submitting fake LOAs. But it is sort of the business VPS providers are in. Is it some sort of serious crime in the US though? Well, just submit the LOA from outside the US. Plus, the entity being defrauded is the IP holder, not the VPS provider or their customer. If you are an IP holder, good luck getting the VPS provider to give you a copy of the fake LOA. It is not in their interest to throw their customers under the bus. You would have to give them a court order. So if you look for unrouted IP space, registered to a non-US organization (ex. Canada), and submit a fake LOA from another country (London, UK for instance), you are unlikely to get tracked down for wire fraud. And you might ask, well, why would a VPS provider accept an LOA from the UK for an IP block registered to a Canadian organization? Well, clearly it isn’t in the VPS provider’s interest to look into the LOAs too much. As long as the IP space is unrouted, they will approve it. The LOA is basically just a liability shield for the VPS provider. It is not a crime to be deceived, though the due diligence beggars belief. So I had this happen. There was a /24 being hijacked by a VPS provider. I told them this was fraud, and they asked me if I wanted to “rescind the LOA”. I told them I never gave them a LOA. They dropped the /24 immediately. They refused to provide a copy of the LOA. So pretty hard to pursue any sort of wire fraud charges. So a VPS provider asking for a paper LOA is basically asking you to lie to them, to protect them from liability. They will just drop the IP prefix if there is any contact from the actual IP holder. Tom
On Feb 26, 2024, at 10:57 AM, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
Hi, (please see inline) On Mon, 26 Feb 2024, Tom Samplonius wrote:
There is one purpose: to facilitate IP fraud, and maintain currently fraudulently routed IPs.
Yes!
Anyone can dummy up a LOA. And there is still quite a lot of unrouted IP space.
Yes. But the endgame is not always the same, when miscreants push fake LOAs (for routing). I was recently made aware about https://loa.tools This is how easy it gets......
VPS providers know this, and know their customers are submitting fake LOAs.
Then it's a good idea to require cryptographic evidence of ownership/authorization, by resorting to RPKI/ROV.
But it is sort of the business VPS providers are in.
That can by true for some. I hope it isn't true for the majority of them.
Is it some sort of serious crime in the US though? Well, just submit the LOA from outside the US. Plus, the entity being defrauded is the IP holder, not the VPS provider or their customer. If you are an IP holder, good luck getting the VPS provider to give you a copy of the fake LOA. It is not in their interest to throw their customers under the bus. You would have to give them a court order. So if you look for unrouted IP space, registered to a non-US organization (ex. Canada), and submit a fake LOA from another country (London, UK for instance), you are unlikely to get tracked down for wire fraud.
Good example, but there are also some less central jurisdictions/coutries/territories, where local law enforcement cooperation is even harder to get. And miscreants know this very well.
And you might ask, well, why would a VPS provider accept an LOA from the UK for an IP block registered to a Canadian organization? Well, clearly it isn?t in the VPS provider?s interest to look into the LOAs too much.
While it doesn't change anything in the "interest" vector, resorting to RPKI/ROV would probably be less work.
As long as the IP space is unrouted, they will approve it. The LOA is basically just a liability shield for the VPS provider. It is not a crime to be deceived, though the due diligence beggars belief.
Even if the IP space is routed, can't anycast be invoked...? :-)))
So I had this happen. There was a /24 being hijacked by a VPS provider. I told them this was fraud, and they asked me if I wanted to ?rescind the LOA?. I told them I never gave them a LOA. They dropped the /24 immediately. They refused to provide a copy of the LOA. So pretty hard to pursue any sort of wire fraud charges.
That's the thing with LOAs for routing, the only way to be sure is to check if there is a valid ROA with the prefix, length and ASN. :-) If the customer can't make a valid ROA, or make the legitimate owner produce one, then the claim on the LOA is bogus...
So a VPS provider asking for a paper LOA is basically asking you to lie to them, to protect them from liability. They will just drop the IP prefix if there is any contact from the actual IP holder.
If the legitimate IP holder has closed shop, there will not be a contact. And miscreants also know this very well... Cheers, Carlos
Tom
On Feb 26, 2024, at 10:57 AM, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
I don't have any examples of anyone still using paper LOAs except for Cogent. Aaron On 2/26/2024 12:57 PM, Seth Mattinen via NANOG wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
We just switched over to IRR routing with Cogent, it is available. It's just not on by default. Best Regards, Jason On 2/26/24 3:14 PM, Aaron Wendel wrote:
I don't have any examples of anyone still using paper LOAs except for Cogent.
Aaron
On 2/26/2024 12:57 PM, Seth Mattinen via NANOG wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
On 2/26/24 10:57, Seth Mattinen via NANOG wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
* They're an authoritative signed document with legal penalties for forgery. * The same LOA is often required by datacenter operators and other third parties for cross-connect authority, etc. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Hi Seth, LOAs can't be considered more trustworthy than IRR objects. The RIRs operate IRRdb services as part of the services they offer which network operators should be using instead of the free and paid non-authoritative IRRdb operators. If you don’t mind, could you please reach out to me off-list with who the VPS hosting provider is that is only accepting LOAs? I’d like to reach out to them to discuss their decision. I’m doing a talk at APRICOT 2024 on using ROAs to replace LOAs. In my view there's no reason why network operators cannot use ROAs instead to validate the routes received from their peers, be they upstream or downstream. Regards, Christopher Hawker Sent from my iPhone On 27 Feb 2024, at 1:57 am, Seth Mattinen via NANOG <nanog@nanog.org> wrote: Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records? And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that. ~Seth
Hi All, There is this blogpost from the FIRST netsec-sig group, about this topic, available at https://www.first.org/blog/20231222-Is-the-LoA-DoA-for-Routing I totally agree with Christopher. The above blogpost ends with (for those who don't like to follow links): "With the current level of RPKI adoption, now is time to adopt it as the best current practice, to discontinue the usage of LOAs for authorization of routing, and to instead rely on ROV, ROAs, and the cryptographic trust we all can obtain from them!" Best Regards, Carlos On Tue, 27 Feb 2024, Christopher Hawker wrote:
Hi Seth,
LOAs can't be considered more trustworthy than IRR objects. The RIRs operate IRRdb services as part of the services they offer which network operators should be using instead of the free and paid non-authoritative IRRdb operators.
If you don?t mind, could you please reach out to me off-list with who the VPS hosting provider is that is only accepting LOAs? I?d like to reach out to them to discuss their decision.
I?m doing a talk at APRICOT 2024 on using ROAs to replace LOAs. In my view there's no reason why network operators cannot use ROAs instead to validate the routes received from their peers, be they upstream or downstream.
Regards, Christopher Hawker
Sent from my iPhone
On 27 Feb 2024, at 1:57?am, Seth Mattinen via NANOG <nanog@nanog.org> wrote:
Why do companies still insist on, or deploy new systems that rely on paper LOA for IP and ASN resources? How can this be considered more trustworthy than RIR based IRR records?
And I'm not even talking about old companies, I have a situation right now where a VPS provider I'm using will no longer use IRR and only accepts new paper LOAs. In the year 2024. I don't understand how anyone can go backwards like that.
~Seth
participants (16)
-
Aaron Wendel -
Carlos Friaças -
Christopher Hawker -
Daniel Marks -
Jason Canady -
Jay Acuna -
Jay Hennigan -
joenanog＠nym.hush.com -
John Kristoff -
Matt Erculiani -
Peter Potvin -
Ren Provo -
Sean Donelan -
Seth Mattinen -
Tom Beecher -
Tom Samplonius