Reaching out to ARIN members about their RPKI INVALID prefixes
Dear NANOG, when I approached ARIN about how they feel about reaching out to their members about prefixes that are unreachable in a route origin validation (ROV) environment, John Curran (CEO ARIN) referred me to you (see email bellow - quoted with permission). The question I asked ARIN was specifically:
Would you be open to reach out to your affected members to inform them about their affected IP prefixes?
John Curran (CEO ARIN) wrote:
If there is evidence of community Interest, then ARIN can conduct a community consultation to determine our best role in this area, but you first should encourage discussion within the network operator community at appropriate forums.
So here is my question to the network operator community in the ARIN region to gather if there are any (dis)agreements/opinions about such a notification by ARIN: What do you think about the idea that ARIN actively informs their affected members about prefixes that are unreachable in an RPKI ROV environment? The goal of that outreach/notification would be - to reduce the number of broken legacy ROAs from the past - reduce the negative impact on reachability of affected members. looking forward to receiving your feedback! kind regards, nusenu [1] https://medium.com/@nusenu/towards-cleaning-up-rpki-invalids-d69b03ab8a8c John Curran wrote:
Subject: Reaching out to ARIN members about their RPKI INVALID prefixes
Nusenu -
Thank you for writing us - the project (and Medium post on same) are quite interesting.
I think you’ve got several options for pursuing your objectives, including –
1) Reaching out to parties that already track and report on Internet routing hygiene (e.g. Geoff Huston at http://bgp.potaroo.net, the RPKI validator team at RIPE, the NIST RPKI Deployment monitor - https://rpki-monitor.antd.nist.gov) to see if of them would like to report on this information and/or contact those with invalids)
2) Raising the issue in the ARIN region via the NANOG operator forum - this would make an excellent lightening talk for you (or someone else familiar with it already attending) to speak about at the upcoming NANOG Vancouver meeting. If there is evidence of community Interest, then ARIN can conduct a community consultation to determine our best role in this area, but you first should encourage discussion within the network operator community at appropriate forums. It is not appropriate for ARIN staff to be proposing this additional role for the organization, as we within the ARIN staff follow community direction rather than set it.
Thanks! /John
John Curran President and CEO ARIN
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole. Owen
On Sep 18, 2018, at 4:56 AM, nusenu <nusenu-lists@riseup.net> wrote:
Dear NANOG,
when I approached ARIN about how they feel about reaching out to their members about prefixes that are unreachable in a route origin validation (ROV) environment, John Curran (CEO ARIN) referred me to you (see email bellow - quoted with permission).
The question I asked ARIN was specifically:
Would you be open to reach out to your affected members to inform them about their affected IP prefixes?
John Curran (CEO ARIN) wrote:
If there is evidence of community Interest, then ARIN can conduct a community consultation to determine our best role in this area, but you first should encourage discussion within the network operator community at appropriate forums.
So here is my question to the network operator community in the ARIN region to gather if there are any (dis)agreements/opinions about such a notification by ARIN:
What do you think about the idea that ARIN actively informs their affected members about prefixes that are unreachable in an RPKI ROV environment?
The goal of that outreach/notification would be - to reduce the number of broken legacy ROAs from the past - reduce the negative impact on reachability of affected members.
looking forward to receiving your feedback!
kind regards, nusenu
[1] https://medium.com/@nusenu/towards-cleaning-up-rpki-invalids-d69b03ab8a8c
John Curran wrote:
Subject: Reaching out to ARIN members about their RPKI INVALID prefixes
Nusenu -
Thank you for writing us - the project (and Medium post on same) are quite interesting.
I think you’ve got several options for pursuing your objectives, including –
1) Reaching out to parties that already track and report on Internet routing hygiene (e.g. Geoff Huston at http://bgp.potaroo.net, the RPKI validator team at RIPE, the NIST RPKI Deployment monitor - https://rpki-monitor.antd.nist.gov) to see if of them would like to report on this information and/or contact those with invalids)
2) Raising the issue in the ARIN region via the NANOG operator forum - this would make an excellent lightening talk for you (or someone else familiar with it already attending) to speak about at the upcoming NANOG Vancouver meeting. If there is evidence of community Interest, then ARIN can conduct a community consultation to determine our best role in this area, but you first should encourage discussion within the network operator community at appropriate forums. It is not appropriate for ARIN staff to be proposing this additional role for the organization, as we within the ARIN staff follow community direction rather than set it.
Thanks! /John
John Curran President and CEO ARIN
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Owen, On Tue, Sep 18, 2018 at 10:23:42AM -0700, Owen DeLong wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole.
1/ You may be overlooking the fact that many networks peer directly with what (for them) are the important sources/destinations. The semantics of RPKI ROAs help block illegitimate more-specifics, and the short AS_PATH between players prevents a hijacker from inserting themself. In other words - the most important AS_PATHs are 1 hop. The Internet's dense interconnectedness is saving its bacon. 2/ Another approach to achieve path validation for 1 hop is through mechanisms such what NTT calls 'peerlock'. https://www.youtube.com/watch?v=CSLpWBrHy10 3/ Lastly, some folks are innovating in this space to help automate concepts such as peerlock through what is called ASPA. ASPA is intended as an out-of-band, deployable alternative to BGPSec. https://tools.ietf.org/html/draft-azimov-sidrops-aspa-profile https://tools.ietf.org/html/draft-azimov-sidrops-aspa-verification I think you underestimate how valuable RPKI based Origin Validation (even just by itself) is in today's Internet landscape. If you are aware of other efforts or more fruitful approaches please let us know. Kind regards, Job
On Tue, Sep 18, 2018 at 10:36 AM Job Snijders <job@ntt.net> wrote:
Owen,
On Tue, Sep 18, 2018 at 10:23:42AM -0700, Owen DeLong wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole. I think you underestimate how valuable RPKI based Origin Validation (even just by itself) is in today's Internet landscape.
If you are aware of other efforts or more fruitful approaches please let us know.
Perhaps said another way: "How would you figure out what prefixes your bgp peer(s) should be sending you?" (in an automatable, and verifiable manner) -chris
On Sep 18, 2018, at 11:06 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Sep 18, 2018 at 10:36 AM Job Snijders <job@ntt.net <mailto:job@ntt.net>> wrote: Owen,
On Tue, Sep 18, 2018 at 10:23:42AM -0700, Owen DeLong wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole. I think you underestimate how valuable RPKI based Origin Validation (even just by itself) is in today's Internet landscape.
If you are aware of other efforts or more fruitful approaches please let us know.
Perhaps said another way:
"How would you figure out what prefixes your bgp peer(s) should be sending you?" (in an automatable, and verifiable manner)
-chris
In theory, that’s what IRRs are for. In practice, while they offer better theoretical capabilities if stronger authentication were added, the current implementation and acceptance leaves much to be desired. However, even in theory, RPKI offers nothing of particular benefit even in its best case of widespread implementation. Owen
On Sep 18, 2018, at 3:04 PM, Owen DeLong <owen@delong.com> wrote:
On Sep 18, 2018, at 11:06 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Sep 18, 2018 at 10:36 AM Job Snijders <job@ntt.net> wrote: Owen,
On Tue, Sep 18, 2018 at 10:23:42AM -0700, Owen DeLong wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole. I think you underestimate how valuable RPKI based Origin Validation (even just by itself) is in today's Internet landscape.
If you are aware of other efforts or more fruitful approaches please let us know.
Perhaps said another way:
"How would you figure out what prefixes your bgp peer(s) should be sending you?" (in an automatable, and verifiable manner)
-chris
In theory, that’s what IRRs are for.
In practice, while they offer better theoretical capabilities if stronger authentication were added, the current implementation and acceptance leaves much to be desired.
Judging a global ecosystem just by what ARIN does is perhaps some of the issue. ARIN seems to be the outlier here as has been measured. An ARIN prefix ROA is less valuable than the other regions and this is IMO deliberate on the part of ARIN.
However, even in theory, RPKI offers nothing of particular benefit even in its best case of widespread implementation.
Disagree, but that’s ok. I know at $dayJob I’m preparing the way, but it’s much harder than it should be due to the nature of our business. - Jared
On Sep 18, 2018, at 12:09 PM, Jared Mauch <jared@puck.nether.net> wrote:
On Sep 18, 2018, at 3:04 PM, Owen DeLong <owen@delong.com> wrote:
On Sep 18, 2018, at 11:06 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Sep 18, 2018 at 10:36 AM Job Snijders <job@ntt.net> wrote: Owen,
On Tue, Sep 18, 2018 at 10:23:42AM -0700, Owen DeLong wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole. I think you underestimate how valuable RPKI based Origin Validation (even just by itself) is in today's Internet landscape.
If you are aware of other efforts or more fruitful approaches please let us know.
Perhaps said another way:
"How would you figure out what prefixes your bgp peer(s) should be sending you?" (in an automatable, and verifiable manner)
-chris
In theory, that’s what IRRs are for.
In practice, while they offer better theoretical capabilities if stronger authentication were added, the current implementation and acceptance leaves much to be desired.
Judging a global ecosystem just by what ARIN does is perhaps some of the issue. ARIN seems to be the outlier here as has been measured. An ARIN prefix ROA is less valuable than the other regions and this is IMO deliberate on the part of ARIN.
However, even in theory, RPKI offers nothing of particular benefit even in its best case of widespread implementation.
Disagree, but that’s ok. I know at $dayJob I’m preparing the way, but it’s much harder than it should be due to the nature of our business.
- Jared
What does RPKI offer other than a way to know what to spoof in a prepend for your forged announcement? Owen
On Tue, Sep 18, 2018 at 12:04 PM Owen DeLong <owen@delong.com> wrote:
On Sep 18, 2018, at 11:06 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Sep 18, 2018 at 10:36 AM Job Snijders <job@ntt.net> wrote:
Owen,
On Tue, Sep 18, 2018 at 10:23:42AM -0700, Owen DeLong wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole. I think you underestimate how valuable RPKI based Origin Validation (even just by itself) is in today's Internet landscape.
If you are aware of other efforts or more fruitful approaches please let us know.
Perhaps said another way:
"How would you figure out what prefixes your bgp peer(s) should be sending you?" (in an automatable, and verifiable manner)
-chris
In theory, that’s what IRRs are for.
it's not worked out so far. there's no real authorization/authentication of note on the data set via the irr. you have no real way of knowing that 'as12 should be announcing 157.130.0.0/16' ... except by chasing the arin/ripe/etc records today, something that those orgs stamp and which machines could validate without people using eyeballs would sure be nice... Oh, that's what RPKI is supposed to provide.
In practice, while they offer better theoretical capabilities if stronger authentication were added, the current implementation and acceptance leaves much to be desired.
and has for approximately 30 yrs... I don't imagine magically it's going to get better in the next 30 either.
However, even in theory, RPKI offers nothing of particular benefit even in
its best case of widespread implementation.
"rir says owen can originate route FOO" "ROA for 157.130.1.0/24 says OWEN can originate" those seem like valuable pieces of information. Especially since I can know this through some machine parseable fashion. -chris
Owen
"rir says owen can originate route FOO" "ROA for 157.130.1.0/24 <http://157.130.1.0/24> says OWEN can originate"
Nope… ROA says (e.g.) AS1734 (or anyone willing to impersonate AS1734) can originate 192.159.10.0/24.
those seem like valuable pieces of information. Especially since I can know this through some machine parseable fashion.
I would agree if you had some way to distinguish AS1734 originating FOO from AS174 originating FOO with a prepend of AS1734. Owen
On Tue, Sep 18, 2018 at 02:35:44PM -0700, Owen DeLong wrote:
"rir says owen can originate route FOO" "ROA for 157.130.1.0/24 says OWEN can originate"
Nope… ROA says (e.g.) AS1734 (or anyone willing to impersonate AS1734) can originate 192.159.10.0/24.
I'd phrase slightly different (assuming there is only one ROA): the ROA says ONLY AS1734 (or anyone willing to impersonate AS1734) can originate 192.159.10.0/24. With IRR, the crucial addition of the word "ONLY" in the above sentence is not possible.
those seem like valuable pieces of information. Especially since I can know this through some machine parseable fashion.
I would agree if you had some way to distinguish AS1734 originating FOO from AS174 originating FOO with a prepend of AS1734.
In the common scenario you can distinguish those with today's technology. As mentioned before - dense peering (having the shortest AS_PATH) or the peerlock approach for all *practical* intents solve the issue of path validation. You can try spoofing AS 7018 - you'll notice that your announcements won't make it into NTT. Having just that validated path (which is only one hop) is a very strong defense mechanism. Let's take another example: Google offers access to one of the world's largest DNS resolver services, Cloudflare hosts lots of authoritative DNS. According to PeeringDB they have quite some locations in common with each other so let's assume they directly peer with each other. If both sides create ROAs for their prefixes, and ROAs for the prefixes that host the auth side, and both sides perform RPKI Origin Validation; an attacker cannot inject itself between the two. One can argue "this only helped google and cloudflare" - but on the other hand anno 2018 the Internet experience for the average end user is composed from services hosted by a relatively small number of providers. Preventing disruptions of communication between just those few players has significant implications for all of us. Kind regards, Job
On Sep 18, 2018, at 14:58 , Job Snijders <job@ntt.net> wrote:
On Tue, Sep 18, 2018 at 02:35:44PM -0700, Owen DeLong wrote:
"rir says owen can originate route FOO" "ROA for 157.130.1.0/24 says OWEN can originate"
Nope… ROA says (e.g.) AS1734 (or anyone willing to impersonate AS1734) can originate 192.159.10.0/24.
I'd phrase slightly different (assuming there is only one ROA): the ROA says ONLY AS1734 (or anyone willing to impersonate AS1734) can originate 192.159.10.0/24.
With IRR, the crucial addition of the word "ONLY" in the above sentence is not possible.
That depends. If you ONLY allow the maintainer of NET-192.159.10.0/24 to update the route objects for it, then the word ONLY is effectively present by the lack of any other route objects.
those seem like valuable pieces of information. Especially since I can know this through some machine parseable fashion.
I would agree if you had some way to distinguish AS1734 originating FOO from AS174 originating FOO with a prepend of AS1734.
In the common scenario you can distinguish those with today's technology. As mentioned before - dense peering (having the shortest AS_PATH) or the peerlock approach for all *practical* intents solve the issue of path validation. You can try spoofing AS 7018 - you'll notice that your announcements won't make it into NTT. Having just that validated path (which is only one hop) is a very strong defense mechanism.
You chopped out my example, which had equal AS Path lengths. Sure, I might not be able to announce something to you for an AS that you’re directly peered with, but I can still spoof pretty much anything that’s more than one hop away as long as I can get one of your peers to pass it along.
Let's take another example: Google offers access to one of the world's largest DNS resolver services, Cloudflare hosts lots of authoritative DNS. According to PeeringDB they have quite some locations in common with each other so let's assume they directly peer with each other. If both sides create ROAs for their prefixes, and ROAs for the prefixes that host the auth side, and both sides perform RPKI Origin Validation; an attacker cannot inject itself between the two.
Again, you’re reducing the problem set to single hop which I admit RPKI solves (though I’d argue that if someone has access to insert themselves between the two physically, RPKI does little to help)
One can argue "this only helped google and cloudflare" - but on the other hand anno 2018 the Internet experience for the average end user is composed from services hosted by a relatively small number of providers. Preventing disruptions of communication between just those few players has significant implications for all of us.
So RPKI is great if we can just reduce the internet diameter to 1, in which case MD5 passwords on your BGP sessions pretty much accomplishes the same thing with a lot less kerfuffle. Owen
On Tue, Sep 18, 2018 at 06:18:00PM -0700, Owen DeLong wrote:
That depends. If you ONLY allow the maintainer of NET-192.159.10.0/24 to update the route objects for it, then the word ONLY is effectively present by the lack of any other route objects.
Ah, so you are now applying the RPKI Origin Validation procedure to a non-existent flavor of IRR? :-) Today I can create route-objects covering 192.159.10.0/24 in a number of IRR databases and there is nothing you can do to prevent that, this simply is not the case with RPKI. I prefer an existing system (RPKI) over hypotheticals (Owen's IRR). Secondly, I've also noticed you only emphasize an adversarial angle (origin spoofing), but there are other angles too. The majority of today's BGP problems are attributable to operator mistakes (misconfigurations). Analysis has shown that most BGP incidents happen on weekdays rather than in the weekend. The number keys on our keyboards are quite close to each other and Origin Validation is very effective against typos. Another angle is bugs in BGP implementations: your neighbors doing origin validation reduces the impact and propagation of incorrect announcements from your network should you run into a software defect.
So RPKI is great if we can just reduce the internet diameter to 1
Agreed, in other words: RPKI is offers tangible benefits to those that peer directly with each other, or use peerlock.
in which case MD5 passwords on your BGP sessions pretty much accomplishes the same thing with a lot less kerfuffle.
Uhh... you may want to refresh your memory on what BGP is and how TCP-MD5 works. Kind regards, Job
in which case MD5 passwords on your BGP sessions pretty much accomplishes the same thing with a lot less kerfuffle.
oh gosh, sorry I missed this in the previous conversation... for folk following along at home: TCP-MD5 is really REALLY just: "better CRC(checksum)" on your BGP session, and is in no way related to which routes your bgp-peer should/could/will be sending you over that peering. Please do not confuse/conflate BGP / TCP-MD5 with routing-security :( Steve Bellovin would be shocked and appalled at such conflation.
On Sep 19, 2018, at 00:44 , Job Snijders <job@ntt.net> wrote:
On Tue, Sep 18, 2018 at 06:18:00PM -0700, Owen DeLong wrote:
That depends. If you ONLY allow the maintainer of NET-192.159.10.0/24 to update the route objects for it, then the word ONLY is effectively present by the lack of any other route objects.
Ah, so you are now applying the RPKI Origin Validation procedure to a non-existent flavor of IRR? :-)
Today I can create route-objects covering 192.159.10.0/24 in a number of IRR databases and there is nothing you can do to prevent that, this simply is not the case with RPKI. I prefer an existing system (RPKI) over hypotheticals (Owen's IRR).
Secondly, I've also noticed you only emphasize an adversarial angle (origin spoofing), but there are other angles too.
The majority of today's BGP problems are attributable to operator mistakes (misconfigurations). Analysis has shown that most BGP incidents happen on weekdays rather than in the weekend. The number keys on our keyboards are quite close to each other and Origin Validation is very effective against typos. Another angle is bugs in BGP implementations: your neighbors doing origin validation reduces the impact and propagation of incorrect announcements from your network should you run into a software defect.
Sure, but given the email thread that started this all, if people start enforcing RPKI based origin validation, do you think it will create fewer or more mistakes in this regard? It appears to me that the complexity of RPKI and other issues will lead to RPKI causing more human-factors based routing accidents than it will likely prevent.
So RPKI is great if we can just reduce the internet diameter to 1
Agreed, in other words: RPKI is offers tangible benefits to those that peer directly with each other, or use peerlock.
Agreed, noting the assumptions built into the theory that everyone can use peerlock.
in which case MD5 passwords on your BGP sessions pretty much accomplishes the same thing with a lot less kerfuffle.
Uhh... you may want to refresh your memory on what BGP is and how TCP-MD5 works.
Admittedly, it doesn’t cover the fat finger cases above, but, it does cover the “know who you’re accepting the route from” issue for one hop out and in a way that doesn’t allow you to create a time-bomb that becomes visible on a delayed basis when you fat-finger it. Owen
On Tue, Sep 18, 2018 at 12:04:19PM -0700, Owen DeLong wrote:
Perhaps said another way:
"How would you figure out what prefixes your bgp peer(s) should be sending you?" (in an automatable, and verifiable manner)
In theory, that’s what IRRs are for.
You may be overlooking the fact that the semantics of an IRR route object are subtly different than those of a RPKI ROA. The Prefix-to-AS Mapping Database concept as introduced Section 2 of RFC 6811 is a huge step forward compared to the (somewhat loosely defined) semantics of IRR route objects. RPKI ROAs are more than "IRR with crypto": IRR objects are basically a list of unverifiable attestations with no proof of termination. Whereas when that same information is published through the RPKI, we now know two things: (1) the owner of the resource consented to the creation of the object and (2) any BGP UPDATE that conflicts with covering RPKI ROAs is invalid. In other words, RPKI and the Prefix-to-AS validation procedure give us much more definitive inputs for routing policies compared to what can be derived from the IRR. I simply view RPKI as a successor to the IRR system with some much needed improvements.
In practice, while they offer better theoretical capabilities if stronger authentication were added, the current implementation and acceptance leaves much to be desired.
Luckily multiple projects are passing through the pipeline to reduce the risk some IRRs represented. https://www.ripe.net/manage-ips-and-asns/db/impact-analysis-for-nwi-5-implem... https://teamarin.net/2018/07/12/the-path-forward/ Considering that RPKI and IRR will co-exist in one form or another for a wihle it is encouraging to see success in patching loopholes.
However, even in theory, RPKI offers nothing of particular benefit even in its best case of widespread implementation.
I can't really wrap my head around how you can arrive at such a conclusion in light of all the information that has been provided to you. So perhaps we'll have to agree to disagree. RPKI is useful in context of a defense in depth strategy. If one combines "peerlock" AS_PATH filters with origin validation the end result is bullet proof. Even if NTT is the only one to deploy this combination, the benefits are notable. Kind regards, Job
On Sep 18, 2018, at 2:34 PM, Job Snijders <job@ntt.net> wrote:
On Tue, Sep 18, 2018 at 12:04:19PM -0700, Owen DeLong wrote:
Perhaps said another way:
"How would you figure out what prefixes your bgp peer(s) should be sending you?" (in an automatable, and verifiable manner)
In theory, that’s what IRRs are for.
You may be overlooking the fact that the semantics of an IRR route object are subtly different than those of a RPKI ROA. The Prefix-to-AS Mapping Database concept as introduced Section 2 of RFC 6811 is a huge step forward compared to the (somewhat loosely defined) semantics of IRR route objects.
RPKI ROAs are more than "IRR with crypto": IRR objects are basically a list of unverifiable attestations with no proof of termination. Whereas
Right… Hence my call for IRRs with stronger authentication and validation. (i.e. IRRs run by RIRs that have the same level of maintenance authentication and authorization requirements as current RPKI implementations).
when that same information is published through the RPKI, we now know two things: (1) the owner of the resource consented to the creation of the object and (2) any BGP UPDATE that conflicts with covering RPKI ROAs is invalid.
Yes, but, what you don’t know is whether any BGP UPDATE that contains the valid origin ASN as origin came from the origin ASN it claims. Instead, you provided a cryptographically signed recipe for believable spoofing. Actually, they’re quite a bit less… They provide no path data. ROAs are useful for one hop level validation. At the second AS hop they are 100% useless.
In other words, RPKI and the Prefix-to-AS validation procedure give us much more definitive inputs for routing policies compared to what can be derived from the IRR.
Please explain to me how you distinguish good from bad in the following scenario… You peer with AS6939. You receive routes for 2001:db8:f300::/48 with the following AS Paths 1. 6939 1239 54049 2312 1734 2. 6939 44046 12049 174 1734 Which one is valid? Which one is not? How did the ROA tell you this? With the IRR, I can (theoretically) compare an AS Path to the valid set of path associations documented in RPSL. (modulo lack of participation/implementation, which RPKI likewise suffers from). With RPKI, I can’t tell anything.
I simply view RPKI as a successor to the IRR system with some much needed improvements.
Your view is, IMHO, very distorted.
In practice, while they offer better theoretical capabilities if stronger authentication were added, the current implementation and acceptance leaves much to be desired.
Luckily multiple projects are passing through the pipeline to reduce the risk some IRRs represented.
https://www.ripe.net/manage-ips-and-asns/db/impact-analysis-for-nwi-5-implem... https://teamarin.net/2018/07/12/the-path-forward/
Considering that RPKI and IRR will co-exist in one form or another for a wihle it is encouraging to see success in patching loopholes.
Yep… Properly implemented and adopted, IRRs show some progress for actual path validation abilities, including origin.
However, even in theory, RPKI offers nothing of particular benefit even in its best case of widespread implementation.
I can't really wrap my head around how you can arrive at such a conclusion in light of all the information that has been provided to you. So perhaps we'll have to agree to disagree.
See above. What is it you believe RPKI offers beyond 1 hop that is anything more than a cryptographically signed recipe for believable spoofing?
RPKI is useful in context of a defense in depth strategy. If one combines "peerlock" AS_PATH filters with origin validation the end result is bullet proof. Even if NTT is the only one to deploy this combination, the benefits are notable.
Indeed, if peerlock gets wider deployment, it might prove useful. But unless I’m really misunderstanding peerlock, I don’t see that RPKI brings much else to the table in addition. Owen
On Tue, Sep 18, 2018 at 02:44:30PM -0700, Owen DeLong wrote:
ROAs are useful for one hop level validation. At the second AS hop they are 100% useless.
This conversation cannot be had without acknowledging there are multiple layers of defense in securing BGP. We should also acknowledge that the majority of Internet traffic passes over AS_PATHs that are only one hop. Networks that exchange significant amounts of traffic, tend to peer directly with each other.
In other words, RPKI and the Prefix-to-AS validation procedure give us much more definitive inputs for routing policies compared to what can be derived from the IRR.
Please explain to me how you distinguish good from bad in the following scenario… You peer with AS6939. You receive routes for 2001:db8:f300::/48 with the following AS Paths
1. 6939 1239 54049 2312 1734 2. 6939 44046 12049 174 1734
Which one is valid? Which one is not? How did the ROA tell you this?
Both path 1 and 2 are invalid, because of peerlock we'd never accept 1239 behind 6939, or 174 behind 6939. AS_PATH filtering combined with Origin Validation is where the magic is.
RPKI is useful in context of a defense in depth strategy. If one combines "peerlock" AS_PATH filters with origin validation the end result is bullet proof. Even if NTT is the only one to deploy this combination, the benefits are notable.
Indeed, if peerlock gets wider deployment, it might prove useful. But unless I’m really misunderstanding peerlock, I don’t see that RPKI brings much else to the table in addition.
Wide deployment is not relevant, this is a unilateral defense mechanism, so I fear there may indeed be a degree of misunderstanding. Kind regards, Job
On Sep 18, 2018, at 15:07 , Job Snijders <job@ntt.net> wrote:
On Tue, Sep 18, 2018 at 02:44:30PM -0700, Owen DeLong wrote:
ROAs are useful for one hop level validation. At the second AS hop they are 100% useless.
This conversation cannot be had without acknowledging there are multiple layers of defense in securing BGP. We should also acknowledge that the majority of Internet traffic passes over AS_PATHs that are only one hop. Networks that exchange significant amounts of traffic, tend to peer directly with each other.
Actually, I don’t buy that at all. Without going into too much detail, I know of at least one former employer who is quite well peered, distributes a great deal of traffic and sends a tremendous amount of it via multiple ASNs.
In other words, RPKI and the Prefix-to-AS validation procedure give us much more definitive inputs for routing policies compared to what can be derived from the IRR.
Please explain to me how you distinguish good from bad in the following scenario… You peer with AS6939. You receive routes for 2001:db8:f300::/48 with the following AS Paths
1. 6939 1239 54049 2312 1734 2. 6939 44046 12049 174 1734
Which one is valid? Which one is not? How did the ROA tell you this?
Both path 1 and 2 are invalid, because of peerlock we'd never accept 1239 behind 6939, or 174 behind 6939. AS_PATH filtering combined with Origin Validation is where the magic is.
OK, poor examples crafted at random. Point is there are plenty of valid AS Paths out there that you can’t actually validate.
RPKI is useful in context of a defense in depth strategy. If one combines "peerlock" AS_PATH filters with origin validation the end result is bullet proof. Even if NTT is the only one to deploy this combination, the benefits are notable.
Indeed, if peerlock gets wider deployment, it might prove useful. But unless I’m really misunderstanding peerlock, I don’t see that RPKI brings much else to the table in addition.
Wide deployment is not relevant, this is a unilateral defense mechanism, so I fear there may indeed be a degree of misunderstanding.
Point being that there are very very few ASNs using peer lock. Peer lock alone AIUI pretty well covers the issue. RPKI provides very little (if any) verification. Owen
On Tue, Sep 18, 2018 at 6:22 PM Owen DeLong <owen@delong.com> wrote:
On Sep 18, 2018, at 15:07 , Job Snijders <job@ntt.net> wrote:
On Tue, Sep 18, 2018 at 02:44:30PM -0700, Owen DeLong wrote:
ROAs are useful for one hop level validation. At the second AS hop they are 100% useless.
This conversation cannot be had without acknowledging there are multiple layers of defense in securing BGP. We should also acknowledge that the majority of Internet traffic passes over AS_PATHs that are only one hop. Networks that exchange significant amounts of traffic, tend to peer directly with each other.
Actually, I don’t buy that at all.
Without going into too much detail, I know of at least one former employer who is quite well peered, distributes a great deal of traffic and sends a tremendous amount of it via multiple ASNs.
an IP resource holder can sign multiple ROA for a single prefix, it's perfectly valid to have: 157.130.0.0/16 AS112 157.130.0.0/16 AS113 157.130.0.0/16 AS701 So 'peered well via multiple ASN isn't really a problem here'. Are you making an argument of the form: "1% of the problems are not solved so this solution can't possibly work" ? Using ROA from the RPKI system tied through/to the IRR data and applied as filters on the bgp sessions of a network should provide that ASN with more assurance that they will not accept and propagate a route hijack or mistake. Adding in validation is nice as well, and may allow you to be a little less diligent about route filtering... I think more than 1 protection is a good plan though (OV + filters seems sane to me, especially in a world where you can automate that whole lot)
In other words, RPKI and the Prefix-to-AS validation procedure give us much more definitive inputs for routing policies compared to what can be derived from the IRR.
Please explain to me how you distinguish good from bad in the following scenario… You peer with AS6939. You receive routes for 2001:db8:f300::/48 with the following AS Paths
1. 6939 1239 54049 2312 1734 2. 6939 44046 12049 174 1734
Which one is valid? Which one is not? How did the ROA tell you this?
Both path 1 and 2 are invalid, because of peerlock we'd never accept 1239 behind 6939, or 174 behind 6939. AS_PATH filtering combined with Origin Validation is where the magic is.
OK, poor examples crafted at random. Point is there are plenty of valid AS Paths out there that you can’t actually validate.
I think Job's point is that there really are note that many... perhaps that's an NTT particular view? I know that my production network's view is similar in 'shorter as paths'. I expect that in large-isp-land it's more prevalent than not.
RPKI is useful in context of a defense in depth strategy. If one combines "peerlock" AS_PATH filters with origin validation the end result is bullet proof. Even if NTT is the only one to deploy this combination, the benefits are notable.
Indeed, if peerlock gets wider deployment, it might prove useful. But unless I’m really misunderstanding peerlock, I don’t see that RPKI brings much else to the table in addition.
Wide deployment is not relevant, this is a unilateral defense mechanism, so I fear there may indeed be a degree of misunderstanding.
Point being that there are very very few ASNs using peer lock. Peer lock alone AIUI pretty well covers the issue. RPKI provides very little (if any) verification.
I suppose if you are happy just doing peer lock on/for your network and customers then cool! The RPKI system isn't being forced on you. It seems like a helpful addition to me, but that may not be your outlook. -chris
On Sep 18, 2018, at 21:29 , Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Sep 18, 2018 at 6:22 PM Owen DeLong <owen@delong.com <mailto:owen@delong.com>> wrote:
On Sep 18, 2018, at 15:07 , Job Snijders <job@ntt.net <mailto:job@ntt.net>> wrote:
On Tue, Sep 18, 2018 at 02:44:30PM -0700, Owen DeLong wrote:
ROAs are useful for one hop level validation. At the second AS hop they are 100% useless.
This conversation cannot be had without acknowledging there are multiple layers of defense in securing BGP. We should also acknowledge that the majority of Internet traffic passes over AS_PATHs that are only one hop. Networks that exchange significant amounts of traffic, tend to peer directly with each other.
Actually, I don’t buy that at all.
Without going into too much detail, I know of at least one former employer who is quite well peered, distributes a great deal of traffic and sends a tremendous amount of it via multiple ASNs.
an IP resource holder can sign multiple ROA for a single prefix, it's perfectly valid to have: 157.130.0.0/16 <http://157.130.0.0/16> AS112 157.130.0.0/16 <http://157.130.0.0/16> AS113 157.130.0.0/16 <http://157.130.0.0/16> AS701
I did not mean that they originate the traffic from multiple ASNs, I mean that a tremendous amount of their traffic goes origin->ASHOP1->ASHOP2->…->END USER.
So 'peered well via multiple ASN isn't really a problem here'. Are you making an argument of the form: "1% of the problems are not solved so this solution can't possibly work” ?
Except you misunderstood my meaning. Perhaps my fault, but hopefully the above clarification helps you see my point.
Using ROA from the RPKI system tied through/to the IRR data and applied as filters on the bgp sessions of a network should provide that ASN with more assurance that they will not accept and propagate a route hijack or mistake. Adding in validation is nice as well, and may allow you to be a little less diligent about route filtering... I think more than 1 protection is a good plan though (OV + filters seems sane to me, especially in a world where you can automate that whole lot)
Again, unless you can trust the data in the IRR to build a complete list of valid AS Paths from the ORIGIN, you can’t safely reject a fake route that has the correct prepend. The ability to do that strikes me as questionable at best in the vast majority of cases.
In other words, RPKI and the Prefix-to-AS validation procedure give us much more definitive inputs for routing policies compared to what can be derived from the IRR.
Please explain to me how you distinguish good from bad in the following scenario… You peer with AS6939. You receive routes for 2001:db8:f300::/48 with the following AS Paths
1. 6939 1239 54049 2312 1734 2. 6939 44046 12049 174 1734
Which one is valid? Which one is not? How did the ROA tell you this?
Both path 1 and 2 are invalid, because of peerlock we'd never accept 1239 behind 6939, or 174 behind 6939. AS_PATH filtering combined with Origin Validation is where the magic is.
OK, poor examples crafted at random. Point is there are plenty of valid AS Paths out there that you can’t actually validate.
I think Job's point is that there really are note that many... perhaps that's an NTT particular view? I know that my production network's view is similar in 'shorter as paths'. I expect that in large-isp-land it's more prevalent than not.
Presuming s/note/not/ I suspect there might be some validity to that viewpoint for supposed “Tier 1” networks. Once you hit Tier 2 and consider their subscribers in the mix, it gets a lot fuzzier and less likely that it is a valid perspective. For the average eyeball network, I bet it’s a complete non-starter.
RPKI is useful in context of a defense in depth strategy. If one combines "peerlock" AS_PATH filters with origin validation the end result is bullet proof. Even if NTT is the only one to deploy this combination, the benefits are notable.
Indeed, if peerlock gets wider deployment, it might prove useful. But unless I’m really misunderstanding peerlock, I don’t see that RPKI brings much else to the table in addition.
Wide deployment is not relevant, this is a unilateral defense mechanism, so I fear there may indeed be a degree of misunderstanding.
Point being that there are very very few ASNs using peer lock. Peer lock alone AIUI pretty well covers the issue. RPKI provides very little (if any) verification.
I suppose if you are happy just doing peer lock on/for your network and customers then cool! The RPKI system isn't being forced on you. It seems like a helpful addition to me, but that may not be your outlook.
Actually, from my perspective, neither one is practical/useful due to the lack of supporting data to achieve it. My perspective, however, is closer to the eyeball these days, so YMMV at the core. Owen
On Sep 19, 2018, at 8:55 PM, Owen DeLong <owen@delong.com> wrote:
Actually, from my perspective, neither one is practical/useful due to the lack of supporting data to achieve it.
I suggest you look at some of the cool research that was done with various prefixes from different regions. You can see the problem with ARIN prefixes fairly easily and how they’re harder to secure as a result. This seems to be broken by design on the part of ARIN based on my limited experiences interacting with the community folk. https://nlnog.net/static/nlnogday2018/8_Measuring_RPKI_ben_NLNOG_2018.pdf And the video here: https://www.youtube.com/watch?v=uDIQDpGObdc It’s super interesting to see which RIR prefixes perform better when it comes to the same security technology. - Jared
tor. 20. sep. 2018 02.56 skrev Owen DeLong <owen@delong.com>:
Again, unless you can trust the data in the IRR to build a complete list of valid AS Paths from the ORIGIN, you can’t safely reject a fake route that has the correct prepend.
Or you can have each hob validate. For example if HE.net did RPKI validation, it would be effective due to their large number of peerings. If my network has HE.net as one of my uplinks, someone might fake the route via one of my other uplinks, but we would not pick that route due to longer AS path. Regards Baldur
There's a lot to sift through in this thread (most of all assertions lacking evidence), but this needs to be called out: On Tue, Sep 18, 2018 at 06:21:56PM -0700, Owen DeLong wrote: [snip]
Point being that there are very very few ASNs using peer lock. Peer lock
Despite the cutesy neologism, filtering against the acceptance of big/important/private peers's ASNs from unplanned vectors is very common and was a standard part of the belt and suspenders toolkit long, long ago.* When I drove 6079, I distinctly recall it coming up in conversations with representatives from 2828 (it might have been the concentric days) and others in the hallways at NANOG. Cheers, Joe * Barring those who never cared about forwarding quality or path integrity and would say "LOL someone gives me free transit to you". -- Posted from my personal account - see X-Disclaimer header. Joe Provo / Gweep / Earthling
On Sep 18, 2018, at 10:35 AM, Job Snijders <job@ntt.net> wrote:
Owen,
On Tue, Sep 18, 2018 at 10:23:42AM -0700, Owen DeLong wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole.
1/ You may be overlooking the fact that many networks peer directly with what (for them) are the important sources/destinations. The semantics of RPKI ROAs help block illegitimate more-specifics, and the short AS_PATH between players prevents a hijacker from inserting themself. In other words - the most important AS_PATHs are 1 hop. The Internet's dense interconnectedness is saving its bacon.
While this may be true for a handful of well peered ASNs, it’s certainly not common around the wider internet.
2/ Another approach to achieve path validation for 1 hop is through mechanisms such what NTT calls 'peerlock'. https://www.youtube.com/watch?v=CSLpWBrHy10 <https://www.youtube.com/watch?v=CSLpWBrHy10>
Single hop is relatively easy. It’s 2+ hop where things get far more interesting. It’s convenient to reduce the problem set to the one you can easily solve, but ignoring the rest of the problem set smacks of hand-waving and “insert magic here”.
3/ Lastly, some folks are innovating in this space to help automate concepts such as peerlock through what is called ASPA. ASPA is intended as an out-of-band, deployable alternative to BGPSec.
OK, but IIRC, it’s rather orthogonal to RPKI.
https://tools.ietf.org/html/draft-azimov-sidrops-aspa-profile https://tools.ietf.org/html/draft-azimov-sidrops-aspa-verification
I think you underestimate how valuable RPKI based Origin Validation (even just by itself) is in today's Internet landscape.
I think you overestimate it. Owen
Owen DeLong:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole.
regardless of how one might think about RPKI, there are ROAs out there that reduce the visibility/reachability of certain prefixes and the general assumption is that announced prefixes would like to be reachable even if the operator doesn't care about RPKI and ROAs from the past anymore, he most likely cares about reachability from a pure operational point of view. my email was not about: "How much does one like RPKI?" it is about whether it is acceptable that RIRs (and more specifically ARIN in this mailing list's context) notify affected parties of their prefixes that suffer from stale ROAs. Even if one dislikes RPKI entirely the opinion could still be "yes notifying those parties makes sense to restore reachability". -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
On Wed, Sep 19, 2018 at 12:51 AM nusenu <nusenu-lists@riseup.net> wrote:
Owen DeLong:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole.
regardless of how one might think about RPKI, there are ROAs out there that reduce the visibility/reachability of certain prefixes and the general assumption is that announced prefixes would like to be reachable even if the operator doesn't care about RPKI and ROAs from the past anymore, he most likely cares about reachability from a pure operational point of view.
So, a lot like dnssec ... if you enable the RPKI functions (publish roas) I think it's very much a responsibility of the publisher to provide the correct information in an on-going and stable manner. This seems bad, at first blush, but you will not always be here to offer these recalcitrant folk a pointer to how to fix themselves, and TODAY there's: "little" penalty when it comes to getting this RPKI thing wrongly... So, ideally the folk who are 'doin it wrong' can learn, get operational proceses/procedures/personnel in place and take action for the long term... right? :)
my email was not about: "How much does one like RPKI?"
sorry, 'most' emails that mention RPKI are: "how much do you like the flavor of rpki?" :)
it is about whether it is acceptable that RIRs (and more specifically ARIN in this mailing list's context) notify affected parties of their prefixes that suffer from stale ROAs.
This I still think is a bad plan.. mostly because I don't think it'll help :( I think what helps is: "Oh, I cant get to <foo> and <bar> and <most of the internet>" .... I think folk that CARE will do the right thing, folk that 'think they care' won't and will soon get disconnected from the tubez. I apologize a tad if my view that: "breaking people will force them to fix themselves" is .... rough :( Even if one dislikes RPKI entirely the opinion could still be "yes
notifying those parties makes sense to restore reachability".
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
On Wed, Sep 19, 2018 at 01:07:42AM -0700, Christopher Morrow wrote:
it is about whether it is acceptable that RIRs (and more specifically ARIN in this mailing list's context) notify affected parties of their prefixes that suffer from stale ROAs.
This I still think is a bad plan.. mostly because I don't think it'll help :( I think what helps is: "Oh, I cant get to <foo> and <bar> and <most of the internet>" .... I think folk that CARE will do the right thing, folk that 'think they care' won't and will soon get disconnected from the tubez.
I apologize a tad if my view that: "breaking people will force them to fix themselves" is .... rough :(
What about an one-off outreach effort? We need to somehow kickstart the feedback loop, especially if we expect effects to become forceful. I'm hoping that if the invalid count is low enough it'll become more attractive for more people flip the switch and deploy OV. Kind regards, Job
On Wed, Sep 19, 2018 at 1:19 AM Job Snijders <job@instituut.net> wrote:
On Wed, Sep 19, 2018 at 01:07:42AM -0700, Christopher Morrow wrote:
it is about whether it is acceptable that RIRs (and more specifically ARIN in this mailing list's context) notify affected parties of their prefixes that suffer from stale ROAs.
This I still think is a bad plan.. mostly because I don't think it'll help :( I think what helps is: "Oh, I cant get to <foo> and <bar> and <most of the internet>" .... I think folk that CARE will do the right thing, folk that 'think they care' won't and will soon get disconnected from the tubez.
I apologize a tad if my view that: "breaking people will force them to fix themselves" is .... rough :(
What about an one-off outreach effort?
first I'm certainly happy about any progress on the 'RPKI DONE RIGHT' direction, and specifically Job you as a person have made some awesome progress here getting IXP/ISP folk to move to OV and RPKI deployments, and adding RPKI/ROA data into the NTT IRR. but.. I'm skeptical of distinct efforts like this. I think something like (I think these folk still offer this service: "bgp monitoring") BgpMon's monitoring service is what we should aim for: "A service that RPKI users have signed up for" Else: "ends up in spam folder" :(
We need to somehow kickstart the feedback loop, especially if we expect effects to become forceful. I'm hoping that if the invalid count is low enough it'll become more attractive for more people flip the switch and deploy OV.
Sure.... but: "can not access a majority of the internet?" seems like a good signal to the affected folks.
Kind regards,
Job
What about an one-off outreach effort?
Makes sense to me. As someone who (at least pretends to) care, I was very much unaware of RPKI before seeing discussion about it on NANOG and #ix. That said, having recently done this with ARIN... they've got a long way to go before it's a simple process (like RIPE). Submitting numerous tickets over a 3 day period doesn't strike me as particularly efficient. If outreach was done and widely taken up, I'd think ARIN's help desk will struggle to meet the demand. If this is the case and it's a multi-week process to get RPKI set up, it would be expected that people will give up part way through the process.
On Wed, Sep 19, 2018 at 1:33 AM Phil Lavin <phil.lavin@cloudcall.com> wrote:
What about an one-off outreach effort?
Makes sense to me. As someone who (at least pretends to) care, I was very much unaware of RPKI before seeing discussion about it on NANOG and #ix.
That said, having recently done this with ARIN... they've got a long way to go before it's a simple process (like RIPE). Submitting numerous tickets over a 3 day period doesn't strike me as particularly efficient. If outreach was done and widely taken up, I'd think ARIN's help desk will struggle to meet the demand. If this is the case and it's a multi-week process to get RPKI set up, it would be expected that people will give up part way through the process.
Phil. Thanks, this is interesting input.. I expected that the system arin setup was on-par with that which ripe/apnic have setup... huh, I'm surprised that it required any tickets at all to accomplish :(
On 19 Sep 2018, at 10:37, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Sep 19, 2018 at 1:33 AM Phil Lavin <phil.lavin@cloudcall.com> wrote:
What about an one-off outreach effort?
Makes sense to me. As someone who (at least pretends to) care, I was very much unaware of RPKI before seeing discussion about it on NANOG and #ix.
That said, having recently done this with ARIN... they've got a long way to go before it's a simple process (like RIPE). Submitting numerous tickets over a 3 day period doesn't strike me as particularly efficient. If outreach was done and widely taken up, I'd think ARIN's help desk will struggle to meet the demand. If this is the case and it's a multi-week process to get RPKI set up, it would be expected that people will give up part way through the process.
Phil. Thanks, this is interesting input.. I expected that the system arin setup was on-par with that which ripe/apnic have setup... huh, I'm surprised that it required any tickets at all to accomplish :(
ARIN offers all of the features that the other RIRs do, but usability remains a (big) barrier. I did a talk at NANOG several years ago demonstrating how usability of the hosted RPKI system greatly impacted adoption and data quality in the RIPE region: https://youtu.be/R2VV_APOFL8 At the time, a lot of effort went into providing a hosted RPKI system that suggested ROAs based on best practices, showed what the impact on BGP announcements was going to be and sent alerts when misconfigurations or hijacks occurred. This gives operators the confidence to use and maintain the system. As a result, the data set is now big and high quality enough for operators to start dropping invalids. I’d be interested to hear how many operators in the ARIN region would be willing to set up ROAs (and maintain them!) if it weren’t so hard to do. This might entice ARIN to address the usability issue. Because non-repudiation or not, this process shouldn’t have to take several tickets and several days. Be that as it may, we fully intend to build a Delegated CA that is on par with RIPE’s user experience so that operators can run RPKI themselves in a usable way. Alex Band NLnet Labs
Phil Lavin:
That said, having recently done this with ARIN... they've got a long way to go before it's a simple process (like RIPE). Submitting numerous tickets over a 3 day period doesn't strike me as particularly efficient.
If outreach was done and widely taken up,
I just want to reiterate that this is not about an outreach program "Create ROAs for all your prefixes NOW", this is just about notifying those ~30 orgs that have likely misconfigured ROAs that result in unreachable prefixes in an ROV environment.
I'd think ARIN's help desk will struggle to meet the demand. If this is the case and it's a multi-week process to get RPKI set up, it would be expected that people will give up part way through the process. That is a great input, we certainly would not want to cause a bottleneck at ARIN's helpdesk.
To avoid overwhelming the help desk, these ~30 organizations could be notified one at a time (i.e. notify 5 organizations per week and be done in a ~month), I assume that is a manageable amount of tickets per day. -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Looks like a certain CDN has volunteered to do that for you. Owen
On Sep 19, 2018, at 01:19 , Job Snijders <job@instituut.net> wrote:
On Wed, Sep 19, 2018 at 01:07:42AM -0700, Christopher Morrow wrote:
it is about whether it is acceptable that RIRs (and more specifically ARIN in this mailing list's context) notify affected parties of their prefixes that suffer from stale ROAs.
This I still think is a bad plan.. mostly because I don't think it'll help :( I think what helps is: "Oh, I cant get to <foo> and <bar> and <most of the internet>" .... I think folk that CARE will do the right thing, folk that 'think they care' won't and will soon get disconnected from the tubez.
I apologize a tad if my view that: "breaking people will force them to fix themselves" is .... rough :(
What about an one-off outreach effort?
We need to somehow kickstart the feedback loop, especially if we expect effects to become forceful. I'm hoping that if the invalid count is low enough it'll become more attractive for more people flip the switch and deploy OV.
Kind regards,
Job
Christopher Morrow wrote:
This seems bad, at first blush, but you will not always be here to offer these recalcitrant folk a pointer to how to fix themselves
that is correct but I don't expect that (to be around forever) to be necessary, once the amount of invalids are low, big operators could deploy ROV, and once that is the case operators will get an immediate effect should they create incorrect ROAs, which will cause a learning effect. At that point the amount of misconfigured ROAs would automatically remain low because ROV somewhat forces proper ROAs.
it is about whether it is acceptable that RIRs (and more specifically ARIN in this mailing list's context) notify affected parties of their prefixes that suffer from stale ROAs.
This I still think is a bad plan.. mostly because I don't think it'll help :(
If such an attempt to make people aware about their broken ROAs has no effect at all but I did no harm, than I'm fine with it because we at least tried. I'm not sure I can follow the "lets not send these 31 emails because it is such a big effort and they will just end up in the spam folder with no effect." line of reasoning. Do you think we would be doing more harm than good by sending out these 31 emails?
I think what helps is: "Oh, I cant get to <foo> and <bar> and <most of the internet>" .... I think folk that CARE will do the right thing, folk that 'think they care' won't and will soon get disconnected from the tubez.
I apologize a tad if my view that: "breaking people will force them to fix themselves" is .... rough :(
I believe it would be more polite to tell them first before you force anything on them by enabling ROV, but your way of doing it would certainly be more efficient ;) -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
On Sep 19, 2018, at 00:46 , nusenu <nusenu-lists@riseup.net> wrote:
Owen DeLong:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole.
regardless of how one might think about RPKI, there are ROAs out there that reduce the visibility/reachability of certain prefixes and the general assumption is that announced prefixes would like to be reachable even if the operator doesn't care about RPKI and ROAs from the past anymore, he most likely cares about reachability from a pure operational point of view.
Yep… And the easy recipe for one which doesn’t care about RPKI to restore reachability is “delete the ROAs”.
my email was not about: "How much does one like RPKI?”
I have no impression that it was. I thought it was about “Should we consume more RIR resources dealing with this additional pain likely to be caused by RPKI?”
it is about whether it is acceptable that RIRs (and more specifically ARIN in this mailing list's context) notify affected parties of their prefixes that suffer from stale ROAs.
I agree with Mr. Morrow that this would end in pain.
Even if one dislikes RPKI entirely the opinion could still be "yes notifying those parties makes sense to restore reachability”.
Agreed. However, whether I liked RPKI or not, I’d still say that notification by the RIRs is prone to sadness. My initial intent was merely to state that I prefer the RIRs not waste additional resources on this, including notification. Owen
On 18 Sep 2018, at 1:23 PM, Owen DeLong <owen@delong.com> wrote:
Personally, since all RPKI accomplishes is providing a cryptographically signed notation of origin ASNs that hijackers should prepend to their announcements in order to create an aura of credibility, I think we should stop throwing resources down this rathole.
Owen - Other than use of resources towards an initiative that some operators value (and others do not), is there any reason that notifying RPKI users about invalid prefixes is not a wise idea; i.e. are you aware of any technical downside to doing so? /John John Curran President and CEO ARIN
(popping back to the top of the thread.. sorry) On Tue, Sep 18, 2018 at 7:58 AM nusenu <nusenu-lists@riseup.net> wrote:
Dear NANOG,
when I approached ARIN about how they feel about reaching out to their members about prefixes that are unreachable in a route origin validation (ROV) environment, John Curran (CEO ARIN) referred me to you (see email bellow - quoted with permission).
Perhaps this was answered elsewhere, but: "Why is this something ARIN (the org) should take on?" Why can't (or why isn't) this something that 'many' monitoring/alerting companies/orgs are offering? it's unclear, to me, why ARIN is in any better position than any other party to perform this sort of activity? I would expect that, at the base level, "I just got random/unexpected email from ARIN?" will get dropped in the spam-can, while: "My monitoring company to which I signed up/contracted emailed into my ticket-system for action.. better go do something!" is the path to incentivize. The question I asked ARIN was specifically:
Would you be open to reach out to your affected members to inform them about their affected IP prefixes?
'how?' (email to the tech-contact? etc? did they sign up for said monitoring and point to the right destination email catcher?)
Christopher Morrow wrote:
Perhaps this was answered elsewhere, but: "Why is this something ARIN (the org) should take on?"
Thanks for this question, I believe this is an important one. I reasoned about why I think RIRs are in a good position to send these emails here: [1] but I will quote from it for convenience:
Notifying affected IP Holders
The natural next step (and that was our initial intention when looking at INVALIDs) would be to send out emails to affected IP holders and ask them to address the INVALIDs but although that could be automated, we believe the impact would be better, if that email came from some trusted entity like the RIR relevant to the affected IP holder instead of a random entity they never had any contact before (us).
Asking RIRs to reach out to their members also scales better since every RIR would only have to take care of their own members. [...]
[1] https://medium.com/@nusenu/towards-cleaning-up-rpki-invalids-d69b03ab8a8c
Why can't (or why isn't) this something that 'many' monitoring/alerting companies/orgs are offering?
There are companies offering BGP monitoring including RPKI ROAs, but the affected IP holders are unlikely customers of those monitoring services or generally aware of the problem.
it's unclear, to me, why ARIN is in any better position than any other party to perform this sort of activity? I would expect that, at the base level, "I just got random/unexpected email from ARIN?" will get dropped in the spam-can, while: "My monitoring company to which I signed up/contracted emailed into my ticket-system for action.. better go do something!" is the path to incentivize.
The problem is how do you make operators aware of the problem in the first place.
The question I asked ARIN was specifically:
Would you be open to reach out to your affected members to inform them about their affected IP prefixes?
'how?' (email to the tech-contact? etc? did they sign up for said monitoring and point to the right destination email catcher?)
Yes that is what I had in mind (notification via email to the tech contact). kind regards, nusenu -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
On Tue, Sep 18, 2018 at 1:33 PM nusenu <nusenu-lists@riseup.net> wrote:
Christopher Morrow wrote:
Perhaps this was answered elsewhere, but: "Why is this something ARIN (the org) should take on?"
Thanks for this question, I believe this is an important one.
I reasoned about why I think RIRs are in a good position to send these emails here: [1] but I will quote from it for convenience:
Notifying affected IP Holders
The natural next step (and that was our initial intention when looking at INVALIDs) would be to send out emails to affected IP holders and ask them to address the INVALIDs but although that could be automated, we believe the impact would be better, if that email came from some trusted entity like the RIR relevant to the affected IP holder instead of a random entity they never had any contact before (us).
Asking RIRs to reach out to their members also scales better since every RIR would only have to take care of their own members. [...]
i don't know that the contacts the RIR has for the entity is necessarily the one that controls/deals-with the RPKI data though. I also think that generally if folk set all that up they probably know (or will soon enough) that they have a mistake. Generally speaking, I think "folk should fix themselves, and maintain/monitor their configuration", that ARIN (or anyone else sending 'unsolicited email') here is going to end badly in the worst case and 'not have any effect' in the majority of cases.
[1] https://medium.com/@nusenu/towards-cleaning-up-rpki-invalids-d69b03ab8a8c
Why can't (or why isn't) this something that 'many' monitoring/alerting companies/orgs are offering?
There are companies offering BGP monitoring including RPKI ROAs, but the affected IP holders are unlikely customers of those monitoring services or generally aware of the problem.
ok, maybe they should though? :)
it's unclear, to me, why ARIN is in any better position than any other party to perform this sort of activity? I would expect that, at the base level, "I just got random/unexpected email from ARIN?" will get dropped in the spam-can, while: "My monitoring company to which I signed up/contracted emailed into my ticket-system for action.. better go do something!" is the path to incentivize.
The problem is how do you make operators aware of the problem in the first place.
ideally they are aware of thier own config, have a person(s) responsible for managing that configuration and care about reachability... if they don't have that today, they will soon enough.
The question I asked ARIN was specifically:
Would you be open to reach out to your affected members to inform them about their affected IP prefixes?
'how?' (email to the tech-contact? etc? did they sign up for said monitoring and point to the right destination email catcher?)
Yes that is what I had in mind (notification via email to the tech contact).
i'm positive that will end in sadness.
kind regards, nusenu
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
On Sep 18, 2018, at 2:15 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Sep 18, 2018 at 1:33 PM nusenu <nusenu-lists@riseup.net <mailto:nusenu-lists@riseup.net>> wrote: Christopher Morrow wrote:
Perhaps this was answered elsewhere, but: "Why is this something ARIN (the org) should take on?"
Thanks for this question, I believe this is an important one.
I reasoned about why I think RIRs are in a good position to send these emails here: [1] but I will quote from it for convenience:
Notifying affected IP Holders
The natural next step (and that was our initial intention when looking at INVALIDs) would be to send out emails to affected IP holders and ask them to address the INVALIDs but although that could be automated, we believe the impact would be better, if that email came from some trusted entity like the RIR relevant to the affected IP holder instead of a random entity they never had any contact before (us).
Asking RIRs to reach out to their members also scales better since every RIR would only have to take care of their own members. [...]
i don't know that the contacts the RIR has for the entity is necessarily the one that controls/deals-with the RPKI data though.
It sort of has to be, as managing your RPKI data (at least in the ARIN region) involves doing it through your ARIN On-Line account which must be associated with the ORG associated with the resources in question.
I also think that generally if folk set all that up they probably know (or will soon enough) that they have a mistake.
You overestimate some things here.
Generally speaking, I think "folk should fix themselves, and maintain/monitor their configuration", that ARIN (or anyone else sending 'unsolicited email') here is going to end badly in the worst case and 'not have any effect' in the majority of cases.
Agreed.
[1] https://medium.com/@nusenu/towards-cleaning-up-rpki-invalids-d69b03ab8a8c <https://medium.com/@nusenu/towards-cleaning-up-rpki-invalids-d69b03ab8a8c>
Why can't (or why isn't) this something that 'many' monitoring/alerting companies/orgs are offering?
There are companies offering BGP monitoring including RPKI ROAs, but the affected IP holders are unlikely customers of those monitoring services or generally aware of the problem.
ok, maybe they should though? :)
I love a good tautology.
it's unclear, to me, why ARIN is in any better position than any other party to perform this sort of activity? I would expect that, at the base level, "I just got random/unexpected email from ARIN?" will get dropped in the spam-can, while: "My monitoring company to which I signed up/contracted emailed into my ticket-system for action.. better go do something!" is the path to incentivize.
The problem is how do you make operators aware of the problem in the first place.
ideally they are aware of thier own config, have a person(s) responsible for managing that configuration and care about reachability... if they don't have that today, they will soon enough.
Optimist! Owen
On Tue, Sep 18, 2018 at 2:32 PM Owen DeLong <owen@delong.com> wrote:
On Sep 18, 2018, at 2:15 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Tue, Sep 18, 2018 at 1:33 PM nusenu <nusenu-lists@riseup.net> wrote:
Christopher Morrow wrote:
Perhaps this was answered elsewhere, but: "Why is this something ARIN (the org) should take on?"
Thanks for this question, I believe this is an important one.
I reasoned about why I think RIRs are in a good position to send these emails here: [1] but I will quote from it for convenience:
Notifying affected IP Holders
The natural next step (and that was our initial intention when looking at INVALIDs) would be to send out emails to affected IP holders and ask them to address the INVALIDs but although that could be automated, we believe the impact would be better, if that email came from some trusted entity like the RIR relevant to the affected IP holder instead of a random entity they never had any contact before (us).
Asking RIRs to reach out to their members also scales better since every RIR would only have to take care of their own members. [...]
i don't know that the contacts the RIR has for the entity is necessarily the one that controls/deals-with the RPKI data though.
It sort of has to be, as managing your RPKI data (at least in the ARIN region) involves doing it through your ARIN On-Line account which must be associated with the ORG associated with the resources in question.
I think I can manage my employer's RPKI data, and I'm not on the tech/admin/etc handles... I think I can also ask the person(s) who are to do it and they may have no idea what I'm asking beyond: "click these 5 things, type that thing, thanks!"
I also think that generally if folk set all that up they probably know
(or will soon enough) that they have a mistake.
You overestimate some things here.
probably, but ... eventually if your internet gets very small you'll look at why.
Generally speaking, I think "folk should fix themselves, and maintain/monitor their configuration", that ARIN (or anyone else sending 'unsolicited email') here is going to end badly in the worst case and 'not have any effect' in the majority of cases.
Agreed.
perhaps this is really my point: "I have no confidence that ARIN doing this (or anyone else except an upstream network/peer) is going to effect change"
[1] https://medium.com/@nusenu/towards-cleaning-up-rpki-invalids-d69b03ab8a8c
Why can't (or why isn't) this something that 'many' monitoring/alerting companies/orgs are offering?
There are companies offering BGP monitoring including RPKI ROAs, but the affected IP holders are unlikely customers of those monitoring services or generally aware of the problem.
ok, maybe they should though? :)
I love a good tautology.
I'm glad you enjoyed it... you found the easter-egg! Generally speaking though (and trying to be constructive) people who use BGP to manage reachability to their network resources really should monitor what their resources look like externally... and folk do offer services which do these things. "As seen on TV .... bgp monitoring for you!" :)
it's unclear, to me, why ARIN is in any better position than any other party to perform this sort of activity? I would expect that, at the base level, "I just got random/unexpected email from ARIN?" will get dropped in the spam-can, while: "My monitoring company to which I signed up/contracted emailed into my ticket-system for action.. better go do something!" is the path to incentivize.
The problem is how do you make operators aware of the problem in the first place.
ideally they are aware of thier own config, have a person(s) responsible for managing that configuration and care about reachability... if they don't have that today, they will soon enough.
Optimist!
yea... it's probably as optimistic as your hope that irr data will get better? :) I think as more things depend on reachability the state will improve... or that's my hope, yes.
Owen
Christopher Morrow:
Yes that is what I had in mind (notification via email to the tech contact).
i'm positive that will end in sadness.
we can also send snail mail :) after all ~80 or so entities is a manageable amount of organizations to notify in the ARIN region. -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Christopher Morrow wrote:
Yes that is what I had in mind (notification via email to the tech contact).
i'm positive that will end in sadness.
we can also send snail mail :) after all ~80 or so entities is a manageable amount of organizations to notify in the ARIN region.
correction: in the ARIN region there are just about 30 organizations to contact (~80 was for APNIC) -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
On Tue, Sep 18, 2018 at 4:54 PM nusenu <nusenu-lists@riseup.net> wrote:
Christopher Morrow wrote:
Yes that is what I had in mind (notification via email to the tech contact).
i'm positive that will end in sadness.
we can also send snail mail :) after all ~80 or so entities is a manageable amount of organizations to notify in the ARIN region.
correction: in the ARIN region there are just about 30 organizations to contact (~80 was for APNIC)
cool, sounds like you are all done in just ~20 USD of stamps.
nusenu wrote : What do you think about the idea that ARIN actively informs their affected members about prefixes that are unreachable in an RPKI ROV environment?
Support, although I doubt it would achieve the desired result. I support it for the following reason : when someone starts to block invalid prefixes, they would not have the excuse to say "we did not know about it". Michel. TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...
participants (12)
-
Alex Band -
Baldur Norddahl -
Christopher Morrow -
Jared Mauch -
Job Snijders -
Job Snijders -
Joe Provo -
John Curran -
Michel Py -
nusenu -
Owen DeLong -
Phil Lavin