Follow up to previous post regarding SAAVIS
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want? route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768 2905 701 3561 13768 1221 4637 3561 13768 3549 3561 13768 3277 3267 174 3561 13768 6539 3561 13768 16150 3549 3561 13768 701 3561 13768 3267 174 3561 13768 6453 3561 13768 3582 3701 3356 3561 13768 This is probably a fairly major problem... -Drew
On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver<drew.weaver@thenap.com> wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want?
sadly savvis didn't learn the pccw lesson, which is also the turk-telecom lesson which is also the as7007 lesson which is... fairly sad really in 2009. for the sake of $diety put a prefix-filter on your customer bgp sessions, it ain't hard! -chris
route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768 2905 701 3561 13768 1221 4637 3561 13768 3549 3561 13768 3277 3267 174 3561 13768 6539 3561 13768 16150 3549 3561 13768 701 3561 13768 3267 174 3561 13768 6453 3561 13768 3582 3701 3356 3561 13768
This is probably a fairly major problem...
-Drew
On Wed, 12 Aug 2009, Christopher Morrow wrote:
On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver<drew.weaver@thenap.com> wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want? sadly savvis didn't learn the pccw lesson, which is also the turk-telecom lesson which is also the as7007 lesson which is... fairly sad really in 2009. for the sake of $diety put a prefix-filter on your customer bgp sessions, it ain't hard!
sounds too much like "work" to me. not interested. -Dan
On Wed, Aug 12, 2009 at 2:20 PM, <goemon@anime.net> wrote:
On Wed, 12 Aug 2009, Christopher Morrow wrote:
On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver<drew.weaver@thenap.com> wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want?
sadly savvis didn't learn the pccw lesson, which is also the turk-telecom lesson which is also the as7007 lesson which is... fairly sad really in 2009. for the sake of $diety put a prefix-filter on your customer bgp sessions, it ain't hard!
sounds too much like "work" to me. not interested.
<http://www.taoofsummer.net/wp-content/uploads/2009/07/sadpanda.jpg>
On Wed, Aug 12, 2009 at 11:20:28AM -0700, goemon@anime.net wrote:
On Wed, 12 Aug 2009, Christopher Morrow wrote:
On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver<drew.weaver@thenap.com> wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want? sadly savvis didn't learn the pccw lesson, which is also the turk-telecom lesson which is also the as7007 lesson which is... fairly sad really in 2009. for the sake of $diety put a prefix-filter on your customer bgp sessions, it ain't hard!
sounds too much like "work" to me. not interested.
The irony is that MCI had (and C&W maintained for quite some time) a functional, highly automated IRR-based customer filtering system [props to the Cary team]. Somewhere along the M&A highway, the work to maintain it was substituted with the work to dismantle it. Sad to see crap emitted with 3561 in the path. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Aug 12, 2009, at 4:37 PM, Joe Provo wrote:
On Wed, Aug 12, 2009 at 11:20:28AM -0700, goemon@anime.net wrote:
On Wed, 12 Aug 2009, Christopher Morrow wrote:
On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver<drew.weaver@thenap.com> wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want? sadly savvis didn't learn the pccw lesson, which is also the turk-telecom lesson which is also the as7007 lesson which is... fairly sad really in 2009. for the sake of $diety put a prefix-filter on your customer bgp sessions, it ain't hard!
sounds too much like "work" to me. not interested.
The irony is that MCI had (and C&W maintained for quite some time) a functional, highly automated IRR-based customer filtering system [props to the Cary team]. Somewhere along the M&A highway, the work to maintain it was substituted with the work to dismantle it. Sad to see crap emitted with 3561 in the path.
I've come to the conclusion that if someone put a nice web2.0+ interface on creating and managing these objects it would be a lot easier. If there were a customer portal where you could visit to say "update my prefix-list/acl to include the following new prefix(es), and push the change /now/" I presume that would drive customer utilization of these services and allow people to manage things "better". There are lots of leaks all the time, as can be evidenced by the leak detection stuff I set up here: http://puck.nether.net/bgp/leakinfo.cgi - Jared
Jared Mauch wrote:
I've come to the conclusion that if someone put a nice web2.0+ interface on creating and managing these objects it would be a lot easier.
I've looked into IRR several times, usually after events like PCCW. Each time the amount of work to 1) figure out how to implement IRR and 2) actually implementing it far outweighed the benefit of doing it for my network. I would love to implement it and looking towards the future I will someday have to. Until it becomes much easier to do so, I don't foresee smaller SPs like myself allocating the necessary resources to an IRR project until we're forced to because of our individual growth or an idiot PCCW-type event that generates lots of bad PR.
There are lots of leaks all the time, as can be evidenced by the leak detection stuff I set up here:
Crossing fingers hoping I'm not in that list... Justin
Date: Wed, 12 Aug 2009 16:12:39 -0500 From: Justin Shore <justin@justinshore.com>
Jared Mauch wrote:
I've come to the conclusion that if someone put a nice web2.0+ interface on creating and managing these objects it would be a lot easier.
I've looked into IRR several times, usually after events like PCCW. Each time the amount of work to 1) figure out how to implement IRR and 2) actually implementing it far outweighed the benefit of doing it for my network. I would love to implement it and looking towards the future I will someday have to. Until it becomes much easier to do so, I don't foresee smaller SPs like myself allocating the necessary resources to an IRR project until we're forced to because of our individual growth or an idiot PCCW-type event that generates lots of bad PR.
While a web 2.0 app would be very nice, it's really not that hard to do now. You do need the IRRToolSet or something similar. the IRRToolSet has languished for a long time and was getting harder and harder to keep running, but the move to sourceforge and the massive number of updates and clean ups should soon result in a 5.0 release that builds cleanly on most Unix/Linux platforms with a modern C++ compiler. Once that happens, it's really pretty simple to use the IRR for this sort of thing and, while the IRR data quality is pretty poor, it's a vast improvement over crossing your fingers and sticking your head in the sand. Except for the ugliness of the Perl I wrote well over a decade ago (when I was just learning Perl), I'd try to talk the powers that be into allowing me to release it as an example, but the relevant code is really, really trivial. (Not that government would let me without vast quantities of paperwork.) -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On Wed, Aug 12, 2009 at 02:30:38PM -0700, Kevin Oberman wrote: [snip]
While a web 2.0 app would be very nice, it's really not that hard to do now. You do need the IRRToolSet or something similar. the IRRToolSet has languished for a long time and was getting harder and harder to keep running, but the move to sourceforge and the massive number of updates and clean ups should soon result in a 5.0 release that builds cleanly on
[insert plug for the revitalization going on at irrtoolset@lists.isc.org] The internal piece (ras's code, etc) is the easy part. The customer- facing piece isn't particularly difficult either. Getting it past the inexplicably-powerful branding people in marketing and having a management team with the iron will to dictate that the pointy-clicky form is not just the standard vector but the ONLY vector for non-IRR clued users. That was the support the cary team had at old 3561. Most ISPs don't have that level of management clue & willpower, as the same "but they will go to $competator who doesn't require it!" which has screwed up everything from domain registration to responsible BGP announcements fouls the customer interface as well. Account reps wanting an exception 'just this once' are the norm. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Wed, Aug 12, 2009 at 05:41:03PM -0400, Joe Provo wrote:
Most ISPs don't have that level of management clue & willpower, as the same "but they will go to $competator who doesn't require it!" which has screwed up everything from domain registration to responsible BGP announcements fouls the customer interface as well. Account reps wanting an exception 'just this once' are the norm.
I would make the opposite argument, my business would NEVER go to any network which didn't support IRR (and a bunch of other simple but important things, like a full set of non-secret BGP communities). It's amazing the number of networks that excludes in this day and age. And not even because "omg IRR is good because someone told me so and we should support it", but because I've seen FAR too much grief caused by humans typoing prefix-lists, or taking days to process them. It is the height of absurdity that this would ever be considered an acceptable solution to the problem. But most of all I'm amazed that we as network operators have managed to take such a simple concept as a protocol for storing and recursively retrieving a list of prefixes in a database and turned it into such a sloppy mess. We really shot ourselves in the foot with the complexity of RPSL, which tries to be everything to everyone rather than actually provide a simple effective way to maintain prefix-lists. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Perhaps this is a stupid question, but does each SP need to run their own physical RR? Isn't this something that could be hosted? Frank -----Original Message----- From: Richard A Steenbergen [mailto:ras@e-gerbil.net] Sent: Wednesday, August 12, 2009 5:55 PM To: Joe Provo Cc: nanog@nanog.org Subject: Re: Follow up to previous post regarding SAAVIS On Wed, Aug 12, 2009 at 05:41:03PM -0400, Joe Provo wrote:
Most ISPs don't have that level of management clue & willpower, as the same "but they will go to $competator who doesn't require it!" which has screwed up everything from domain registration to responsible BGP announcements fouls the customer interface as well. Account reps wanting an exception 'just this once' are the norm.
I would make the opposite argument, my business would NEVER go to any network which didn't support IRR (and a bunch of other simple but important things, like a full set of non-secret BGP communities). It's amazing the number of networks that excludes in this day and age. And not even because "omg IRR is good because someone told me so and we should support it", but because I've seen FAR too much grief caused by humans typoing prefix-lists, or taking days to process them. It is the height of absurdity that this would ever be considered an acceptable solution to the problem. But most of all I'm amazed that we as network operators have managed to take such a simple concept as a protocol for storing and recursively retrieving a list of prefixes in a database and turned it into such a sloppy mess. We really shot ourselves in the foot with the complexity of RPSL, which tries to be everything to everyone rather than actually provide a simple effective way to maintain prefix-lists. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Wed, Aug 12, 2009 at 07:37:00PM -0500, Frank Bulk wrote:
Perhaps this is a stupid question, but does each SP need to run their own physical RR? Isn't this something that could be hosted?
The data itself is stored on a distributed network of databases, so there is technically no reason any SP needs to run their own. However, they often do, because when a customer can't figure something out it gives them access to go in and tweak the customers' records for them. Unfortunately the distributed nature of the databases is one of the biggest problems with the IRR system. Anyone can run an irrd, there is no inherient authentication of the data. In order to get your irrd "recognized" all you have to do is get mirrored by a database that other people query and boom you're in the system. What tends to happen is someone puts a route into a database and then completely forgets about it, so there are a huge number of completely bogus routes out there which are never going to get cleaned up. The other problem is that when a SP has a customer who "can't figure it out", a typical course of action is to just "register the route for them" rather than try to explain it to them. Unfortunately, the same thing as above happens here, you end up with a big pile of people who register a big pile of routes in a big pile of random databases, often times completely unnecessarily, and they'll never be removed either. The biggest problem with the entire system is the way that data gets into it, and the lack of incentive for people to ever remove data from it. But as a mechanism to allow the routes which need to be allowed, and mostly prevent accidental leaks, it works. For more information, take a look at: http://www.nanog.org/meetings/nanog44/presentations/Tuesday/RAS_irrdata_N44.... -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Wed, Aug 12, 2009 at 9:16 PM, Richard A Steenbergen<ras@e-gerbil.net> wrote:
On Wed, Aug 12, 2009 at 07:37:00PM -0500, Frank Bulk wrote:
Perhaps this is a stupid question, but does each SP need to run their own physical RR? Isn't this something that could be hosted?
The data itself is stored on a distributed network of databases, so there is technically no reason any SP needs to run their own. However, they often do, because when a customer can't figure something out it gives them access to go in and tweak the customers' records for them.
Another reason one might choose to run their own is you never want to answer a customer with: "Well, we query the remote systems when we need to build prefix lists and apparently the remote system blacklisted us because we queried too many times this hour/minute/day/week.. sorry 'bout that!" It gives you control over the data format, availability, freshness (sorta) and 'security' of the data you'd update your network with. Not everyone has that set of requirements, but if you ask L3 or other folks that do IRR based 'auto filtering' I bet they have an answer along these lines.
Unfortunately the distributed nature of the databases is one of the biggest problems with the IRR system. Anyone can run an irrd, there is no inherient authentication of the data. In order to get your irrd "recognized" all you have to do is get mirrored by a database that other people query and boom you're in the system. What tends to happen is someone puts a route into a database and then completely forgets about it, so there are a huge number of completely bogus routes out there which are never going to get cleaned up.
drum, drum, drum.. rpki! (and hopefully having that tied back to a bill you pay... see sidr@ietf.org or wherever that list emanates from these days)
The other problem is that when a SP has a customer who "can't figure it out", a typical course of action is to just "register the route for them" rather than try to explain it to them. Unfortunately, the same thing as above happens here, you end up with a big pile of people who register a big pile of routes in a big pile of random databases, often times completely unnecessarily, and they'll never be removed either.
The biggest problem with the entire system is the way that data gets into it, and the lack of incentive for people to ever remove data from it. But as a mechanism to allow the routes which need to be allowed, and mostly prevent accidental leaks, it works.
The above 2 paragraphs I think encapsulate what happened to ConEd in ~2005? (or someone-or-other-edison in newyork) Old/stale data which cropped up in an unusual manner :(
For more information, take a look at:
http://www.nanog.org/meetings/nanog44/presentations/Tuesday/RAS_irrdata_N44....
btw, +1 on irrtoolset.. good stuff. -Chris
On Wed, Aug 12, 2009 at 08:16:38PM -0500, Richard A Steenbergen wrote: [snip]
Unfortunately the distributed nature of the databases is one of the biggest problems with the IRR system. Anyone can run an irrd, there is
You misspelled "largest strength". FOlks get to choose which registries to believe in what order, not required to have a single [politicized] entity. If folks mistakenly believe there is a 1:1 correspendence between IRR data and BGP tables, they will lose. The IRR data is more of a "flight plan", a set of what-is-possible per the originator of the data. [snip]
people query and boom you're in the system. What tends to happen is someone puts a route into a database and then completely forgets about it, so there are a huge number of completely bogus routes out there which are never going to get cleaned up.
Lots of folks set up systems for provisioning without deprovisioning. This is not an IRR problem, but a sloppy-human problem. Folks that get stuck with provisioning generally aren't incented to remove billable resources. CF good processes and management with backbone. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Wed, Aug 12, 2009 at 10:06:49PM -0400, Joe Provo wrote:
On Wed, Aug 12, 2009 at 08:16:38PM -0500, Richard A Steenbergen wrote: [snip]
Unfortunately the distributed nature of the databases is one of the biggest problems with the IRR system. Anyone can run an irrd, there is
You misspelled "largest strength". FOlks get to choose which registries to believe in what order, not required to have a single [politicized] entity.
Well, actually, no. I'm not aware of any mechanism under which you can effectively choose who to believe and in what order, nor do I think that it would make any real difference in the long run even if you could. IRR database mirroring is like being a tier 1, you have to peer with every other database out there in order to obtain a full view. That means there are two ways you can get access to all the data you need, you can either query someone else who maintains all those mirrors, or you can run your own irr db and do all the mirroring yourself. RADB is the de facto "main db" in the IRR system, it not only originates the vast majority of the routes but it maintains the most comprehensive mirroring of every other active IRR db. RADB currently tracks a total of 32 databases: http://www.radb.net/mirrorlist.html At the end of the day the results are the same, whether the data is in your local irrd or someone else's db (like RADB). You query them via the same mechanism, which has extremely limited source selection control. Basically the only thing you have control over are the list of IRR databases which are searched, but the results which are returned are a superset of all databases which you selected to search. You don't get to say "only listen to results from LEVEL3's db for this object unless they don't have results there, in which case you listen to SAVVIS" or anything like that. You could query the complete data for every individual route yourself, but this would be a massively difficult undertaking compared to the normal query operation. The normal query operation is by no stretch of the imagination easy either, querying a large ISP can take hours or more depending on the transaction latency between yourself and the server you're querying. In fact this is one of the reasons why querying data from RIPE is such a pain, their query language lacks a recursive service side expansion mechanism so the transaction latency turns querying a large AS-SET into a multi-hour or day long operation. Their whois daemon also has an obnoxious "feature" of forcefully closing the socket after 3 minutes, even if it's in the middle of returning an answer. This is the only real advantage of running your own local irrd, reducing the transaction latency, but it's still a lot of work to maintain the mirror, verify that all the data from all the sources is always importing correctly, etc. And even after you do all this, what does being able to pick a data source order buy you anyways? Do you think you win something by preferring say RADB over LEVEL3 over SAVVIS over ARIN over RIPE over...? You have no real idea where your customer's are keeping their records, or their customers, etc. Where do you draw the line on who's data you look at, and why do we need yet another system where people are left to make a judgement call over who's data they should trust? Personally I'm of the belief that every ISP running their own IRR db is a very bad idea, which is why I have chosen not to run one myself. To quote Vijay, it doesn't scale. The last thing the already very broken IRR system needs is more crap data by more random people spread out over more databases, and the majority of the current db's probably need to be shut down too. There is no reason that this process needs to be politicized, or cost anyone any money to use. Again, we've made a horrible system here. One reasonable solution is to have the server side run the complete query off its local database, and pass the complete results for a prefix-list back to the querier in a single transaction. This is how filtergen.level3.com works, though I personally find their system is be excessively slow. In IRRPT 2.0 development I'm writing a similar type of remote filter generator, which I hope will be useful to some people.
If folks mistakenly believe there is a 1:1 correspendence between IRR data and BGP tables, they will lose. The IRR data is more of a "flight plan", a set of what-is-possible per the originator of the data.
[snip]
people query and boom you're in the system. What tends to happen is someone puts a route into a database and then completely forgets about it, so there are a huge number of completely bogus routes out there which are never going to get cleaned up.
Lots of folks set up systems for provisioning without deprovisioning. This is not an IRR problem, but a sloppy-human problem. Folks that get stuck with provisioning generally aren't incented to remove billable resources. CF good processes and management with backbone.
There is plenty of motivation to add data to IRR to make your announcements work, but no motivation at all to remove data when it is no longer needed. Nobody sees a problem with this until you step back and realize that a lot of networks have IRR records so sloppy that they list nearly every route on the Internet. Why bother filtering at all then? I think if it was as simple as seeing a list of your routes (or customers in your as-set, etc) and having a checkbox to delete old data, people would be more reasonable about maintaining it. RPSL is scary and confusing to a lot of people (and it should probably be scary to everyone at any rate :P), there is no reason it needs to be like this. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On 13/08/2009 04:03, Richard A Steenbergen wrote:
In fact this is one of the reasons why querying data from RIPE is such a pain, their query language lacks a recursive service side expansion mechanism so the transaction latency turns querying a large AS-SET into a multi-hour or day long operation.
I've brought this to RIPE's attention before, but the suggestion was met with a sharp inhalation of breath and a discussion about exactly how difficult that might be. The ripe whoisd code-base is too complicated. [Server side UTF8 support would be nice too, but for entirely different reasons - apparently there are some people in the world who need character sets outside ascii7. Who knew?]
Do you think you win something by preferring say RADB over LEVEL3 over SAVVIS over ARIN over RIPE over...? You have no real idea where your customer's are keeping their records, or their customers, etc. Where do you draw the line on who's data you look at, and why do we need yet another system where people are left to make a judgement call over who's data they should trust?
Yes, this is a bit of a mess alright.
There is plenty of motivation to add data to IRR to make your announcements work, but no motivation at all to remove data when it is no longer needed. Nobody sees a problem with this until you step back and realize that a lot of networks have IRR records so sloppy that they list nearly every route on the Internet. Why bother filtering at all then?
Data with "source: RIPE" is quite good from this point of view, as the mnt-lower: and mnt-routes: fields are delegated by the RIR function in RIPE. There's no doubt that you can get all sorts of crap from non-RIR irrdbs, though.
I think if it was as simple as seeing a list of your routes (or customers in your as-set, etc) and having a checkbox to delete old data, people would be more reasonable about maintaining it. RPSL is scary and confusing to a lot of people (and it should probably be scary to everyone at any rate :P), there is no reason it needs to be like this.
RPSL is scary both from an end-user and a developer point of view. From the developer point of view, if the requirement to support AS path regular expressions were removed, that would remove lots of scary code from irrtoolset. The grammar is also very rich, which is just another word for complicated. From the user point of view, as a means of maintaining full policy routing configuration information (which seemed to be its original goal), it fails quite badly: too complicated to be understood easily; to limited to fulfil its objective. Like lots of technologies which didn't take off as intended (x.500, multicast, etc), it's found a stable niche, although there is no doubt that its prefix filtering capability is very undervalued. Nick
On Wed, Aug 12, 2009 at 10:03:23PM -0500, Richard A Steenbergen wrote:
On Wed, Aug 12, 2009 at 10:06:49PM -0400, Joe Provo wrote:
On Wed, Aug 12, 2009 at 08:16:38PM -0500, Richard A Steenbergen wrote: [snip]
Unfortunately the distributed nature of the databases is one of the biggest problems with the IRR system. Anyone can run an irrd, there is
You misspelled "largest strength". FOlks get to choose which registries to believe in what order, not required to have a single [politicized] entity.
Well, actually, no. I'm not aware of any mechanism under which you can effectively choose who to believe and in what order, nor do I think that it would make any real difference in the long run even if you could.
Experience proves otherwise. L3's filtergen is a great counter-example, where the customer-specific import policy dictates sources to believe regardless of what other stuff is in their local mirror. It happily drops prefixes not matching, so it does "make a real difference" WRT customer filtering. I'm not familiar with DTAG's tools, but would be shocked if they were less featureful. For querying other databases, see IRRd's !s syntax, which specifies sources in order. Also see Net:IRR's $whois->sources() method. For tools based on these, I would presume it be up to your implementation or policy expression as to how you decide the handling on multiple matches. When mentioned, usually the first which matches is specified as 'the one', which is why search order matters. What other purpose does specifying a search order serve?
IRR database mirroring is like being a tier 1, you have to peer with every other database out there in order to obtain a full view. That
Funny, subject of the thread is filtering customers. I don't need to take the same point of view of the RADB ("RADB's mission is to mirror all component databases so as to provide the most complete view possible of the entire IRR.") to do that, just a set of databases in which my customers can be/must be registered. Once upon a time, 3561 had a highly automated and effective way of doing this based in part upon running their own DB and that being the default/requirement for "non-advanced" customers. [snip]
Basically the only thing you have control over are the list of IRR databases which are searched, but the results which are returned are a superset of all databases which you selected to search. You don't get to say "only listen to results from LEVEL3's db for this object unless they don't have results there, in which case you listen to SAVVIS" or anything like that.
If I am running a tool to generate filter lists for my customers, I want to believe my RR, the local RIR, some other RIR that is well run, and then maybe my upstream. Specify that search order and believe the first match. Job done. If you have highly clued downstreams, go the filtergen route and tune source-believability based on customer, or cook up another avenue. There is nothing inherent in the system to prevent this. [snip]
from all the sources is always importing correctly, etc. And even after you do all this, what does being able to pick a data source order buy you anyways? Do you think you win something by preferring say RADB over LEVEL3 over SAVVIS over ARIN over RIPE over...?
Yes, reduced queries and the ability to ignore Bob's Bait and Tackle DB if I know it is part of the "piles of junk databases" you posit will exist. See above.
Where do you draw the line on who's data you look at, and why do we need yet another system where people are left to make a judgement call over who's data they should trust?
I'm not sure who has a better viewpoint on what my customers can/should emit cross my network than me, so yeah I should make the call regarding what databases my customers need to be in for my business. Obviously for multihomed or well-meshed customers you have to approach that sanely or you'll get serious pushback from folks registered elsewhere. In the earlier 3561 days, I had to get them to eat non-MCI sources for us so I wouldn't be doubly & trebly registered. Assuming a perfect global datastore will fail. It doesn't exist now, and migrating there to such a mythical beast is impossible. Run your corner as cleanly as you can and apply as much error correction as possible to the outside world. Since this topic is about running one's corner cleanly, taking the input from known-garbage is a bad plan.
Personally I'm of the belief that every ISP running their own IRR db is a very bad idea, which is why I have chosen not to run one myself. To quote Vijay, it doesn't scale. The last thing the already very broken IRR system needs is more crap data by more random people spread out over more databases, and the majority of the current db's probably need to be shut down too.
Centralized scales better than distributed? Quick, call the 80s - we need HOSTS.TXT back. Of course it isn't applicable for every 10-prefix shop that happens to run BGP because they have multiple 0/0s, but that is merely due to the effort not being worth the return for those people, not anything to do with scaling of the IRR system. If small fries did run their own nodes, they would need to get their providers to slurp the data, or be an LIR or.... I don't see a massive clamor for small fries though. The existing open-door IRR policy has only grown slowly over the years, not exploded. Heck, some older ones [cf this thread's subject, BELL, et al] seem to no longer be used by their owners and just might not be worth querying. So yeah, I think a level of reasonable discrimination based on existing data quality encourages increased data quality.
There is no reason that this process needs to be politicized, or cost anyone any money to use.
Anytime you go down the road of advocating authority centralization, you'll start getting people politicizing the process [cf icann, alternate roots, all the random frothing-at-the-mouth-until-they-fall-over types]. I rather think that can be avoided by properly embracing the distributed databases that do indeed function. Some can be side-stepped with RIR- based IRRs, and decently distributed down to LIRs, but we all know ARIN is still playing catchup here so it doesn't help our sphere in the near term. Money? Assuming a registry-based or provider-customer based DBs then it would be part of the existing relationship, no? Fees were being used at RADB in part as an incentive to get folks to clean up dead records. In Oct of 2002 Larry Blunk reported that they trimmed from 3150 maintainers down to 1972, though I'm not finding any numbers on non-maintainer objects purged ... I'm sure some were just M&A detritus that moved from one maintainer to another.
Again, we've made a horrible system here.
I think you've misspelled 'front end'. The system certainly seems to function, and the entire intent was that SPs would build their own customer-facing tools as well as internal tools. Seems we've fallen down in that regard, but irrpt [even if in php :-P] and the revival of IRRtoolset are indications that folks are still interested in building the internal widgets. In general I think you'd agree that the 'back end' of most all service providers did not keep pace with the growth of commoditization of service.
One reasonable solution is to have the server side run the complete query off its local database, and pass the complete results for a prefix-list back to the querier in a single transaction. [snip]
Again, your complaint isn't against "the IRR", but regarding an implementation specific ... which aligns with what 3561 used to do. We are straying dangerously close back to the thread topic. [snip]
Lots of folks set up systems for provisioning without deprovisioning. This is not an IRR problem, but a sloppy-human problem. Folks that get stuck with provisioning generally aren't incented to remove billable resources. CF good processes and management with backbone.
There is plenty of motivation to add data to IRR to make your announcements work, but no motivation at all to remove data when it is no longer needed. Nobody sees a problem with this until you step back and realize that a lot of networks have IRR records so sloppy that they list nearly every route on the Internet.
See what I wrote above; that is a provisioning/deprovisioning problem, not an IRR problem, and you know that. I know plenty of ISPs that don't perceive their lousy half-hearted attempts at deprovisioning *any* part of service to be a big deal, since improvements there doesn't make them money. As long as physical ports and circuits go back into inventory to sell to someone else, they could care less what data is left dangling. Sad but true. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Thu, Aug 13, 2009 at 12:03:24PM -0400, Joe Provo wrote:
Experience proves otherwise. L3's filtergen is a great counter-example, where the customer-specific import policy dictates sources to believe regardless of what other stuff is in their local mirror. It happily drops prefixes not matching, so it does "make a real difference" WRT
No, level3's filtergen follows the exact "search path" rules I described previously, which has no real impact for any reasonably sized isp. For example, say I describe my as-set and aut-num and routes in altdb, you can have level3 restrict the scope of the initial query to ALTDB::myasset, and you can have level3 restrict the search path to altdb, but what happens when I have a customer in my as-set who registered their prefixes in radb? Now you have to open up the scope there, and ripe, and arin, and level3, and ntt, and savvis, etc. Now say someone comes along and slips an unauthorized route with my origin: aut-num into one of these databases. You have no way to prevent that from happening, when you run the query on my as-set/aut-num you're going to get back the superset of my legit routes + any bogus ones. And this is a good thing, because it's a lot less destructive to have bogus routes in the system than it is to give someone the ability to override legitimate routes with a bogus entry on a "more trusted" db.
customer filtering. I'm not familiar with DTAG's tools, but would be shocked if they were less featureful. For querying other databases, see IRRd's !s syntax, which specifies sources in order. Also see Net:IRR's $whois->sources() method. For tools based on these, I would presume it be up to your implementation or policy expression as to how you decide the handling on multiple matches. When mentioned, usually the first which matches is specified as 'the one', which is why search order matters. What other purpose does specifying a search order serve?
This is the server side search path I talked about, it has nothing to do with any specific client implementation nor is a client implementation practical. See page 34 of: http://www.irrd.net/irrd-user.pdf Again you can restrict a global query, but this provides very little practical benefit. You could dynamically restrict sources per query when you go to do the !i or !g expansion, but there is no information on what you should restrict it to, so again no practical benefit. The only thing Level3 adds that isn't part of the stock query syntax is the top level scope I mentioned above, ALTDB::AS-MYASSET. To support this recursively you would have to run multiple full queries for the full records without server side expansion, which is not practical for anyone with more than a few hundred routes.
If I am running a tool to generate filter lists for my customers, I want to believe my RR, the local RIR, some other RIR that is well run, and then maybe my upstream. Specify that search order and believe the first match. Job done. If you have highly clued downstreams, go the filtergen route and tune source-believability based on customer, or cook up another avenue. There is nothing inherent in the system to prevent this. ...
Yes, reduced queries and the ability to ignore Bob's Bait and Tackle DB if I know it is part of the "piles of junk databases" you posit will exist. See above.
This doesn't work if your customers have customers. I'm also not aware of anyone running any "bad" databases, or for that matter any databases which are of lesser security/quality than the "big boys". Short of what ripe implements because they are the RIR, there is no real security on registrations here, so it doesn't much matter if the database is level3 or bob's bait and tackle. And even given what I consider to be an excessively large list of irr databases today, from the standpoint of keeping good records, I'd be hard pressed to name one on the list who's data I should trust any less than say level3's.
Centralized scales better than distributed? Quick, call the 80s - we need HOSTS.TXT back.
A silly argument. In this case, hosts.txt is equivalent to an ISP having a human manually process an e-mail from a customer, add it to a prefix-list on a router, and then manually e-mail their upstream or peer ISPs to have them update the prefix-list, etc. In many cases centralized (or at least, restricted to some reasonably sized set, obviously nobody is proposing running the entire Internet on a single server run by a single entity) has much better security properties. As far as scale goes, you're talking about a pretty simple database of pretty simple objects here. There is probably more overhead that goes into maintaining the distributed nature of the db than there is actual work generating prefix-lists. :)
There is no reason that this process needs to be politicized, or cost anyone any money to use.
Anytime you go down the road of advocating authority centralization, you'll start getting people politicizing the process [cf icann, alternate roots, all the random frothing-at-the-mouth-until-they-fall-over types]. I rather think that can be avoided by properly embracing the distributed databases that do indeed function. Some can be side-stepped with RIR- based IRRs, and decently distributed down to LIRs, but we all know ARIN is still playing catchup here so it doesn't help our sphere in the near term.
A reasonable amount of authority centralization in this case is at the RIR level, particularly if it adds security mechanisms that provide some level of authorization over who is registering what prefixes. There is no reason that I should need to run my own database, if the system was designed properly.
Again, we've made a horrible system here.
I think you've misspelled 'front end'. The system certainly seems to function, and the entire intent was that SPs would build their own customer-facing tools as well as internal tools. Seems we've fallen down in that regard, but irrpt [even if in php :-P] and the revival of IRRtoolset are indications that folks are still interested in building the internal widgets. In general I think you'd agree that the 'back end' of most all service providers did not keep pace with the growth of commoditization of service.
The databases are full of garbage data, a large portion of the networks who do use it have as-set objects which expand to be completely worthless (either blocking all their customers, or allowing the entire internet), and there is a significant percentage of the bgp speaking Internet who can't figure it out at all (including a lot of otherwise theoretically competent tier 1's). Even of the people who use it, a LOT of it only works because of wide-spread and often unauthorized proxy registration, which IMHO is even more evil than not having it at all. I'm by no means advocating the hosts.txt approach, clearly we NEED a scalable automated system for managing authorized prefixes, but by every measurable standard I can come up with the end result is a festering pile of crap. I really don't think you can completely dismiss the back end (both implementation and design) as part of those problems. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
The other problem is that when a SP has a customer who "can't figure it out", a typical course of action is to just "register the route for them" rather than try to explain it to them. Unfortunately, the same thing as above happens here, you end up with a big pile of people who register a big pile of routes in a big pile of random databases, often times completely unnecessarily, and they'll never be removed either.
The biggest problem with the entire system is the way that data gets into it, and the lack of incentive for people to ever remove data from it. But as a mechanism to allow the routes which need to be allowed, and mostly prevent accidental leaks, it works.
Agreed. Wish most providers take the extra mile effort to advise and facilitate customers on the process. The redundant entries are annoying :-)
On Wed, 12 Aug 2009, Richard A Steenbergen wrote:
I would make the opposite argument, my business would NEVER go to any network which didn't support IRR (and a bunch of other simple but important things, like a full set of non-secret BGP communities). It's amazing the number of networks that excludes in this day and age. And not even because "omg IRR is good because someone told me so and we should support it", but because I've seen FAR too much grief caused by humans typoing prefix-lists, or taking days to process them. It is the height of absurdity that this would ever be considered an acceptable solution to the problem.
I'd be very hesitant to use an upstream that didn't use any filtering method. But, as convenient as I find the IRR system to be (from the customer perspective, at least), I'm quite happy that a couple of our upstreams use other mechanisms instead. I've had prefixes fall out of the IRR a couple of times, leading to outages of IRR-using transit providers. I've had transit providers screw up manually maintained prefix-lists, or had mis-communications resulting in the wrong thing getting removed. With multiple transit providers and multiple systems, they tend not to all have the same filtering problem at the same time. That's a very good thing. I hope the recommendation that comes out of this discussion is to filter somehow, rather than to use any particular filter-generation mechanism. Diversity and redundancy are good things, in processes as well as hardware. -Steve
On Wed, Aug 12, 2009 at 05:55:10PM -0500, Richard A Steenbergen wrote:
On Wed, Aug 12, 2009 at 05:41:03PM -0400, Joe Provo wrote:
Most ISPs don't have that level of management clue & willpower, as the same "but they will go to $competator who doesn't require it!" which has screwed up everything from domain registration to responsible BGP announcements fouls the customer interface as well. Account reps wanting an exception 'just this once' are the norm.
I would make the opposite argument, my business would NEVER go to any network which didn't support IRR (and a bunch of other simple but important things, like a full set of non-secret BGP communities).
You and I would agree, but there are far more edge ASNs than mine, and last I checked you weren't running an edge. Just as with any commoditization, the large number of buyers tends to win out. If the world we wished existied, 3561 wouldn't have lost their good provider-hosted-IRR based filters. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Wed, Aug 12, 2009 at 04:57:07PM -0400, Jared Mauch wrote:
I've come to the conclusion that if someone put a nice web2.0+ interface on creating and managing these objects it would be a lot easier.
Agreed, this is one of the projects I've been working on just haven't had the time to finish it. But please allow me to put in a shameless plug for IRR PowerTools, which many networks (including a couple tier 1s) use to do their IRR: http://sourceforge.net/projects/irrpt/ The highlights are: * Automated retrieval of prefixes registered behind an IRR Object. * Automatic exclusion of bogon or other configured undesirable routes. * Tracking and long-term recording of prefix changes over time. * Automatic aggregation to optimize data and reduce unnecessary changes. * E-mail updates, letting users know that their change was processed. * E-mail alerts to the ISP, letting them know of new routing changes. * E-mail alerts to non-IRR using networks, with plain text changes. * Router config generation, for easy automated config deployment. I'm also in the process of beta testing a new 2.0 version which will be significantly rewritten for easier more scalable use, have a lot fewer dependencies, integrate better with db backend systems to customer prefix-list management, and fully support IPv6. Oh and there might just be a web gui for managing and using it too, if I can find a decent web developer who will do it for free. :) I'm hoping to have this out in the next couple of months. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Richard A Steenbergen wrote:
On Wed, Aug 12, 2009 at 04:57:07PM -0400, Jared Mauch wrote:
I've come to the conclusion that if someone put a nice web2.0+ interface on creating and managing these objects it would be a lot easier.
Agreed, this is one of the projects I've been working on just haven't had the time to finish it. But please allow me to put in a shameless plug for IRR PowerTools, which many networks (including a couple tier 1s) use to do their IRR:
http://sourceforge.net/projects/irrpt/
The highlights are:
* Automated retrieval of prefixes registered behind an IRR Object. * Automatic exclusion of bogon or other configured undesirable routes. * Tracking and long-term recording of prefix changes over time. * Automatic aggregation to optimize data and reduce unnecessary changes. * E-mail updates, letting users know that their change was processed. * E-mail alerts to the ISP, letting them know of new routing changes. * E-mail alerts to non-IRR using networks, with plain text changes. * Router config generation, for easy automated config deployment.
I'm also in the process of beta testing a new 2.0 version which will be significantly rewritten for easier more scalable use, have a lot fewer dependencies, integrate better with db backend systems to customer prefix-list management, and fully support IPv6. Oh and there might just be a web gui for managing and using it too, if I can find a decent web developer who will do it for free. :) I'm hoping to have this out in the next couple of months.
The longer a network develops without using or maintaining IRR data, the more difficult the transition becomes. A truly awesome capability would be to have some way to slurp in current configuration and output IRR objects that can then generate a functionally identical configuration. Even without that, I would happily settle for any improvements to irr tools, powertools or toolsets. I am sort of disappointed that there seemed to be far less then deserved support from those with a stake in this, the registries and vendors. Joe
On Wed, 12 Aug 2009 16:57:07 -0400, Jared Mauch <jared@puck.nether.net> wrote:
I've come to the conclusion that if someone put a nice web2.0+ interface on creating and managing these objects it would be a lot easier.
If there were a customer portal where you could visit to say "update my prefix-list/acl to include the following new prefix(es), and push the change /now/" I presume that would drive customer utilization of these services and allow people to manage things "better".
That's fine... until you learn the hard way 9 times out of 10, the person heading to such a thing is a clueless moron. As much as it is a pain in the rear, having *people* proofing and editing the BGP configuration is far less work than creating the AI needed to keep idiots from doing idiotic things. I've never worked for any of the tier-1 beasts, but I have had to manage dozens of customer BGP sessions. Over a decade, I never dealt with a clueful customer... we cannot announce address space that doesn't belong to you; you cannot announce *our* address space to other ISPs; you have one f'ing link, why do you need BGP? (note: never actually ask that question or mute your phone immediately after asking.) How often do your prefixes change? In my experience adding new netblocks and/or customers, taking a few weeks to get things setup wasn't a problem; it'd take that long to get their connection turned up. (and if they were talking about BGP, sales would be talking to us before the contract was signed.) --Ricky
"the pccw lesson, which is also the turk-telecom lesson" tangent here: what was the pccw and turk-telecom thing? is the turk telco thing the Youtube fiasco? On Wed, Aug 12, 2009 at 10:09 AM, Christopher Morrow < morrowc.lists@gmail.com> wrote:
On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver<drew.weaver@thenap.com> wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want?
sadly savvis didn't learn the pccw lesson, which is also the turk-telecom lesson which is also the as7007 lesson which is... fairly sad really in 2009.
for the sake of $diety put a prefix-filter on your customer bgp sessions, it ain't hard!
-chris
route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768 2905 701 3561 13768 1221 4637 3561 13768 3549 3561 13768 3277 3267 174 3561 13768 6539 3561 13768 16150 3549 3561 13768 701 3561 13768 3267 174 3561 13768 6453 3561 13768 3582 3701 3356 3561 13768
This is probably a fairly major problem...
-Drew
-- Andrew Euell andyzweb [at] gmail [dot] com
On Thu, Aug 13, 2009 at 11:27 PM, Andrew Euell<andyzweb@gmail.com> wrote:
"the pccw lesson, which is also the turk-telecom lesson"
tangent here: what was the pccw and turk-telecom thing? is the turk telco thing the Youtube fiasco?
pccw + pktelecom == youtube incident turk-telecom leaked covad + a bunch of other things ... 4 years back at either the time of NANOG or a US Holiday (Christmas??) Both incidents were: "Providers who didn't filter their customer(s)" -chris
On Wed, Aug 12, 2009 at 10:09 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver<drew.weaver@thenap.com> wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes for whatever IP addresses they want?
sadly savvis didn't learn the pccw lesson, which is also the turk-telecom lesson which is also the as7007 lesson which is... fairly sad really in 2009.
for the sake of $diety put a prefix-filter on your customer bgp sessions, it ain't hard!
-chris
route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768 2905 701 3561 13768 1221 4637 3561 13768 3549 3561 13768 3277 3267 174 3561 13768 6539 3561 13768 16150 3549 3561 13768 701 3561 13768 3267 174 3561 13768 6453 3561 13768 3582 3701 3356 3561 13768
This is probably a fairly major problem...
-Drew
-- Andrew Euell andyzweb [at] gmail [dot] com
participants (15)
-
Andrew Euell -
Christopher Morrow -
Drew Weaver -
Frank Bulk -
goemon＠anime.net -
Jared Mauch -
Joe Maimon -
Joe Provo -
Justin Shore -
Kanagaraj -
Kevin Oberman -
Nick Hilliard -
Richard A Steenbergen -
Ricky Beam -
Steve Gibbard