RE: protocols that don't meet the need...
I am not going to speak for the IETF, but why would they? Their meetings are already open, and to be globally fair the proposed coordinators would have to attend 3-5 extra meetings a year to cover all the ops groups. Tony
-----Original Message----- From: Eastgard, Tom [mailto:tom.eastgard@boeing.com] Sent: Tuesday, February 14, 2006 1:01 PM To: Tony Hain; nanog@merit.edu Subject: RE: protocols that don't meet the need...
-----Original Message----- From: Tony Hain [mailto:alh-ietf@tndh.net] Sent: Tuesday, February 14, 2006 12:35 PM To: nanog@merit.edu Subject: protocols that don't meet the need...
A thought I had on the plane last night about the disconnect between the NANOG and IETF community which leaves protocol development to run open-loop.
Rather than sit back and complain about the results, why not try to synchronize meeting times. Not necessarily hotels, but within a reasonable distance of each other so the issue about ROI for the trip can be mitigated. This will mean that people who regularly attend both will have overlap issues, but if one meeting every year or two is joint there is an opportunity for those who can't justify the extra trips to at least have some feedback to try and close the loop on protocol design.
Would it make sense to ask IETF to provide a focal or coordinate(s?) to NANOG who would host a BOF(s?) on IETF issues --- not to debate, explain or work them but to board the issues and concerns of the operating community? Point being to provide a lightly structured and cost effective mechanism for operators to give feedback without having to attend three more meetings per year?
T. Eastgard
Tony/all,
I am not going to speak for the IETF, but why would they? Their meetings are already open, and to be globally fair the proposed coordinators would have to attend 3-5 extra meetings a year to cover all the ops groups.
I am also not speaking for the IETF (IAB), but the IAB has undertaken the task of trying to bring a little of what's happening in the IETF to the operator community (and hopefully in the process engaging folks to come to the IETF). Now, while many in the IETF argue that there is no such thing as an "operator community", I personally see it differently, and there are many of us who think that operator input is sorely missing from the IETF process. That is one of the reasons we did the NANOG 35 IPv6 multihoming BOF (and are doing the same at the upcoming apricot meeting). So (and again, not speaking for the IAB), my perspective is that we really need your insight and perspectives, more generally, your help in solving some of the difficult problems before us (a viable routing and addressing architecture for IPv6 comes to mind). All of that being said, I would be glad to facilitate with the IETF in any way I can. Perhaps someone from the NANOG PC/SC or Merit can contact me offline and we can look at with our options are. Any takers? Dave
Tony
-----Original Message----- From: Eastgard, Tom [mailto:tom.eastgard@boeing.com] Sent: Tuesday, February 14, 2006 1:01 PM To: Tony Hain; nanog@merit.edu Subject: RE: protocols that don't meet the need...
-----Original Message----- From: Tony Hain [mailto:alh-ietf@tndh.net] Sent: Tuesday, February 14, 2006 12:35 PM To: nanog@merit.edu Subject: protocols that don't meet the need...
A thought I had on the plane last night about the disconnect between the NANOG and IETF community which leaves protocol development to run open-loop.
Rather than sit back and complain about the results, why not try to synchronize meeting times. Not necessarily hotels, but within a reasonable distance of each other so the issue about ROI for the trip can be mitigated. This will mean that people who regularly attend both will have overlap issues, but if one meeting every year or two is joint there is an opportunity for those who can't justify the extra trips to at least have some feedback to try and close the loop on protocol design.
Would it make sense to ask IETF to provide a focal or coordinate(s?) to NANOG who would host a BOF(s?) on IETF issues --- not to debate, explain or work them but to board the issues and concerns of the operating community? Point being to provide a lightly structured and cost effective mechanism for operators to give feedback without having to attend three more meetings per year?
T. Eastgard
On Feb 14, 2006, at 4:47 PM, David Meyer wrote:
Tony/all,
I am not going to speak for the IETF, but why would they? Their meetings are already open, and to be globally fair the proposed coordinators would have to attend 3-5 extra meetings a year to cover all the ops groups.
I am also not speaking for the IETF (IAB), but the IAB has undertaken the task of trying to bring a little of what's happening in the IETF to the operator community (and hopefully in the process engaging folks to come to the IETF). Now, while many in the IETF argue that there is no such thing as an "operator community", I personally see it differently, and there are many of us who think that operator input is sorely missing from the IETF process. That is one of the reasons we did the NANOG 35 IPv6 multihoming BOF (and are doing the same at the upcoming apricot meeting).
Hmm, well, when there is lots of vendor and academia involvement, no, there's no operator community presented in number of things I'm following in the IETF. Take manet, for example, I don't even know to begin where to inject operator concerns/requirements. :-/ I think this is as much an IETF issue as it is of the operator community. Operators need to devote time to IETF to make the work in the IETF most relevant to the operators needs. Best regards, Christian
Christian
On Feb 14, 2006, at 4:47 PM, David Meyer wrote:
Tony/all,
I am not going to speak for the IETF, but why would they? Their meetings are already open, and to be globally fair the proposed coordinators would have to attend 3-5 extra meetings a year to cover all the ops groups.
I am also not speaking for the IETF (IAB), but the IAB has undertaken the task of trying to bring a little of what's happening in the IETF to the operator community (and hopefully in the process engaging folks to come to the IETF). Now, while many in the IETF argue that there is no such thing as an "operator community", I personally see it differently, and there are many of us who think that operator input is sorely missing from the IETF process. That is one of the reasons we did the NANOG 35 IPv6 multihoming BOF (and are doing the same at the upcoming apricot meeting).
Hmm, well, when there is lots of vendor and academia involvement, no, there's no operator community presented in number of things I'm following in the IETF. Take manet, for example, I don't even know to begin where to inject operator concerns/requirements. :-/
Well taken. And further, I would say manet is more the rule than the exception in this respect. BTW, it took me years to become facile with the (IETF) process (if I'm even there now :-)). I can say that I had excellent mentoring (Randy and perhaps a few others), so that helped. Maybe we need something not as formal as an IETF liaison relationship, but perhaps something like that. More thinking required...
I think this is as much an IETF issue as it is of the operator community. Operators need to devote time to IETF to make the work in the IETF most relevant to the operators needs.
Yes, and this has always been an acute problem as long as I've been around. People have day (night, weekend jobs). Co-location of the meetings seems a possible way to start attacking one aspect that problem, with the understanding that perhaps travel isn't the biggest of the problems, but it is a non-trivial issue for many of us. Thanks for the great comments. Dave
David, On Feb 14, 2006, at 5:07 PM, David Meyer wrote:
Hmm, well, when there is lots of vendor and academia involvement, no, there's no operator community presented in number of things I'm following in the IETF. Take manet, for example, I don't even know to begin where to inject operator concerns/requirements. :-/
Well taken. And further, I would say manet is more the rule than the exception in this respect. BTW, it took me years to become facile with the (IETF) process (if I'm even there now :-)). I can say that I had excellent mentoring (Randy and perhaps a few others), so that helped. Maybe we need something not as formal as an IETF liaison relationship, but perhaps something like that. More thinking required...
Thanks for the feedback. I've been following manet as an interested party for a while, with no real mission other than tracking it for emerging technologies R&D. Lately, job is architecting municipal wireless networks (which is really far more than what most people think of when they think Sbux style WISP hotspots). And I'm looking at the IETF for what's been worked out so far with respect to wireless routing protocols for example, and I just can't help sitting here scratching my head about how I would ever use what they've come up with so far. And right now, I really can't without major modifications it seems. And I find that really sad actually. And, don't get me wrong, but I'm not trying to bash them at all. I just think that real world operations needs and concepts like wholesale access aren't even anywhere near the radar screen it seems. And that somehow needs to be fixed. And, yes, municipal wireless is a roller coaster that's still gathering speed, so, expecting that everything's already grown and ready for us are thoroughly unrealistic. But! ;-) Right now the routing protocol on the mesh side will likely be proprietary for some time, which really isn't in the operator's best interest but that's what we have to work with. I/we have a substantial interest in this becoming more than an academic PhD thesis exercise, but something that can really be practically used in the real world. Now, there is stuff in the MPLS community, for example, that I've followed more or less closely for the past 7 yrs that might actually be fruitful, but it too requires substantial tailoring. So, no worries about job security there. ;-)
I think this is as much an IETF issue as it is of the operator community. Operators need to devote time to IETF to make the work in the IETF most relevant to the operators needs.
Yes, and this has always been an acute problem as long as I've been around. People have day (night, weekend jobs). Co-location of the meetings seems a possible way to start attacking one aspect that problem, with the understanding that perhaps travel isn't the biggest of the problems, but it is a non-trivial issue for many of us.
Agreed. I'm headed to the IEEE 802 plenary in a couple of weeks to start working standards body stuff for us as well, some of what needs to happen is lower layer stuff. The less trips and the more I can combine them, the more likely my management will look at my travel expense submissions in a favorable light ;-).. So, the more incentive we can provide with that, the better. A while back, there was a desire to colo ARIN with NANOG. That's really cool to see happen. For me, no offense to anyone, I really couldn't care less at the moment. I'm on the opposite side of the spectrum, ARIN being a vehicle for operationalized networks rather than those who are about to be operationalized. So, perhaps NANOG should be paired up with other industry forums in some kind of rotation.. Anyone got ideas on this? Best regards, Christian
On Tue, Feb 14, 2006 at 07:09:38PM -0500, Christian Kuhtz wrote:
David,
On Feb 14, 2006, at 5:07 PM, David Meyer wrote:
Hmm, well, when there is lots of vendor and academia involvement, no, there's no operator community presented in number of things I'm following in the IETF. Take manet, for example, I don't even know to begin where to inject operator concerns/requirements. :-/
Well taken. And further, I would say manet is more the rule than the exception in this respect. BTW, it took me years to become facile with the (IETF) process (if I'm even there now :-)). I can say that I had excellent mentoring (Randy and perhaps a few others), so that helped. Maybe we need something not as formal as an IETF liaison relationship, but perhaps something like that. More thinking required...
Thanks for the feedback.
Any time.
I've been following manet as an interested party for a while, with no real mission other than tracking it for emerging technologies R&D. Lately, job is architecting municipal wireless networks (which is really far more than what most people think of when they think Sbux style WISP hotspots). And I'm looking at the IETF for what's been worked out so far with respect to wireless routing protocols for example, and I just can't help sitting here scratching my head about how I would ever use what they've come up with so far. And right now, I really can't without major modifications it seems. And I find that really sad actually.
That is a perfect example of the reason why we need more "reality clue" (or whatever) from the ops community in the standards process (choose your SDO). IMO, of course; others will (strenuously) differ with me.
And, don't get me wrong, but I'm not trying to bash them at all. I just think that real world operations needs and concepts like wholesale access aren't even anywhere near the radar screen it seems. And that somehow needs to be fixed. And, yes, municipal wireless is a roller coaster that's still gathering speed, so, expecting that everything's already grown and ready for us are thoroughly unrealistic. But! ;-)
Sure, understood.
Right now the routing protocol on the mesh side will likely be proprietary for some time, which really isn't in the operator's best interest but that's what we have to work with. I/we have a substantial interest in this becoming more than an academic PhD thesis exercise, but something that can really be practically used in the real world.
Now, there is stuff in the MPLS community, for example, that I've followed more or less closely for the past 7 yrs that might actually be fruitful, but it too requires substantial tailoring. So, no worries about job security there. ;-)
I think this is as much an IETF issue as it is of the operator community. Operators need to devote time to IETF to make the work in the IETF most relevant to the operators needs.
Yes, and this has always been an acute problem as long as I've been around. People have day (night, weekend jobs). Co-location of the meetings seems a possible way to start attacking one aspect that problem, with the understanding that perhaps travel isn't the biggest of the problems, but it is a non-trivial issue for many of us.
Agreed. I'm headed to the IEEE 802 plenary in a couple of weeks to start working standards body stuff for us as well, some of what needs to happen is lower layer stuff. The less trips and the more I can combine them, the more likely my management will look at my travel expense submissions in a favorable light ;-).. So, the more incentive we can provide with that, the better.
I talked a bit with the (relatively new) IAD (IETF Administrative types) about what might be done here. This is something that will take some thought and if anything comes of that, some serious planning.
A while back, there was a desire to colo ARIN with NANOG. That's really cool to see happen. For me, no offense to anyone, I really couldn't care less at the moment. I'm on the opposite side of the spectrum, ARIN being a vehicle for operationalized networks rather than those who are about to be operationalized. So, perhaps NANOG should be paired up with other industry forums in some kind of rotation.. Anyone got ideas on this?
Re NANOG/ARIN: The times I've been involved in the joint meetings I've found them very useful. Dave
The big question there is whether it is helpful for an operator of a wired network to comment on a routing technology for a network that is fundamentally dissimilar from his target topology. Not that there is no valid comment - the security issues are certainly related. But if you want to say "but in my continental or global fiber network I don't plan to run a manet, so this is entirely stupid" - which is nearly verbatim the operator comment I got in a discussion of manet routing in a university setting three years ago - the logical answer is "we didn't expect you to; do you have comments appropriate to a regional enterprisish network whose 'core' is a set of unmanned airplanes flying in circles and connects cars, trucks, and other kinds of vehicles?". So operators are certainly welcome in a research group, but I would suggest that operator concerns/requirements be tailored to operational use of a manet network in a context where it *is* appropriate. On Feb 14, 2006, at 1:55 PM, Christian Kuhtz wrote:
Hmm, well, when there is lots of vendor and academia involvement, no, there's no operator community presented in number of things I'm following in the IETF. Take manet, for example, I don't even know to begin where to inject operator concerns/requirements. :-/
Fred, Hmm. Is self-organizing mesh access network with (some) explicitly mobile participants really that dissimilar from what the claimed goal of manet is? Seems to me that's perfectly in scope. Further, I think if you review the charter for the manet wg you could be convinced they're explicitly in scope. And, from EarthLink Municipal Networks perspective, we're hardly a 'wired network' operator a la incumbent telco, even though elements of those types of networks may help bring our wireless mesh to life in the end. So, if what we're doing isn't part of manet, what is the appropriate industry forum to work out IP routing issues etc? What is the appropriate context for manet if it isn't what I read the charter to state? Is it really just, for example, autonomous devices navigating in a sensor network? Best regards, Christian On Feb 15, 2006, at 4:35 PM, Fred Baker wrote:
The big question there is whether it is helpful for an operator of a wired network to comment on a routing technology for a network that is fundamentally dissimilar from his target topology. Not that there is no valid comment - the security issues are certainly related. But if you want to say "but in my continental or global fiber network I don't plan to run a manet, so this is entirely stupid" - which is nearly verbatim the operator comment I got in a discussion of manet routing in a university setting three years ago - the logical answer is "we didn't expect you to; do you have comments appropriate to a regional enterprisish network whose 'core' is a set of unmanned airplanes flying in circles and connects cars, trucks, and other kinds of vehicles?".
So operators are certainly welcome in a research group, but I would suggest that operator concerns/requirements be tailored to operational use of a manet network in a context where it *is* appropriate.
On Feb 14, 2006, at 1:55 PM, Christian Kuhtz wrote:
Hmm, well, when there is lots of vendor and academia involvement, no, there's no operator community presented in number of things I'm following in the IETF. Take manet, for example, I don't even know to begin where to inject operator concerns/requirements. :-/
then fine, I agree that a manet network run by an operator is in scope. I was responding to the comments I have already gotten from network operators who have dumped all over me when I mentioned manet. On Feb 15, 2006, at 1:52 PM, Christian Kuhtz wrote:
Fred,
Hmm. Is self-organizing mesh access network with (some) explicitly mobile participants really that dissimilar from what the claimed goal of manet is? Seems to me that's perfectly in scope.
Further, I think if you review the charter for the manet wg you could be convinced they're explicitly in scope. And, from EarthLink Municipal Networks perspective, we're hardly a 'wired network' operator a la incumbent telco, even though elements of those types of networks may help bring our wireless mesh to life in the end.
So, if what we're doing isn't part of manet, what is the appropriate industry forum to work out IP routing issues etc? What is the appropriate context for manet if it isn't what I read the charter to state? Is it really just, for example, autonomous devices navigating in a sensor network?
Best regards, Christian
On Feb 15, 2006, at 4:35 PM, Fred Baker wrote:
The big question there is whether it is helpful for an operator of a wired network to comment on a routing technology for a network that is fundamentally dissimilar from his target topology. Not that there is no valid comment - the security issues are certainly related. But if you want to say "but in my continental or global fiber network I don't plan to run a manet, so this is entirely stupid" - which is nearly verbatim the operator comment I got in a discussion of manet routing in a university setting three years ago - the logical answer is "we didn't expect you to; do you have comments appropriate to a regional enterprisish network whose 'core' is a set of unmanned airplanes flying in circles and connects cars, trucks, and other kinds of vehicles?".
So operators are certainly welcome in a research group, but I would suggest that operator concerns/requirements be tailored to operational use of a manet network in a context where it *is* appropriate.
On Feb 14, 2006, at 1:55 PM, Christian Kuhtz wrote:
Hmm, well, when there is lots of vendor and academia involvement, no, there's no operator community presented in number of things I'm following in the IETF. Take manet, for example, I don't even know to begin where to inject operator concerns/requirements. :-/
On Tue, Feb 14, 2006 at 01:47:31PM -0800, David Meyer wrote:
IETF). Now, while many in the IETF argue that there is no such thing as an "operator community", I personally see it differently, and there are many of us who think that operator input is sorely missing from the IETF process.
The problem with IETF and IPv6 is from my perspective, that operator input is being rejected as unreasonable or just ignored (shim6). I've stopped wasting time trying to bring operator's views/points across. It's not welcome if it doesn't fit already existing views within IETF regarding IPv6. I know that a lot of active IPv6 folks think the same but hesitate to communicate that openly.
That is one of the reasons we did the NANOG 35 IPv6 multihoming BOF (and are doing the same at the upcoming apricot meeting).
Which is a good thing. But still, many IETF folks deny the fact that they constantly hear that things like shim6 is NOT what the ops folks (the folks that have to actually work with the stuff IETF brings forward) are looking for. And we know that it doesn't. It can't. There is no way to do traffic engineering with any shim6-like system like one can do with BGP as shim6 is a completely host-centric solution. It has no clue about upstream/downstream/peering, ASses etc. Those things that actually make topology and economics. That's aside all the other administrative nightmares associated.
So (and again, not speaking for the IAB), my perspective is that we really need your insight and perspectives, more generally, your help in solving some of the difficult problems before us (a viable routing and addressing architecture for IPv6 comes to mind).
I firmly believe that this train for IPv6 is long gone. We should go forward with IPv6 using the legacy routing architecture and start NOW working on a complete real re-vamp with a PROPER locator/identifier split - not an ugly hack ontop of the traditional IPv4/IPv6 like shim6 which doesn't deliver what ops folks need. Nevertheless, I really welcome IAB's efforts in the matter. Best regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
On 15 feb 2006, at 11.51, Daniel Roesen wrote:
That is one of the reasons we did the NANOG 35 IPv6 multihoming BOF (and are doing the same at the upcoming apricot meeting).
Which is a good thing. But still, many IETF folks deny the fact that they constantly hear that things like shim6 is NOT what the ops folks (the folks that have to actually work with the stuff IETF brings forward) are looking for.
I think YOUR problem is that the chairs in an IETF WG gathers consensus of the members. Additionally I would like to point out that on shim6 specifically I also meets representatives of fairly large providers that see new opportunities with shim6 or id/loc split solutions. To Dave's point that there are people "in IETF" that don't think there is an operator community I would like to add that there certainly is an operator community but their views are not homogenous. And neither is any other groups. Which makes the job of any IETF WG chair to judge consensus among the representatives in a particular WG to do just that, gather consensus and move forward.
And we know that it doesn't. It can't. There is no way to do traffic engineering with any shim6-like system like one can do with BGP as shim6 is a completely host-centric solution. It has no clue about upstream/downstream/peering, ASses etc. Those things that actually make topology and economics. That's aside all the other administrative nightmares associated.
I would like to argue that TE in the general case is orthogonal to upstream/downstream/peering. I am not clear if you are trying to voice concern that you can not do TE, or that shim6 will not give you the ability to control how your customers does TE, or something else. - kurtis -
It's the lack of reality in operational policies that is the real source of frustration in ops communities. People are picking on shim6 because it is used as an argument to back the current policies at a time when it doesn't even have an early alpha-implementation to show for it. Policies built around shim6 may be ok in 5 or 10 years if or when it is mature with supporting technology to handle large networks, but not now. In the meantime we need a policy that can accomodate the need for multihoming of end-sites with *existing* technology. Without such a policy we will have anarchy with LIRs making their own policies (fragmentation) and people telling lies to qualify as a LIR to obtain independent blocks (unless there's a way to delay v6 deployment until there is technology available to back the current policy). //per -- Per Heldal http://heldal.eml.cc/
On 15 feb 2006, at 13.56, Per Heldal wrote:
It's the lack of reality in operational policies that is the real source of frustration in ops communities. People are picking on shim6 because it is used as an argument to back the current policies at a time when it doesn't even have an early alpha-implementation to show for it. Policies built around shim6 may be ok in 5 or 10 years if or when it is mature with supporting technology to handle large networks, but not now. In the meantime we need a policy that can accomodate the need for multihoming of end-sites with *existing* technology. Without such a policy we will have anarchy with LIRs making their own policies (fragmentation) and people telling lies to qualify as a LIR to obtain independent blocks (unless there's a way to delay v6 deployment until there is technology available to back the current policy).
I am certainly no fan of the current rule-set and I have been known to look favourable to a more relaxed ruleset as an intermediary step. Personally I think we should drop the 200 customer rule and give a prefix to all LIRs for the timebeeing. Actually the RIPE community once decided that as the policy... - kurtis -
On Wed, 15 Feb 2006, Daniel Roesen wrote:
There is no way to do traffic engineering with any shim6-like system like one can do with BGP as shim6 is a completely host-centric solution. It has no clue about upstream/downstream/peering, ASses etc. Those things that actually make topology and economics. That's aside all the other administrative nightmares associated.
The current routing model doesn't scale. I don't want to sit 5 years from now needing a router that'll handle 8 million routes to get me through the next 5 years of route growth. PI space for multihoming and AS number growth is a bad thing for scaling and economics, however you look at it. Shim6 would hopefully curb the prefix growth very early in the growth curve as single entities won't need AS to multihome between two different ISPs. I do believe in a completely new solution for multihoming than what we have today, shim6 is the first I've seen so far, I'm open to other suggestions though. Current way of doing it (AS+PI) gives me the creeps when I extrapolate into the future. -- Mikael Abrahamsson email: swmike@swm.pp.se
The current routing model doesn't scale. I don't want to sit 5 years from now needing a router that'll handle 8 million routes to get me through
the
next 5 years of route growth.
You might want to read a NANOG posting made by Sean Doran back in September 1995 http://www.cctec.com/maillists/nanog/historical/9509/msg00047.html People had scaling problems back then because routers were less powerful. They solved those scaling problems without rushing to the vendors, checkbook in hand. You might want to review some of the other discussion on NANOG during that month, September 1995, to see what people were saying about topological addressing.
PI space for multihoming and AS number growth is a bad thing for scaling
and economics, however you look at it.
I disagree. I think the Internet can scale the current routing model to 8 million routes using proxy aggregation, filtering and geo-topological allocation of IPv6 addresses to multihomers.
Shim6 would hopefully curb the prefix growth very early in the growth curve as single entities won't need AS to multihome between two different ISPs.
Who wants to use a technology that will not let them get to Yahoo and other major websites? Shim6 is dead. We either change the routing architecture entirely or we find better ways to aggregate and filter so that the global routing table only has routes that need to be visible everywhere to everyone. Talk to Owen Delong if you think we need to change the whole architecture. In any case, creative thinking is what is needed. Not scary linear extrapolations. --Michael Dillon
MA> Date: Wed, 15 Feb 2006 16:31:56 +0100 (CET) MA> From: Mikael Abrahamsson MA> The current routing model doesn't scale. I don't want to sit 5 years from MA> now needing a router that'll handle 8 million routes to get me through the MA> next 5 years of route growth. MA> MA> PI space for multihoming and AS number growth is a bad thing for scaling and MA> economics, however you look at it. I'm going to suggest something horribly radical (or nostalgic, depending how long one has been in the industry): inter-provider cooperation. Let's examine _why_ the routing table might become large. Lots of smaller players multihoming, yes? Say two million small businesses multihome using SBC and Cox. Must we have two million global ASNs and routes? Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ address space. Each provider announces the aggregate co-op space via the joint ASN as a downstream. This is very similar to a downstream using a private ASN to connect to one upstream in two different locations. i.e., transit provider uses the same ASN for all such customers, and certainly needn't pollute the global table with longer prefixes. We're dealing with _one_ routing policy: hand it to Cox, or hand it to SBC. Why explode it into two million "different" policies? Look at MPLS. It essentially hunts down congruent or similar routing policies, slaps a tag on the packet, and routes based on that. Why not explore options that get it right and coalesce from the get-go? Note also that this is totally op-community. No new protocols required. It can be done today without forklifts. I thought I proposed this at 35. Maybe that was one of the open mic sessions where time ran out... Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On Wed, 15 Feb 2006, Edward B. DREGER wrote:
Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ address space. Each provider announces the aggregate co-op space via the joint ASN as a downstream.
This is unworkable obviously: Think next about SBC and (say) Verizon customers, then what about those with Cox and Verizon, then SBC/Cox/Verizon. etc. regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: It is amazing how complete is the delusion that beauty is goodness.
PJ> Date: Wed, 15 Feb 2006 19:02:11 +0000 (GMT) PJ> From: Paul Jakma PJ> > Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ address PJ> > space. Each provider announces the aggregate co-op space via the joint PJ> > ASN as a downstream. PJ> PJ> This is unworkable obviously: Think next about SBC and (say) Verizon No, it is not unworkable. Think through it a bit more. Although the problem is theoretically O(N^2), in practice it is closer to O(N). Note that _routing itself_ is theoretically an O(N^2) problem. Do we say that it is "unworkable obviously"? No. PJ> customers, then what about those with Cox and Verizon, then SBC/Cox/Verizon. PJ> etc. Yes, one ASN is required per cooperating pair. Just how many pairs do you think there are? Now compare with the number of leaves that [would [like to]] dual-home. If you have 100 providers, each cooperating with every single one of the others, that's 100 * 99 / 2 = 4950 different ASes. Noticeable, but still a long way from 4-octet ASN territory. And guess what? Each downstream would need its own ASN otherwise; this is just one ASN per cooperating pair. How many transit ASes are there? And will each one share a downstream with all of the others? http://www.caida.org/analysis/routing/ I'll hazard a guess that a transit cooperates with, on average, no more than five different other transits. Ergo, linear scaling. The biggest problem is when customer's link to provider A goes down and inbound traffic must flow through provider B. This necessitates some sort of path between A and B where more-specifics can flow. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Once upon a time, Edward B. DREGER <eddy+public+spam@noc.everquick.net> said:
No, it is not unworkable. Think through it a bit more. Although the problem is theoretically O(N^2), in practice it is closer to O(N). Note that _routing itself_ is theoretically an O(N^2) problem. Do we say that it is "unworkable obviously"? No.
There's a difference: computers (routers) handle the O(N^2) routing problem, while people would have to handle the O(N^2) cooperative AS problem.
Yes, one ASN is required per cooperating pair. Just how many pairs do you think there are? Now compare with the number of leaves that [would [like to]] dual-home.
We are a relatively small ISP with just a handful of multihoming customers. However, no two of them have the same other provider. What is gained by us setting up relationships with a bunch of other providers and getting special ASes assigned? What if one of those customers gets a connection to a third upstream, or if they change their upstream? Right now, it doesn't affect us (we don't have to do anything), but in your setup, it would require us to get yet another AS. Only one of our multihoming customers has a connection to someone we already have a connection with, so there's no path between our network and the rest. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
CA> Date: Wed, 15 Feb 2006 14:04:24 -0600 CA> From: Chris Adams CA> There's a difference: computers (routers) handle the O(N^2) routing CA> problem, while people would have to handle the O(N^2) cooperative AS CA> problem. 0.1 ^ 2 < 5000 One must also consider the scalar coefficient. CA> We are a relatively small ISP with just a handful of multihoming Sounds like the O(N^2) coefficient is small. CA> customers. However, no two of them have the same other provider. What CA> is gained by us setting up relationships with a bunch of other providers CA> and getting special ASes assigned? What if one of those customers gets Each one needs an ASN, anyhow. CA> a connection to a third upstream, or if they change their upstream? Third upstream? They're ready for their own portable ASN. Change upstream? In general, change ASN. Slight pain, but not too bad for the average small leaf. If they're the only entity that uses both you and Joe-Bob's Inet, the coop ASN could still be used when they drop Joe-Bob and replace it with SomeOtherNet. CA> Right now, it doesn't affect us (we don't have to do anything), but in CA> your setup, it would require us to get yet another AS. It does affect you now. You add a BGP session to their portable ASN. Instead, you'd add a new ASN when the other upstream was one with whom you were not already cooperating. CA> Only one of our multihoming customers has a connection to someone we CA> already have a connection with, so there's no path between our network CA> and the rest. Again, this is the big rub. :-( Ideally, everyone would peer via an exchange point, or perhaps a local frame or ATM cloud. I'm more than a little hesitant to suggest shared upstreams leaking long prefixes between the lot of you, or anything else that even smells like a VPN. However, for every entity that multihomes between you and SomeOtherNet, there are probably a hundred with the SBC/Cox combination. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Edward B. DREGER wrote:
CA> Date: Wed, 15 Feb 2006 14:04:24 -0600 CA> From: Chris Adams
CA> There's a difference: computers (routers) handle the O(N^2) routing CA> problem, while people would have to handle the O(N^2) cooperative AS CA> problem.
0.1 ^ 2 < 5000
One must also consider the scalar coefficient.
Err, the problem is not the number of AS numbers (other than having to move to 32bit ones). The 'problem' is the number of prefixes in the routing system. The control plane scales rather well and directly benefits from Moore's law. With todays CPU's there is no problem handling 2 million routes and AS numbers. Absolutely not. Things get a bit more hairy with the forwarding plane though. The faster the link speed the less time it has per lookup and the larger the routing table the more routes it has to search in that ever shrinking amount of time. You see, saving on AS numbers is not really going to help much where it matters. IMHO, and I have stated this before, the best way to handle the route issue is to hand out IPv6 /32 for multihoming and make it the routeable entity. Perfect matches in hardware are pretty easy to do for large numbers of them compared to longest match. On the plus side perfect match scales much better too and can be done in parallel or distributed within a routing chip. Doing the same for longest-match requires a lot more effort. With perfect-match having 2 million routes is not much of a problem too. -- Andre
I like this idea, and your argument about simplifying the forwarding plane makes sense. Should every edge node speak the EGP, or is be static from the NSP? My assumption is that they'll generally be static from the NSP, unless multihomed to keep from exchanging routing information about 2^32 prefixes. -ejay
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Andre Oppermann Sent: Wednesday, February 15, 2006 2:42 PM To: Edward B. DREGER Cc: nanog@merit.edu Subject: Re: a radical proposal (Re: protocols that don't meet the need...)
Edward B. DREGER wrote:
CA> Date: Wed, 15 Feb 2006 14:04:24 -0600 CA> From: Chris Adams
CA> There's a difference: computers (routers) handle the
O(N^2) routing
CA> problem, while people would have to handle the O(N^2) cooperative AS CA> problem.
0.1 ^ 2 < 5000
One must also consider the scalar coefficient.
Err, the problem is not the number of AS numbers (other than having to move to 32bit ones). The 'problem' is the number of prefixes in the routing system. The control plane scales rather well and directly benefits from Moore's law. With todays CPU's there is no problem handling 2 million routes and AS numbers. Absolutely not.
Things get a bit more hairy with the forwarding plane though. The faster the link speed the less time it has per lookup and the larger the routing table the more routes it has to search in that
ever shrinking amount of time.
You see, saving on AS numbers is not really going to help much where it matters.
IMHO, and I have stated this before, the best way to handle the route issue is to hand out IPv6 /32 for multihoming and make it the routeable entity. Perfect matches in hardware are pretty easy to do for large numbers of them compared to longest match. On the plus side perfect match scales much better too and can be done in parallel or distributed within a routing chip. Doing the same for longest-match requires a lot more effort. With perfect-match having 2 million routes is not much of a problem too.
-- Andre
AO> Date: Wed, 15 Feb 2006 21:41:53 +0100 AO> From: Andre Oppermann AO> Err, the problem is not the number of AS numbers (other than having to AO> move to 32bit ones). The 'problem' is the number of prefixes in the It's both. AO> routing system. The control plane scales rather well and directly AO> benefits from Moore's law. With todays CPU's there is no problem AO> handling 2 million routes and AS numbers. Absolutely not. For some equipment. However, I encounter a number of 7200s still in service. AO> Things get a bit more hairy with the forwarding plane though. The AO> faster the link speed the less time it has per lookup and the larger AO> the routing table the more routes it has to search in that ever shrinking AO> amount of time. Yes. AO> You see, saving on AS numbers is not really going to help much where it AO> matters. It's also saving on route count. In my example, Cox and SBC partner up and share an ASN and a netblock. That's _one_ global route for a ton of dual-homed leaves. AO> IMHO, and I have stated this before, the best way to handle the route AO> issue is to hand out IPv6 /32 for multihoming and make it the routeable Agree. AO> entity. Perfect matches in hardware are pretty easy to do for large AO> numbers of them compared to longest match. On the plus side perfect AO> match scales much better too and can be done in parallel or distributed AO> within a routing chip. Doing the same for longest-match requires a lot AO> more effort. With perfect-match having 2 million routes is not much of AO> a problem too. All true. But can we wait for all the forklifts? Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Edward B. DREGER wrote:
AO> Date: Wed, 15 Feb 2006 21:41:53 +0100 AO> From: Andre Oppermann
AO> Err, the problem is not the number of AS numbers (other than having to AO> move to 32bit ones). The 'problem' is the number of prefixes in the
It's both.
AO> routing system. The control plane scales rather well and directly AO> benefits from Moore's law. With todays CPU's there is no problem AO> handling 2 million routes and AS numbers. Absolutely not.
For some equipment. However, I encounter a number of 7200s still in service.
So what? The newer 7200s have got NPE-G1's or soon NPE-G2's in them. Comes with 1G RAM default. It's not that your 7 year old NPE-150 can still participate in todays DFZ, is it? We're not going to explode the table to 2 million routes by this evening. It still takes its time. You always had to upgrade to keep up with [speed, pps, routes, features] and it's not going to change. Get over it. I'm not saying only a Cisco CRS-1 or Juniper M640 can handle it.
AO> Things get a bit more hairy with the forwarding plane though. The AO> faster the link speed the less time it has per lookup and the larger AO> the routing table the more routes it has to search in that ever shrinking AO> amount of time.
Yes.
AO> You see, saving on AS numbers is not really going to help much where it AO> matters.
It's also saving on route count. In my example, Cox and SBC partner up and share an ASN and a netblock. That's _one_ global route for a ton of dual-homed leaves.
1) How does this deal with local loop failures and other routing trouble? Think very hard. You see? 2) You are missing the renumbering issue. Multihomed customer doesn't want renumber when he changes any of the ISPs in the mix. That's why everyone wants PI space.
AO> entity. Perfect matches in hardware are pretty easy to do for large AO> numbers of them compared to longest match. On the plus side perfect AO> match scales much better too and can be done in parallel or distributed AO> within a routing chip. Doing the same for longest-match requires a lot AO> more effort. With perfect-match having 2 million routes is not much of AO> a problem too.
All true. But can we wait for all the forklifts?
Well, the policy and some aspects of the implementation have to change anyway. Why not do it in a way that at least scales before we hit the other brickwall? -- Andre
AO> Date: Wed, 15 Feb 2006 22:18:04 +0100 AO> From: Andre Oppermann AO> So what? The newer 7200s have got NPE-G1's or soon NPE-G2's in them. AO> Comes with 1G RAM default. It's not that your 7 year old NPE-150 can AO> still participate in todays DFZ, is it? We're not going to explode It'll be interesting to see if those NPE-G1s can handle all the DSL/cable multihomers and all the flapping. AO> the table to 2 million routes by this evening. It still takes its No, but if word got out that people could multihome effectively between cable and DSL, it'd happen pretty darn quickly. AO> time. You always had to upgrade to keep up with [speed, pps, routes, AO> features] and it's not going to change. Get over it. I'm not saying AO> only a Cisco CRS-1 or Juniper M640 can handle it. No, but people will resist something that their reasonably-new NPE-G1s and M40s can't handle. Get over it. AO> 1) How does this deal with local loop failures and other routing trouble? AO> Think very hard. You see? If you have followed the thread, you will note that this has been addressed. AO> Well, the policy and some aspects of the implementation have to change AO> anyway. AO> Why not do it in a way that at least scales before we hit the other AO> brickwall? I have proposed something that can be done _today_ with existing equipment (except for minor CPE changes). "Buy all new hardware each time someone wants or needs a feature" is not eaxactly scalable, either. Of course, what would I know? I've never been tasked with installing 2000 new line cards because someone failed to exploit a possibility for increased efficiency c/o simple policy decisions. You're simply shifting costs to hardware. If the bottom line is cheaper than providers cooperating, great. I'm not convinced that it is. Your kneejerk "buy more hardware" response is foolish and short-sighted: At the end of the day, _someone_ has to pay for everything. Why not seek the best cost/benefit? Prefix count is a concern. Let's say that IP space had been allocated in such a manner that each ASN has only one prefix. (This could have been achieved through better allocation practices. I digress.) That would be a global table roughly 10% what it is now. If you're going to cite Moore's law, keep this in mind: A factor of 10 is more than three Moore cycles, or roughly five years. That's not something just to blow off. You suggest exact-match lookup because it is efficient. I agree. I'm suggesting administrative policies that are efficient. The two are not mutually exclusive. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
CA> Date: Wed, 15 Feb 2006 14:04:24 -0600 CA> From: Chris Adams CA> Only one of our multihoming customers has a connection to someone we CA> already have a connection with, so there's no path between our network CA> and the rest. I overlooked something: You lack connections at the IP layer. Do you all obtain DSL loops from the same ILEC? Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On Feb 15, 2006, at 2:30 PM, Edward B. DREGER wrote:
The biggest problem is when customer's link to provider A goes down and inbound traffic must flow through provider B. This necessitates some sort of path between A and B where more-specifics can flow.
Are most of the multihomers REALLY a one router shop (implied by your renumbering is easy comment) - although shim6 could help there I guess. You've also eliminated any possibility of the end multihomed site doing any ingress traffic engineering. I suppose they can do egress which is better than shim6 allows... but in today's world where I get a completely different price for transit than my neighbor - this plan is going to screw some the multihomed sites financially.
JP> Date: Thu, 16 Feb 2006 12:05:35 -0500 JP> From: John Payne JP> Are most of the multihomers REALLY a one router shop (implied by your JP> renumbering is easy comment) - although shim6 could help there I guess. Dual-homed leaves, particularly those who [would] use DSL and cable? Yes. And it wasn't just implied -- I explicitly said it. :-) JP> You've also eliminated any possibility of the end multihomed site doing any JP> ingress traffic engineering. I suppose they can do egress which is better JP> than shim6 allows... but in today's world where I get a completely different JP> price for transit than my neighbor - this plan is going to screw some the JP> multihomed sites financially. Ingress needs some mechanism; this is true. Back to the Cox/SBC 1.0.0/18 example: 1.0.0/20 = prefer SBC (prepend Cox 1x) 1.0.16/20 = prefer Cox (prepend SBC 2x) 1.0.32/19 = prefer Cox (prepend SBC 1x) Still better than heavily-fragmented /29s that can't be aggregated because next-door neighbors foul up 2^x boundaries. Again, with dual-homed leaves, there just aren't that many separate routing policies possible. If multiple networks have the _same_ routing policy, why not coalesce? (BGP attributes certainly aren't replicated senslessly. Why not? Hardware is cheap, after all.) Want to override that? Great. Use shim6. Maybe it'll fare better than loose source routing and DPA, two things to which shim6 smells similar. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Hello; On Feb 15, 2006, at 2:02 PM, Paul Jakma wrote:
On Wed, 15 Feb 2006, Edward B. DREGER wrote:
Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ address space. Each provider announces the aggregate co-op space via the joint ASN as a downstream.
This is unworkable obviously: Think next about SBC and (say) Verizon customers, then what about those with Cox and Verizon, then SBC/Cox/Verizon. etc.
At first, I thought so too. But, it is a fact that in many locations the number of possible connections is very limited. Here in Western Fairfax, basically just the two mentioned. Why not create aggregation ASN that exploit that ? Otherwise, I think that you are dealing with an explosion in the routing tables as people multihome. A real objection here would be that this would tend to lock out new providers (say I wanted to set up a 802.16 ISP in Northern Virginia - I would not be happy if everyone had to renumber to use me. But, why not allow new ASN's into the aggregation AS pool if they meet some minimal requirements, and are in the right geographical area. In a sense, you would be trading some local inefficiency in return for a greater global efficiency. Regards Marshall Eubanks
regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: It is amazing how complete is the delusion that beauty is goodness.
On Wed, 15 Feb 2006, Marshall Eubanks wrote:
very limited. Here in Western Fairfax, basically just the two mentioned. Why not create aggregation ASN that exploit that ?
Well you don't need to assign an ASN for Cox and SBC to announce a shared prefix for a start off. regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: Today I will gladly share my experience and advice, for there are no sweeter words than "I told you so."
PJ> Date: Wed, 15 Feb 2006 20:46:33 +0000 (GMT) PJ> From: Paul Jakma PJ> Well you don't need to assign an ASN for Cox and SBC to announce a shared PJ> prefix for a start off. Technically true, but administratively not feasible. Coordinating private ASNs would be similar to coordining RFC1918 space between different entities, although it's definitely a nice goal. Or are you alluding to inconsistent origin ASNs? Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Edward B. DREGER wrote:
PJ> Date: Wed, 15 Feb 2006 20:46:33 +0000 (GMT) PJ> From: Paul Jakma
PJ> Well you don't need to assign an ASN for Cox and SBC to announce a shared PJ> prefix for a start off.
Technically true, but administratively not feasible. Coordinating private ASNs would be similar to coordining RFC1918 space between different entities, although it's definitely a nice goal.
I'd say you already fail at coordinating Cox and SBC before any link is up and prefixes get announced. I bet they can't even agree on which of them writes the prefix and ASN application to ARIN. $realworld always wins. -- Andre
AO> Date: Wed, 15 Feb 2006 22:20:21 +0100 AO> From: Andre Oppermann AO> $realworld always wins. Translation: "Shift as much cost as you can to as many other entities as you can." $realworld says that is short-sighted. If selfishly saving $x increases the overall economy's cost by $10x, it doesn't take a genius to see the net loss. I doubt I'll retire within the next five years. I prefer not to pee in the pool in which I must swim. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Edward B. DREGER wrote:
MA> Date: Wed, 15 Feb 2006 16:31:56 +0100 (CET) MA> From: Mikael Abrahamsson
MA> The current routing model doesn't scale. I don't want to sit 5 years from MA> now needing a router that'll handle 8 million routes to get me through the MA> next 5 years of route growth. MA> MA> PI space for multihoming and AS number growth is a bad thing for scaling and MA> economics, however you look at it.
ED> I'm going to suggest something horribly radical (or nostalgic, ED> depending how long one has been in the industry): inter-provider ED> cooperation. ED> ED> Let's examine _why_ the routing table might become large. Lots of ED> smaller players multihoming, yes? Say two million small businesses ED> multihome using SBC and Cox. Must we have two million global ASNs ED> and routes? ED> ED> Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ ED> address space. Each provider announces the aggregate co-op space ED> via the joint ASN as a downstream. This makes a lot of sense. However, as one of those smaller players, who may be multihomed using SBC and Cox, as your example says, I can fairly say that I don't like renumbering very much, and sometimes one finds there is a good business case to be made to switch providers, In short, having an ASN is good for me, if not for the community at large, so how to balance that? Right now, gettin ONE upstream to issue a private asn can be like an amatuer dental extraction, imagine 2 that don't like each other, or more often are totally ambivalent with regards to the others concerns/cares/policies/proceedures, et al. One says xxx00, and the other xxx01, how am I supposed to sort this out? ED> This is very similar to a downstream using a private ASN to connect ED> to one upstream in two different locations. i.e., transit provider ED> uses the same ASN for all such customers, and certainly needn't ED> pollute the global table with longer prefixes. mmmm, okay, ED> We're dealing with _one_ routing policy: hand it to Cox, or hand it ED> to SBC. Why explode it into two million "different" policies? Are we? Or are we dealing with _one_ routing policy: handing it to Cox AND handing it to SBC, who mediates? Right now, it appears as if it would me, the end-user. I'm just not equipped for that. ED> Look at MPLS. It essentially hunts down congruent or similar ED> routing policies, slaps a tag on the packet, and routes based ED> that. Why not explore options that get it right and coalesce from ED> the get-go? ED> Note also that this is totally op-community. No new protocols ED> required. ED> It can be done today without forklifts. Agreed. Good idea. Nice idea, very appealing, really. I just don't know how it would play out in practice between two providers, who as we have seen over recent short history don't necessarily work and play well together. ED> I thought I proposed this at 35. Maybe that was one of the open mic ED> sessions where time ran out...> ED> ED> Eddy
--footer snipped
CM> Date: Wed, 15 Feb 2006 14:37:44 -0500 CM> From: Chip Mefford CM> ED> Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ CM> ED> address space. Each provider announces the aggregate co-op space CM> ED> via the joint ASN as a downstream. CM> CM> This makes a lot of sense. BTW, Paul, FixedOrbit reports 701 as having ~1500 peers and downstreams. As interconnected as even they are, that's still a far cry from the full-mesh O(N^2) situation you seemed to suggest. CM> However, as one of those smaller players, who may be multihomed CM> using SBC and Cox, as your example says, I can fairly say CM> that I don't like renumbering very much, and sometimes one CM> finds there is a good business case to be made to switch providers, CM> In short, having an ASN is good for me, if not for the community CM> at large, so how to balance that? Changing ASNs is easy for small, one-router installations. Renumbering would still be necessary, but that's no different than the status quo. i.e., my proposal does not make this worse. That said, let's think if we can improve in that area. CM> Right now, gettin ONE upstream to issue a private asn can be CM> like an amatuer dental extraction, imagine 2 that don't like each other, CM> or more often are totally ambivalent with regards to the others CM> concerns/cares/policies/proceedures, et al. One says xxx00, and CM> the other xxx01, how am I supposed to sort this out? RIR-issued public ASNs. I probably should have merged the "truly radical" thread with this one. CM> ED> We're dealing with _one_ routing policy: hand it to Cox, or hand it CM> ED> to SBC. Why explode it into two million "different" policies? CM> CM> Are we? Or are we dealing with _one_ routing policy: handing CM> it to Cox AND handing it to SBC, who mediates? Right now, it CM> appears as if it would me, the end-user. I'm just not equipped for that. Exactly. It's _one_ routing policy. Not equipped? A little SOHO router could easily accept default and advertise a prefix or two via BGP. Once upon a time a 2500 held a full table; consumer-grade routers of today boast better CPU and RAM. Okay, so consumers must flash new firmware or forklift their $100 router. Oh well. CM> I just don't know how it would play out in practice between CM> two providers, who as we have seen over recent short history CM> don't necessarily work and play well together. In which case it's time for consumers to vote with their wallets. Or, if that's not possible, perhaps the FCC and SEC (in the USA) need to evaluate certain providers. Hence the "radical" aspect of the suggestion. ;-) Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
<aside: text below seems to reply to me specifically, but for some strange reason you decided to strip my address from your reply.> On Wed, 15 Feb 2006, Edward B. DREGER wrote:
BTW, Paul, FixedOrbit reports 701 as having ~1500 peers and downstreams. As interconnected as even they are, that's still a far cry from the full-mesh O(N^2) situation you seemed to suggest.
I'm not sure what bearing any specific number has on O(n^2) behaviour. However, if you want to look at specific numbers, plug '10' in there, then try '20'. regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: Avoid reality at all costs.
PJ> Date: Wed, 15 Feb 2006 23:41:15 +0000 (GMT) PJ> From: Paul Jakma PJ> <aside: text below seems to reply to me specifically, but for some strange PJ> reason you decided to strip my address from your reply.> That portion did, but the rest of my message did not. VZW's 1xRTT service was getting ugly, so I didn't re-paste your headers from the original message. PJ> > BTW, Paul, FixedOrbit reports 701 as having ~1500 peers and downstreams. PJ> > As interconnected as even they are, that's still a far cry from the PJ> > full-mesh O(N^2) situation you seemed to suggest. PJ> PJ> I'm not sure what bearing any specific number has on O(n^2) behaviour. O(N^2) only becomes problematic [when it actually happens and] when the net result is large. For a given N, O(N^2) can be smaller thatn O(ln N) if the latter has a smaller coefficient. Moreover, I'm convinced the problem isn't O(N^2) in practice. Someone with more math skills than any poster in this thread (self included) needs to weigh in, but... again... Empirically speaking, how many different transits service the same geographic areas _and_ will share a downstream? "Lots" of providers in 1 Wilshire, Telehouse NY, PAIX, et cetera, yet any of those locations is lucky to have 1% of the total transit networks. I'll spell it out again: In reality, one need not worry about each transit AS sharing an ASN with every other transit AS. The Internet is not a full mesh, peering is not full-mesh, and it baffles me no end why so many people think coop ASNs _would_ be full mesh. Stop. Examine. Think. Then respond. PJ> However, if you want to look at specific numbers, plug '10' in there, then PJ> try '20'. I started out with 100. See previous posts. Now, show me the market with 100 ASes where downstreams will connect to every last combination of them. BTW: With the status quo, each downstream needs its own ASN and announces its own prefixes. Let's also keep in mind the self-bounding nature of the problem. Hint: 30k transit ASes * 30k transit ASes / 2 = 450M combinations Does anyone here really believe that 450M people will dual home, and that _all_ will have _separate_ provider combinations? Coop ASNs/IP save ASNs and aggregate routes. Full stop. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On Thu, 16 Feb 2006, Edward B. DREGER wrote:
Stop. Examine. Think. Then respond.
[...]
Coop ASNs/IP save ASNs and aggregate routes. Full stop.
Maybe I missed it, but is there something in your solution that keeps dual-homed leaves from having to renumber when changing ISPs? In your concept, is there some "ownership" of the address space on the part of the customer? I know that being able to swing to a new provider (due one of a hundred reasons, including Layer 8 manangement decisions) without having to renumber or suffer significant downtime is one of the things that makes dual-homing appealing. -- John A. Kilpatrick john@hypergeek.net Email| http://www.hypergeek.net/ john-page@hypergeek.net Text pages| ICQ: 19147504 remember: no obstacles/only challenges
JAK> Date: Wed, 15 Feb 2006 18:51:16 -0800 (PST) JAK> From: John A. Kilpatrick JAK> Maybe I missed it, but is there something in your solution that keeps JAK> dual-homed leaves from having to renumber when changing ISPs? In your Note: I'm approaching this from a "something to do today" IPv4 standpoint that also works for IPv6. Subnets lengths are IPv4. I suggested IP policies similar to current ones: longer than /24 requires renumbering no matter what, [/24,some_boundary] is a chunk from provider space, and shorter than some_boundary may be PI. Note that /24 is arbitrary; I use it because that is what's found in the wild today. This could be changed. Likewise, some_boundary has existing values. Others proposed region-specific IPs. I once believed strongly in that, then had mixed feelings... and now believe we should perhaps look at Australia. In a word, no, I'm not approaching PI space. It's attractive, but requires bigger routing tables or source routing. IPv6 with /32 prefixes (conducieve to exact-match hardware) handles the former. SHIM6 resembles the latter. Want to try SHIM6? Build an IPv4 analog today: Use multihop eBGP or similar to advertise one's upstream routers, then expect the remote end to loose source route the return traffic. JAK> concept, is there some "ownership" of the address space on the part of the JAK> customer? I know that being able to swing to a new provider (due one of a JAK> hundred reasons, including Layer 8 manangement decisions) without having to Put more directly, IP ranges should be separate from routing policy, even for networks of only 10 hosts. I'll address this in a separate thread. Folks, brush up on your procmail skills... JAK> renumber or suffer significant downtime is one of the things that makes JAK> dual-homing appealing. PI space and multihoming are orthogonal. A network can have one without the other. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On Wed, Feb 15, 2006 at 06:51:16PM -0800, John A. Kilpatrick wrote:
On Thu, 16 Feb 2006, Edward B. DREGER wrote:
Stop. Examine. Think. Then respond.
Something about history repeating applies. those who weren't around then should re-visit tli's ISPAC proposal from 96 and the associated discussion on both nanog and cidrd archives before regurgitating it. Flatly, there are economic pressures to carry deaggregates and they have undermined/reversed the progress from cidr. Solve that [approaches other than longest-match always wins] or else yes the issue will carry to v6. This is one aspect of the "get ietf to re-examine routing" folks at the sessions were on about. Then we get back to "the ietf is composed of vendors who want to see your capital dollars on a recurring basis" sub-topic... Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Wed, 15 Feb 2006, Joe Provo wrote:
On Wed, Feb 15, 2006 at 06:51:16PM -0800, John A. Kilpatrick wrote:
On Thu, 16 Feb 2006, Edward B. DREGER wrote:
Stop. Examine. Think. Then respond.
Something about history repeating applies. those who weren't around then should re-visit tli's ISPAC proposal from 96 and the associated discussion on both nanog and cidrd archives before regurgitating it.
Flatly, there are economic pressures to carry deaggregates and they have undermined/reversed the progress from cidr. Solve that [approaches other than longest-match always wins] or else yes the issue will carry to v6. This is one aspect of the "get ietf to re-examine routing" folks at the sessions were on about. Then we get back to "the ietf is composed of vendors who want to see your capital dollars on a recurring basis" sub-topic...
In line with this... (I had to point this out in a different context on another list before): Networks announce prefixes because doing so makes them money. Networks that listen to these prefixes do so because that makes them money. In light of this, the current size of the routing table is a function of the number of businesses or organizations that would like to multihome limited to those that can afford to pay for the capital costs of a router capable of doing BGP, the operational cost of maintaining it, and the service cost of buying Internet connectivity that includes BGP sessions with their transit providers. Where ever you see that function trending is how big you can expect the routing table to become because of economic pressure. While there are not as many businesses and organizations as people on the planet, as an exercise imagine 4 billion prefixes. For the sake of simplicity assume a 32 bit forwarding tables (4 billion entries) for each interface on a router expandable to 256 interfaces, with a byte (256 possible forwarding destinations) per entry for forwarding (4 GB of RAM). Such a thing could be made now and would attribute to a very small fraction of the cost we currently pay for new Cisco cards. You might not get as good port density for the physical form factor, however it is doable now. This is separate from the convergence discussion for 4 billion prefixes, etc etc etc. The link speed required to be able to converge within a minute is left to the reader. heh. :) Mike. +----------------- H U R R I C A N E - E L E C T R I C -----------------+ | Mike Leber Direct Internet Connections Voice 510 580 4100 | | Hurricane Electric Web Hosting Colocation Fax 510 580 4151 | | mleber@he.net http://www.he.net | +-----------------------------------------------------------------------+
Mike Leber wrote:
In line with this... (I had to point this out in a different context on another list before):
Networks announce prefixes because doing so makes them money. Networks that listen to these prefixes do so because that makes them money.
Indeed. That is precisely the reason why we'll see IPv6 de-aggregates routed in the future just the way we see it with PA space and the swamp today.
In light of this, the current size of the routing table is a function of the number of businesses or organizations that would like to multihome limited to those that can afford to pay for the capital costs of a router capable of doing BGP, the operational cost of maintaining it, and the service cost of buying Internet connectivity that includes BGP sessions with their transit providers.
Well said. $realworld at work.
Where ever you see that function trending is how big you can expect the routing table to become because of economic pressure.
While there are not as many businesses and organizations as people on the planet, as an exercise imagine 4 billion prefixes.
For the sake of simplicity assume a 32 bit forwarding tables (4 billion entries) for each interface on a router expandable to 256 interfaces, with a byte (256 possible forwarding destinations) per entry for forwarding (4 GB of RAM). Such a thing could be made now and would attribute to a very small fraction of the cost we currently pay for new Cisco cards. You might not get as good port density for the physical form factor, however it is doable now.
Hence my case for perfect match. Doing 4 billion prefixes with longest-match is doable as well, but either more expensive when you want constant lookup times or you have to live with varibale lookup times and the max pps is defined by the traffic and route pattern. Perfect match is the reason why even the cheapest DLink gigabit ethernet switch is able to switch packets at wirespeed.
This is separate from the convergence discussion for 4 billion prefixes, etc etc etc. The link speed required to be able to converge within a minute is left to the reader. heh. :)
BGP can be made more efficient with this large number of prefixes. Don't worry on this. -- Andre
On Wed, 15 Feb 2006, Mike Leber wrote:
While there are not as many businesses and organizations as people on the planet, as an exercise imagine 4 billion prefixes.
At the moment mobile IP is not implemented using the global routing infrastructure, because it can't scale to 4 billion prefixes. But if it *could* scale that well, mobile IP would be much simpler. One prefix per device. Tony. -- f.a.n.finch <dot@dotat.at> http://dotat.at/ BISCAY: WEST 5 OR 6 BECOMING VARIABLE 3 OR 4. SHOWERS AT FIRST. MODERATE OR GOOD.
Moreover, I'm convinced the problem isn't O(N^2) in practice. Someone with more math skills than any poster in this thread (self included) needs to weigh in, but... again...
Math skills are not needed. This is a technical and business problem, not a mathematical one. I tried to sell just such a cooperative multihoming solution about 4 years ago when I was with Ebone. We had a client who needed highly available resilient connectivity and part of the RFP was that they wanted two provider networks. We chose BT as our partner because the Ebone fibre network was largely discontinuous with BT's network. It doesn't make sense to do this with two networks who share fibre or conduits or rights-of-way. In our case, the customer was large enough that we could do it by having BT announce a part of our address space, something on the order of a /20. The actual customer peering with both of us would have used private ASNs. The plan was scuttled by financial difficulties which resulted in the demise of the Ebone network. In any case, I agree that this is something that should be available in the market but since it only makes sense if the two providers are largely discontinuous, I don't think that there will be many possible pairings in most cities. Once you get past the technical issue of who is a suitable partner, the big hurdle is on the commercial side getting two competitors to agree on terms to provide a joint service. Although this would be useful for small multihomers, I don't see that as possible until after a large enterprise customer forces two competitors to negotiate terms. --Michael Dillon
On Feb 15, 2006, at 9:13 AM, Edward B. DREGER wrote:
Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ address space. Each provider announces the aggregate co-op space via the joint ASN as a downstream.
Interesting. This is what has been called metropolitan addressing. I'm certainly not the one who first proposed it, although I have thought about it for a while, dating at least as far back as 2001. The crux of the concept as several *have* proposed it is that a regional authority - a city, perhaps, or a consortium of ISPs, or in the latest version of the proposal I have seen the country of Korea - gets a prefix, and sets up an arrangement. SOHOs that want to multihome within its territory are able to get small (/48? /56?) prefixes from it, and providers that deliver service in the area may opt in to supporting such SOHO prefixes. If they opt in, they are agreeing to: - join a local IXP, which may be a physical switch or virtualized by a set of bilateral agreements. - outside the region, they advertise the prefix of the regional authority - within the region, for customers that have gotten such a prefix, if they have connectivity to the customer they advertise the customer's prefix to the ISPs at the IXP. Note that the customer is not expected to run BGP or get an AS number, but either the regional authority gets an AS number or each serving ISP is deemed authorized to originate the prefix in its BGP announcements. But if a SOHO has two ISPs, both advertise its prefix within the region, and when a packet is sent to the prefix from wherever, any ISP that is delivering service to the SOHO can legitimately deliver it, and if one gets the packet but is not the servicing ISP, it knows how to hand the packet to the appropriate ISP at the IXP. This turns the business model of routing on its head. Typically today if Alice is using ISP AliceNet and Bob is using ISP BobNet, Alice hands her packet to AliceNet, AliceNet gets it to BobNet in the cheapest way it can, and BobNet carries it halfway around the world to Bob. Bob's ISP carries the burden of most of the work. But in this model, if AliceNet happens to also provide service in Bob's region, AliceNet might carry the packet to the region and only give it to BobNet for the last 500 feet. Whenever I have talked about the model with an ISP, I have gotten blasted. Basically, I have been told that (1) any idea on operations proposed in the IETF is a bad idea because the IETF doesn't listen to operators (2) the ISPs aren't going to be willing to make settlement payments among themselves in accordance with the plan (3) routing isn't good enough to support it (4) and in any event, this makes it too easy to change ISPs In short, "hell no". So, since nobody in the IETF (according to you) is supporting this model, what I understand from your remark and this thread is that the IETF is not responsive to ideas proposed by operators and doesn't come up with things operators can use, taking as an example that it hasn't told you how to implement metropolitan addressing. Did I get that right? I'm not sure how to proceed, given the level of invective I get in any discussion with anyone on the topic. Note 1: PI addressing for edge networks that can qualify under a sensible set of rules (current ones are inadequate) for an AS number is the preferred way to handle an enterprise of a size r complexity comparable to a small (or large) ISP. Note 2: Provider-provisioned addresses continue to make sense for folks that don't plan to multihome.
I'm sure I'm going to regret posting this, if for no other reason than that I will immediately start receiving more spam, and I suspect that I am just re-stating things that TLi and others have been trying to state both here and on PPML, but I guess I just can't resist today... [Disclaimer: I don't work for a provider these days; in fact, I work for the same vendor that Fred does, so it may seem odd that we are arguing... but I did work for university/regional/national/global ISPs from 1988 until 2001 and during this time did participate, to some degree, in the IETF process. I even tried to contribute to the IPNG process early on until it became clear that the political process that drove the selection of ipv6 had very little connection to operational reality. In case it isn't obvious, these views are mine alone and do not represent the position of my or your employer]
Interesting. This is what has been called metropolitan addressing. I'm certainly not the one who first proposed it, although I have thought about it for a while, dating at least as far back as 2001. ... This turns the business model of routing on its head. Typically today if Alice is using ISP AliceNet and Bob is using ISP BobNet, Alice hands her packet to AliceNet, AliceNet gets it to BobNet in the cheapest way it can, and BobNet carries it halfway around the world to Bob. Bob's ISP carries the burden of most of the work. But in this model, if AliceNet happens to also provide service in Bob's region, AliceNet might carry the packet to the region and only give it to BobNet for the last 500 feet.
To address your points:
Whenever I have talked about the model with an ISP, I have gotten blasted. Basically, I have been told that
(1) any idea on operations proposed in the IETF is a bad idea because the IETF doesn't listen to operators
Would you disagree that the IPNG process essentially ignored the "hard" issues (multihoming, endpoint-id/routing locator split, easy/transparent renumbering, etc.) that were raised some 10 or more years ago? It may have been "operators" who were most vocal in raising these issues (since they are the ones who are suffering and will suffer the consequences of their not being addressed -- no pun intended) but there were some pretty smart people who didn't work for operators (e.g. JNC, TLi) who also argued for something better than "IP with bigger addresses" as being needed for IPNG. This certainly gave some credence to the idea that the IETF "doesn't listen to operators" or to the others calling for a re-examination of the routing architecture. Slight digression: I recall getting up during the plenary (at the time, I was very public-speaking-averse, so the memory is vivid) at the Amsterdam IETF (July, 1993) back when the whole "IP isn't going to scale; we need to build something better" sentiment was starting. I stated that any solution that didn't deal with transparent renumbering and multihoming was a non- tarter. There was lots of applause then and promises that these issues would definitely be covered. We still wound up with a non-functional ipv6.
(2) the ISPs aren't going to be willing to make settlement payments among themselves in accordance with the plan (3) routing isn't good enough to support it (4) and in any event, this makes it too easy to change ISPs
In short, "hell no".
It's a little more basic than that. I'm no graph theory expert and reading such stuff gives me a headache, but I do understand that abstraction (summarization or aggregation) of routing information is only possible if the identifiers that are used for numbering network elements (the "addresses") are assigned in a manner that is isomporphic to the network topology. TLi started writing a good paper which described this in terms of sets and subsets; unfortunately, I don't think it ever saw the light of day). Those who propose "geo-topological" addressing, an oxymoron if ever there were one, are effectively dictating how the network topology is to be organized, with rather profound implications for provider business models. If addresses are assigned in this manner, then service providers whose networks span multiple address assignment domains (connect to more than one city or however the geograpic areas are split up) must: a) connect to all designated interconnection facilities associated with the address assignment authorities in the geographic areas they wish to serve and 1) carry all more-specific routes for all providers in all of the cities that they serve (which eliminates aggregation) or 2) provide free transit service for any customer of a competitor in a geographic area whose addresses are aggregated or 3) enter into a settlement agreement (which implies a regulatory regime unprecedented in the Internet business) with all other providers in geographic areas which they serve Is it any surprise that large service providers are fundamentally opposed to such a radical change in Internet business practices, one which effectively dictates how they have to build their networks, what interconnection facilities they must join, and how they must interact with competitors, either by offering free transit service or by negotiating settlement contracts? Until the IP "address" is replaced by an endpoint identifier and a routing locator, it will not be possible to design a scalable routing architecture. Years ago, some smart people (much smarter than me) tried to make it clear how vitally important this distinction was. But the IPNG political process ignored those people and the result was the undeployable mess that is ipv6. If you want the Internet to scale to millions of customer sites that have full flexibility to multihome with providers of their choice, the id/locator split is essential; it may be possible to acheive this and still use ipv6 packet formats but incompatible implementation changes to "address" handling semantics are needed at the transport and internetworking layer (think: 8+8/GSE and "agile transport identifier use"). --Vince
It's a little more basic than that. I'm no graph theory expert and reading such stuff gives me a headache, but I do understand that abstraction (summarization or aggregation) of routing information is only possible if the identifiers that are used for numbering network elements (the "addresses") are assigned in a manner that is isomporphic to the network topology. TLi started writing a good paper which described this in terms of sets and subsets; unfortunately, I don't think it ever saw the light of day).
One of the first things I ever learned from Yakov (at the first IETF I ever attended): "Addressing can follow topology or topology can follow addressing. Choose one." He has since pointed out that this may not be strictly true when considering VPN technologies. Dave
On Thu, 16 Feb 2006, David Meyer wrote:
One of the first things I ever learned from Yakov (at the first IETF I ever attended):
"Addressing can follow topology or topology can follow addressing. Choose one."
So which one was it when you guys were developing ipv6 in late 1990s? (or more correctly which one did you think you were doing because I think its still up for debates considering PI allocation issues, etc)
He has since pointed out that this may not be strictly true when considering VPN technologies.
Dave
Since meeting Yakov years ago, I have always tried to teach network designers to consider addressing and topology together. It hasn't always worked. Many don't care about network population estimates and demographics, some don't recognize those terms, and, a few just want enough Class C networks to get their job done for the day. Cutler At 2/16/2006 10:41 AM -0800, David Meyer wrote: One of the first things I ever learned from Yakov (at the first IETF I ever attended): "Addressing can follow topology or topology can follow addressing. Choose one." He has since pointed out that this may not be strictly true when considering VPN technologies. Dave - James R. Cutler james.cutler@consultant.com
On Thu, Feb 16, 2006 at 02:42:49PM -0500, James R. Cutler wrote:
Since meeting Yakov years ago, I have always tried to teach network designers to consider addressing and topology together.
It hasn't always worked. Many don't care about network population estimates and demographics, some don't recognize those terms, and, a few just want enough Class C networks to get their job done for the day.
Yep, very true. Dave
Cutler
At 2/16/2006 10:41 AM -0800, David Meyer wrote: One of the first things I ever learned from Yakov (at the first IETF I ever attended):
"Addressing can follow topology or topology can follow addressing. Choose one."
He has since pointed out that this may not be strictly true when considering VPN technologies.
Dave
- James R. Cutler james.cutler@consultant.com
- join a local IXP, which may be a physical switch or virtualized by a set of bilateral agreements.
Why should they join an IXP if they already have private peering arrangements?
- outside the region, they advertise the prefix of the regional authority
Mixing government with operations? If you favor doing that then why not just give IPv6 addresses to the various national governments and let the UN sort it out? Personally I disagree with any scheme which calls for national or municipal governments to assign IPv6 addresses to end users. Dressing it up as a "regional authority" does not make it any nicer. Forcing people to join an unecessary IX is not the way to solve the problem of regional aggregation of routes. This is a purely technical problem which can be solved by the RIR practices in allocating IPv6 addresses. If they would allocate addresses in a geo-topological manner then end users and ISPs would be free to aggregate routes outside of their region without any involvement of governments or any requirement to join consortia or IXes. It does require the users of such geo-topological addresses to ensure that in THEIR region, there is sufficient interconnectivity (physical and policy) between ISPs for the addressing to work. But that does not need to be determined or managed centrally. Geo-topological addressing refers to RIRs reserving large blocks of designated addresses for areas served my large cities (over 100,000) population. When end users are located in fringe areas roughly equidistant between two or more such centers, the RIR simply asks the end user (or ISP) which is the center to which they want to connect (communicate). This addressing scheme operates in parallel with the existing provider-oriented IPv6 addressing scheme but uses a different block of IPv6 addresses out of the 7/8ths that are currently reserved. No hardware or software changes are required for this to work, merely some geographical/economical research to determine the relative sizes of the address pool to be reserved for each of the
Note that the customer is not expected to run BGP or get an AS number, but either the regional authority gets an AS number or each serving ISP is deemed authorized to originate the prefix in its BGP announcements. But if a SOHO has two ISPs, both advertise its prefix within the region, and when a packet is sent to the prefix from wherever, any ISP that is delivering service to the SOHO can legitimately deliver it, and if one gets the packet but is not the servicing ISP, it knows how to hand the packet to the appropriate ISP at the IXP.
This turns the business model of routing on its head. Typically today if Alice is using ISP AliceNet and Bob is using ISP BobNet, Alice hands her packet to AliceNet, AliceNet gets it to BobNet in the cheapest way it can, and BobNet carries it halfway around the world to Bob. Bob's ISP carries the burden of most of the work. But in this model, if AliceNet happens to also provide service in Bob's region, AliceNet might carry the packet to the region and only give it to BobNet for the last 500 feet.
Whenever I have talked about the model with an ISP, I have gotten blasted. Basically, I have been told that
(1) any idea on operations proposed in the IETF is a bad idea because the IETF doesn't listen to operators (2) the ISPs aren't going to be willing to make settlement payments among themselves in accordance with the plan (3) routing isn't good enough to support it (4) and in any event, this makes it too easy to change ISPs
In short, "hell no".
So, since nobody in the IETF (according to you) is supporting this model, what I understand from your remark and this thread is that the IETF is not responsive to ideas proposed by operators and doesn't come up with things operators can use, taking as an example that it hasn't told you how to implement metropolitan addressing.
Did I get that right?
I'm not sure how to proceed, given the level of invective I get in any discussion with anyone on the topic.
Note 1: PI addressing for edge networks that can qualify under a sensible set of rules (current ones are inadequate) for an AS number is the preferred way to handle an enterprise of a size r complexity comparable to a small (or large) ISP.
Note 2: Provider-provisioned addresses continue to make sense for folks that don't plan to multihome.
On 16-feb-2006, at 0:15, Fred Baker wrote:
On Feb 15, 2006, at 9:13 AM, Edward B. DREGER wrote:
Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ address space. Each provider announces the aggregate co-op space via the joint ASN as a downstream.
Interesting. This is what has been called metropolitan addressing. I'm certainly not the one who first proposed it, although I have thought about it for a while, dating at least as far back as 2001.
The crux of the concept as several *have* proposed it is that a regional authority - a city, perhaps, or a consortium of ISPs, or in the latest version of the proposal I have seen the country of Korea - gets a prefix, and sets up an arrangement. SOHOs that want to multihome within its territory are able to get small (/48? /56?) prefixes from it, and providers that deliver service in the area may opt in to supporting such SOHO prefixes.
[...]
Whenever I have talked about the model with an ISP, I have gotten blasted.
Well, the way you outline above isn't the only way to do aggregation on something other than provider. A while back I worked on this, see http://www.muada.com/drafts/draft-van-beijnum-multi6-isp-int-aggr-01.txt The idea is that a border router within an ISP/carrier network no longer holds a full copy of the global routing table, but only carries a subset. The AS as a whole still has a full view of the entire table, but aggregates make packets flow to a router that holds the appropriate part of the global routing table, and then that router hands the packets off to the right neighboring AS. The aggregates are only used within the AS so there is no free transit. Obviously it works best if there is interconnection in the metro area in question, but it can also be made to work without dense interconnection. Based on NANOG shim6 feedback and the push for IPv6 PI in the RIRs, I think it's time to really look at this and/or other non-traditional ways to aggregate. Apart from traffic engineering (which should be solvable) the main problem with shim6 is that it doesn't give users provider independent address space, and it's becoming pretty clear that many users REALLY want this, not withstanding all the IETF efforts to make renumbering easier. Some sort of non-provider aggregation would allow portable address space for end-users without starting a time delayed meltdown of the global routing table. Another advantage is that such a mechanism makes it possible to start using aggregatable PI space as normale PI space immediately, and only implement aggregation in individual ASes (no coordination necessary) as the size of the routing table increases. I dropped this approach 2.5 years ago when it turned out that there was no support for it in the multi6 working group, but the heavy criticism of shim6, the push for PI in IPv6 and the fact that geographic aggregation keeps coming up from time to time suggests that it's probably not a bad investment of time for the IETF to look at this and see if there's something there. Maybe in the form of a BOF? Iljitsch
Edward B. DREGER:
...Of course not. Let SBC and Cox obtain a _joint_ ASN and _joint_ address space. Each provider announces the aggregate co-op space via the joint ASN as a downstream....
So this is a good proposal if I*(I-1)/2 < C where C = number of ASNs issued to dual-homed customers I = number of ASNs issued to "Transit Providers" said customers might select from (Note that it is bigger than that on the left if anyone, god forbid, has *more* than two ISPs) My guess is that even with all the consolidation in the industry, the left side grows too quickly for this to be a good idea. (It'd probably be a great way to finish using up the rest of V4 space, though) And that discounts all the reasons why, short of a law forcing them to do so, transit providers wouldn't want to do this anyway. Matthew Kaufman matthew@eeph.com
MK> Date: Wed, 15 Feb 2006 15:35:27 -0800 MK> From: Matthew Kaufman MK> So this is a good proposal if I*(I-1)/2 < C where MK> C = number of ASNs issued to dual-homed customers MK> I = number of ASNs issued to "Transit Providers" said customers might select MK> from MK> MK> (Note that it is bigger than that on the left if anyone, god forbid, has MK> *more* than two ISPs) Note that I specifically said "dual-homed leaves". MK> My guess is that even with all the consolidation in the industry, the left MK> side grows too quickly for this to be a good idea. (It'd probably be a great MK> way to finish using up the rest of V4 space, though) Wrong, wrong, wrong. The left side of your equation assumes that EVERY transit provider will cooperate with EVERY other transit provider. Do all ~30k transit providers service your region? Didn't think so. How about even 1% of them? I doubt it. Let's compare _actual needed_ coop ASNs with _actual needed_ status quo ASNs. Separate theoretical upper bound from what's gonna happen in the real world. Now let's look at the bigger issue of route consolidation. Follow along carefully, folks. Want to dual-home to SBC and Cox? Great. You get IP space from 1.0.0/18 which is advertised via AS64511. Lots of leaf dual-homers do the same, yet there is ONE route in the global table for the lot of you. SBC and Cox interconnect and swap packets when someone's local loop goes *poof*. Flaps within 1.0.0/18 never hit the outside world. Everyone is happy. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Edward B. DREGER:
Want to dual-home to SBC and Cox? Great. You get IP space from
1.0.0/18
which is advertised via AS64511. Lots of leaf dual-homers do the same, yet there is ONE route in the global table for the lot of you. SBC and Cox interconnect and swap packets when someone's local loop goes *poof*. Flaps within 1.0.0/18 never hit the outside world.
Everyone is happy.
Except for either SBC or Cox, whichever thinks the other is getting the short end of the stick for the long-haul transport. And except for the dual-homers who decide that they like Cox but that they want to use some mom-and-pop multihomed ISP in their town instead of SBC, who all of a sudden need to renumber (which they didn't need to do in the PI model OR if their PA addresses are from Cox) *and* need to try to convince Cox to set up a shared-address/shared-AS arrangement with the mom-and-pop ISP, which is NOT going to happen because clearly the benefits are hugely asymmetric. I believe, in fact, that due to the perceived (and actual) asymmetries of the resulting situation, the number of providers who would agree to enter into such an arrangement with another provider can be counted on zero fingers. (But that certainly fixes the O(n^2) problem) And given the choice between having to renumber (A) any time you changed ANY ONE of your upstreams (this model) and (B) having to renumber only if you changed the first upstream you got (the existing PA model) and (C) having to renumber never (the existing PI model), I think customers would choose C, then maybe B, and almost certainly not A. So other than the downsides for the ISPs and the customers, this is a great idea, but it doesn't sell any additional routers. Matthew Kaufman matthew@eeph.com
On 15-Feb-2006, at 19:33, Edward B. DREGER wrote:
Want to dual-home to SBC and Cox? Great. You get IP space from
1.0.0/18
which is advertised via AS64511. Lots of leaf dual-homers do the same, yet there is ONE route in the global table for the lot of you. SBC and Cox interconnect and swap packets when someone's local loop goes *poof*. Flaps within 1.0.0/18 never hit the outside world.
Personally, if I was going to multi-home, I would far prefer that my various transit providers don't cooperate at all, and have sets of peers and/or upstream transit providers that are as different as possible from each others'. The last thing I need are operational procedures which are shared between them. If all you want is last-mile redundancy, surely you can just attach twice to the same ISP and avoid all the routing complications completely? I get the feeling that there's a lot of solutions-designing going on in this thread without the benefit of prior problem-stating. Joe
JA> Date: Thu, 16 Feb 2006 12:44:27 -0500 JA> From: Joe Abley JA> Personally, if I was going to multi-home, I would far prefer that my various JA> transit providers don't cooperate at all, and have sets of peers and/or JA> upstream transit providers that are as different as possible from each JA> others'. The last thing I need are operational procedures which are shared JA> between them. The biggest sharing would be IP assignment. Let 'A' start at one end of the pool, 'B' at the other, and they'll meet in the "middle". When one hits the boundary, it can be moved. "You're multihoming with 'A' and with us? Okay, fill in the box on your router that says 'ASN' with '64511'." JA> If all you want is last-mile redundancy, surely you can just attach twice to JA> the same ISP and avoid all the routing complications completely? Naturally. JA> I get the feeling that there's a lot of solutions-designing going on in this JA> thread without the benefit of prior problem-stating. Problem: Consumers want to multihome. They may have a dynamic /32, or a /27 if they're "big". They want to do this right here, right now, today, with IPv4, using two separate upstreams. http://www.internetworldstats.com/stats.htm claims ~1B internet users worldwide. Let's pretend that 1% of those were to SOmultiHOme, and that no routes are coalesced. That's 10M new routes. I argue that the current combination of technology and administrative policy cannot support that. Indeed, if it could, _why_ are providers not accepting /32 announcements? If there's no technical reason not to, and a financial reason to, why is it not done? After all, hardware is cheap, upgrading is a fact of life, and allowing SOHO users to multihome their /29 makes money! Why wait for IPv6 to use /32 perfect match? Let's do it today, with IPv4! Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
On 16-Feb-2006, at 13:32, Edward B. DREGER wrote:
JA> I get the feeling that there's a lot of solutions-designing going on in this JA> thread without the benefit of prior problem-stating.
Problem:
Consumers want to multihome.
That sentence needs profound expansion before it's going to be reasonable to assess any proposed solution. (Why do they want to multi-home? What do they hope to achieve? Redundancy? Load sharing? What trade-offs are reasonable, e.g. with respect to the stability of individual sessions across re-homing events? In a transaction carried between two hosts, do clients and servers have different requirements?) We tried to catch a reasonable number of motivations in RFC 3582, but I bet we missed plenty. Joe
On Wed, 15 Feb 2006 16:31:56 +0100 (CET), "Mikael Abrahamsson" <swmike@swm.pp.se> said: [snip]
The current routing model doesn't scale. I don't want to sit 5 years from now needing a router that'll handle 8 million routes to get me through the next 5 years of route growth.
agree!
PI space for multihoming and AS number growth is a bad thing for scaling and economics, however you look at it.
agree!
Shim6 would hopefully curb the prefix growth very early in the growth curve as single entities won't need AS to multihome between two different ISPs.
agree! [snip] All is well if shim6 succeeds it seems ... 5-10 years into the future. Do we all agree to postpone v6 till then? If not there's a need for an intermediary solution. To me it seems like people want 2 things: 1. A working solution. The only alternative with current technology is PI end-site assignments. 2. Reasonable predictability. To make ever-lasting technologies and policies may be the dream in some research communities. The rest of us have to work with what we got and accept that we have to upgrade and make substatial changes to our networks from time to time. An alternative to satisfy those who fear the long term effect of a growing routing-table could be temporary end-site assignments from dedicated address-blocks. At some point in the future, when new-and-mature technology exist, the RIR-community could decide on new policies and decide to re-claim the entire block on e.g. a 24-month notice. ... just my $.02 compromise ;) //per -- Per Heldal http://heldal.eml.cc/
How do you count # of networks? 8M means - 8M of independent, multihomed companies. What is the reson to expect so many? Don't forget that today's number of networks is multiplied few times because you (foten) need to get more than 1 allocation. And what is a problem with 8M networks in next 8 years (if we easily handle 200K just now)? No, this model is well scalable and we better solve other, REAL problems, not mistical _# of networks_ one. ----- Original Message ----- From: "Per Heldal" <heldal@eml.cc> To: "Mikael Abrahamsson" <swmike@swm.pp.se> Cc: <nanog@merit.edu> Sent: Wednesday, February 15, 2006 11:45 AM Subject: Re: protocols that don't meet the need...
On Wed, 15 Feb 2006 16:31:56 +0100 (CET), "Mikael Abrahamsson" <swmike@swm.pp.se> said: [snip]
The current routing model doesn't scale. I don't want to sit 5 years
from
now needing a router that'll handle 8 million routes to get me through the next 5 years of route growth.
agree!
PI space for multihoming and AS number growth is a bad thing for scaling and economics, however you look at it.
agree!
Shim6 would hopefully curb the prefix growth very early in the growth curve as single entities won't need AS to multihome between two
different
ISPs.
agree!
[snip]
All is well if shim6 succeeds it seems ... 5-10 years into the future. Do we all agree to postpone v6 till then?
If not there's a need for an intermediary solution. To me it seems like people want 2 things:
1. A working solution. The only alternative with current technology is PI end-site assignments.
2. Reasonable predictability. To make ever-lasting technologies and policies may be the dream in some research communities. The rest of us have to work with what we got and accept that we have to upgrade and make substatial changes to our networks from time to time. An alternative to satisfy those who fear the long term effect of a growing routing-table could be temporary end-site assignments from dedicated address-blocks. At some point in the future, when new-and-mature technology exist, the RIR-community could decide on new policies and decide to re-claim the entire block on e.g. a 24-month notice.
... just my $.02 compromise ;)
//per -- Per Heldal http://heldal.eml.cc/
On Thu, 16 Feb 2006, Alexei Roudnev wrote:
allocation. And what is a problem with 8M networks in next 8 years (if we easily handle 200K just now)?
Each unit I buy that has to handle 1M routes instead of 256k routes today costs $10-30k or more extra because of this requirement. Yes, it's no technical problem, but it's money out that could be spent on other things. -- Mikael Abrahamsson email: swmike@swm.pp.se
Mikael Abrahamsson wrote:
On Thu, 16 Feb 2006, Alexei Roudnev wrote:
allocation. And what is a problem with 8M networks in next 8 years (if we easily handle 200K just now)?
Each unit I buy that has to handle 1M routes instead of 256k routes today costs $10-30k or more extra because of this requirement. Yes, it's no technical problem, but it's money out that could be spent on other things.
I don't know for how long you're in the game but if it is more than five years then please plot the cost and capabilities of the routers you bought in that time against the growth of the routing table. I have an authentic 4 digit AS number (easily found using the ASNumber Firefox plugin) and when I look at the plot for the past 9 years I don't see any problem for the future. You either get constantly more capabilities for the same amount on money or you pay less money for the same amount of capabilities. The ISP business is like any other business where you have to keep investing to stay ahead. The other way (used primarily by the RIAA) is trying to fix the legislation around your existing investments and business model. If we all were to stop investing and stay at what we have got now the global economy would implode. -- Andre
Daniel, On Wed, Feb 15, 2006 at 11:51:12AM +0100, Daniel Roesen wrote:
On Tue, Feb 14, 2006 at 01:47:31PM -0800, David Meyer wrote:
IETF). Now, while many in the IETF argue that there is no such thing as an "operator community", I personally see it differently, and there are many of us who think that operator input is sorely missing from the IETF process.
The problem with IETF and IPv6 is from my perspective, that operator input is being rejected as unreasonable or just ignored (shim6). I've stopped wasting time trying to bring operator's views/points across. It's not welcome if it doesn't fit already existing views within IETF regarding IPv6. I know that a lot of active IPv6 folks think the same but hesitate to communicate that openly.
I understand your point, and hey, you're not the only one who sees it this way (and IPv6 isn't the only area where this problem is perceived to exist). Suppose we stipulate that your perspective (which is shared by many), namely that operator input is being rejected/ignored, is true. Then if we agree that IPv6 is here (for some value of "here", and as you mention below), then we need to find a way to solve the problems that we've been talking about here. My sense is that the likely place to do that is in the IETF (being the SDO that does this kind of work). So if you agree with what I've said above, how do we break the impasse and move forward? Like you, I'm interested in forwarding our common set of goals, and it seems pretty obvious that we need to find (or revive) a scalable routing architecture for IPv6. So lets find a way to do that (BTW, if I had the answer, I'd be the first to provide it. This is pretty thorney, as you've pointed out).
That is one of the reasons we did the NANOG 35 IPv6 multihoming BOF (and are doing the same at the upcoming apricot meeting).
Which is a good thing. But still, many IETF folks deny the fact that they constantly hear that things like shim6 is NOT what the ops folks (the folks that have to actually work with the stuff IETF brings forward) are looking for. And we know that it doesn't. It can't. There is no way to do traffic engineering with any shim6-like system like one can do with BGP as shim6 is a completely host-centric solution. It has no clue about upstream/downstream/peering, ASses etc. Those things that actually make topology and economics. That's aside all the other administrative nightmares associated.
So I started writing up a I-D (the IETF coin of the realm) on this topic, but got sidetracked making slides for Apricot. I'd welcome anyone who wants to help me with that document (my approach was to outline the issues as a basis for bringing this discussion into the IETF). I could use the help. BTW, the doc is called Issues In Traffic Engineering with SHIM6 draft-meyer-shim6-and-te-00.txt but I haven't published it yet. Again, anyone who wants to contribute/write/whatever, insight/thoughts greatly appreciated.
So (and again, not speaking for the IAB), my perspective is that we really need your insight and perspectives, more generally, your help in solving some of the difficult problems before us (a viable routing and addressing architecture for IPv6 comes to mind).
I firmly believe that this train for IPv6 is long gone. We should go forward with IPv6 using the legacy routing architecture and start NOW working on a complete real re-vamp with a PROPER locator/identifier split - not an ugly hack ontop of the traditional IPv4/IPv6 like shim6 which doesn't deliver what ops folks need.
Nevertheless, I really welcome IAB's efforts in the matter.
Thanks, that is much appreciated. So on the theory that the best way to finish a task is to start it, let's start working on the document I mentioned above (if folks want to). If folks have other ideas, lets get those on the table too. Dave
On Wed, 15 Feb 2006, Daniel Roesen wrote:
It has no clue about upstream/downstream/peering, ASses etc. Those things that actually make topology and economics. That's aside all the other administrative nightmares associated.
Oki, let's step back a bit and look at shim6 from another angle, the user angle. Shim6 will enable me to create my ssh session over my wired connection, keep it there for an hour, I can then enable my wireless connection and shim6 will announce my new address to the other end. I can then pull out my wired connection and just be wireless, and still keep the TCP connection up and running. This is roaming, and users want it (hell, I want it). So this might be one way why the people developing shim6 doesn't seem to care about your views on the subject, because it doesn't only address the routing problem, it addresses other things as well. I guess I have to do a disclaimer that the above is what I understand shim6 to be able to do, but I might be mistaken, if so, flame away. The internet succeeded because of its end-to-end nature and freedom to create new applications that the network people didn't need to bother their heads with. Shim6 is the same thing, it will happen whether we want it or not. If users find it useful, it will florish. Better to be open than to try to stop it just because it doesn't fit into the model of today. Also on the pricing issue, there is already a huge pricing difference between units that'll do 8k LPM routes and 1M LPM routes, I imagine the difference in 5 years will be the same between 64k LPM routes and 8M routes, I'd rather pay the lower price if the 64k LPM route unit because we didn't need to scale address space to every enterprise that want to multihome. -- Mikael Abrahamsson email: swmike@swm.pp.se
participants (28)
-
Alexei Roudnev -
Andre Oppermann -
Chip Mefford -
Chris Adams -
Christian Kuhtz -
Daniel Roesen -
David Meyer -
Edward B. DREGER -
Ejay Hire -
Fred Baker -
Iljitsch van Beijnum -
James R. Cutler -
Joe Abley -
Joe Provo -
John A. Kilpatrick -
John Payne -
Kurt Erik Lindqvist -
Marshall Eubanks -
Matthew Kaufman -
Michael.Dillon＠btradianz.com -
Mikael Abrahamsson -
Mike Leber -
Paul Jakma -
Per Heldal -
Tony Finch -
Tony Hain -
Vince Fuller -
william(at)elan.net