Dave, I think you still are not understanding my point. NAP's can't do everything for everyone. You have to pick what's important and optimize for that. You can't do everything well. Here's a couple of other things we thought about. FDDI NAP - good short term prospects, tried and true technology. Capa city even more of a problem than SMDS, won't even support a 155 Mb/s vBNS connection. Switched FDDI over ATM helps the bandwidth problem a bit, but I don't really understand how that will extend to a large-scale ne twork. 2 things. FDDI is available on almost everyone's router hardware. It's mature and stable. You can deploy a concentrator based solution, then if capacity gets tight, you install a DEC gigaswitch. This is also a mature product, and has good congested state behavior (degrades gracefully). This sort of switched FDDI solution should provide reasonable capacity at the outset. Next, the IMPORTANT thing to realize is that this transition has to go well. You are talking about the entire operational core of the Internet being changed out. It's better to make things easier up front, get moved to a new architecture while protecting existing service. That means you have to make absolutely sure that reliability is the #1 criteria for an interconnect. The LAN-emulation folks in the Forum have lots of work yet to do with large LAN emulation networks. The equipment for LAN emulation over ATM is i n the same stage of "immaturity" as the IP/ATM equipment. Bletch. Multimedia bridging isn't the answer. The network management issues here are atrocious to deal with. Multimedia NAP - My best reply to this is Peter Ford's statement, "NAP s Don't Route". They're a layer two service. Some layer 2 interworking may be possible in the future, and we're looking at FR<->ATM interworking boxes right now. Agreed. You shouldn't route. And I think multimedia bridging is a very bad idea, at least for this use. I shudder to think about debugging problems because of different MTU settings in different media types. Cheap, low speed connections to the NAP - We got squeezed by product availability here. We know that there is a bigger market in selling T 1 speed connections to the NAPs than T3 and above. However, we had a requirem ent to support multiple T3's initially (remember that the minimum purpose of the NAPs is to support the existing NSFNET backbone for transition purpose s, the NSPs who are providing access to RNP's and who have a requirement to c onnect to all of the priority NAPs, and the vBNS). FDDI and SMDS weren't going to survive for long under this type of load, but ATM doesn't yet have T1 capabili ty (although it's coming). So, we proposed ATM, starting at 45 and 155 M b/s initially, and extending to 1.5 Mb/s and 622 Mb/s as switching equipme nt becomes available. Believe me, we'll be offering T1 speed connections to the SF and Chicago NAPs just as soon as we can make it work. Why offer T1 connections? The primary goal of the NAP's in the first phase of the transition is to support the replacement of 1 core NSP with a few NSP's with minimum disruption. THIS is what should be optimized for. Are you confident that all the management procedures and technologies we are deploying will support a gob of T1 NAP attached AS's? This is highly questionable in my mind, esp. with PVC's that have to be configured by hand. ATM immaturity - ATM itself isn't immature, it's the router and ADSU t echnology that's immature. We've been running ATM in our lab since (believe it or not) 1986, and switch vendors have been shipping equipment for about 3 year s now. I can't believe you said this!! The routers and ADSU's are what your customers care about. You are picking technology that's optimized for your own world view, not that of what your customers can best deal with. Also, most of the ATM switch vendors think they can get away with not learning from the lessons of the router people and build an ATM switch like a voice switch, with only enough buffering to support CBR type applications. When you push these switches even into light congestion, they start dropping cells like mad because there isn't enough buffering in them to allow the remote TCP's enough time to see the congestion building up and flow control their sending rate. This has to deal with some reasonable delay since most of these high speed streams won't be coming from geographically close to the NAP. And when you do drop cells, you won't drop all the cells out of a frame, you'll deliver 99% of them, and all that data will be tossed by the endpoint when it tries to reassemble the frame. Alan Romanow and Sally Floyd did a nice paper on this problem, and proposed a workaround, which no ATM switch vendor has implemented. So whatever problems you have from the previous problem, it's exacerbated by this issue. It's really good you have all that bandwidth to deliver these dead cells. Oh, that's right, your customers are paying for this capacity right? I'm sure your cost scheme gives people credits for incompletely delivered frames, right? We're beginning to see second generation products on the market right now. Since the Toronto IETF, we've been running tests on the SF NAP configu ration, and the problems we've found (and fixed) have all been with the ADSU's or routers. We do have a working configuration for the 1490/AAL5 protoco l stack, and are running load tests on it now. I'd really like to know the exact hardware configurations that were tested and what loads you ran through them. Most of our experiences show bad things happen when you put things under stress. Bi-lateral/multi-lateral NAP traffic agreements - There's no rules her e at all. The agreements are completely up to the NSPs who connect to the NAP. A CIX-like arrangement where all parties agree to exchange traffic w/o settlements is just as possible as N**2 bi-lateral agreements with tra ffic sensitive settlements. The NAP managers and NSF have nothing to say h ere. The issue isn't the policy, the issue is whether or not this is expected to work well with manual configuration of all the VC's, and gobs of NAP attached AS's that you've sold all those T1 connections to. If there are 40 attachees, then do you expect that people are going to establish bilateral relationships with all 39 others? And that the RS will never send them a next hop address that they don't have a VC open to right? Seems to me this all gets more complicated with the larger number of attachees. As far as taking a long-range view on the Internet, I'll plead guilty. We're already seeing congestion problems on the Internet today, and that's e ven before we turn the video and audio applications completely loose. My concern is that if we don't get enough capacity into the Internet, and really soon, the growth rate is going to drop badly. I've lived on badly congested Internets (remember the 56 Kb/s NSFNET?), and it's not an experience I wish to repeat. And what happens if the system doen't work right on day 1? I wonder how long you guys will be given a chance to fix it. You guys are STILL not optimizing for the right things. That is the issue. Now, perhaps it might be useful to describe a little bit about some of these issues are dealt with by the FIXes. Maybe some of this experience is useful in this discussion. First off, the FIXes aren't designed to handle anyone attaching. Too many peers would be a problem for the peering approach used there. However, use of the RS could allieviate many of these problems, but you still have increased network management load from each party that attaches. Next, you start off with a colocated facility for the interconnect. You pick a standard mature media for the interconnect, and all the attachees routers have to use that interface. When you have to evolve to a new media, because of capacity, or some other issue, since the routers are colocated, attachees simply add a new interface to their routers, and then move to peering over the new media, while retaining the older media as backup. Historically, the FIXes started as ethernet based. All the routers there were plugged into a multiport transciever. As traffic grew and AS's added more trunk capacity to their FIX routers, the ethernet got congested, and so more capacity was required. So an FDDI concentrator was attached, and the folks who had the biggest load issues ADDED FDDI interfaces to their routers, and offloaded this traffic onto the ring. Noone deactivated their ethernet interfaces, which allowed the ethernet to be used for backup. Some FIX attachees are more than adequately served by the ethernet, and as long as some AS's are ethernet only connected, EVERYONE has to maintain their ethernet interfaces. Eventually, traffic will grow and the next step will likely be a GIGASWITCH. You could move everyone off the concentrator into the switch, or if everyone has FDDI interfaces available, then maybe you throw away the ethernet, and then have people add a new FDDI interface for the GIGASWITCH, and use the concentrator FDDI for backup. As this gets congested, you then might move to ATM (if reliable and multiple interfaces are available), and use the GIGASWITCH as backup. Etc... Key points: Colocation buys you a lot. It means that folks can support whatever long haul media and speed they like to connect to their router at the FIX. This means that some can connect at T1, or DS-3, or SMDS, or ATM, or whatever. It's THEIR choice, and this can change and evolve over time. Since this trunk is part of the attachees network, they use their own normal network management procedures that they would use to interconnect to a customer site. The interconnect isn't over a long haul link. This greatly reduces the network management requirements for the interconnect, now that the long haul link is connected at both ends by a single AS's routers. Next, since the routers are colocated, evolution can occur gradually. New attachment media can be supported with NO additional long haul trunking required. The ability to support multiple interfaces in the router provides for gradual transitions and evolution, as long as a base interconnect technology is used by everyone. Note that customers who need the extra capacity have the incentive to do the work to add interfaces, and the older technology used as a backup. And as people move off the older technology, capacity is reclaimed. And when everyone has moved to a new technology, then you can toss the old interconnect. This above point is very important. You shouldn't attempt to force people to use the end all technology solution on day one. This is what you are doing, and it's a broken approach, esp. given the importance of these attachements. What you need to do is to build in evolvability, because whatever answer people think is right today will be wrong. Things change too quickly for this. And that evolvability needs to occur with gradual transitions, transitions that can be accomodated by multiple interfaces. You are trying to optimize for everything and in the process, will end up with a design that doesn't do anything well. Toss out some of your goals, but NOT those of reliability, operational manageability (do you guys still not have any SNMP access to the switch's port statistics available to NAP attachees?), and use of mature technology which will be absolutely required to make this transition work well. Then reevaluate your design, and I think you and your customers will all be better off. Does this style of interconnect solve all problems? No. Can you reasonably expect such an colocated facility to house hundreds of attachees? No, but then it's not clear to me that the routing technology that we have will scale to this sort of scale either. So it's not clear to me why this should be a hard requirement. Are any of these advantages derived from the fact that it's a Fed interconnect? No, they apply to any interconnect that uses a similiar architecture, with an RS or not. Also, I should point out that the FIXes aren't in the profit making business. That means that noone is out there trying to get people to attach directly to FIXes when they could adequately be served by connecting to one of the attached NSP's. This naturally results in fewer connections rather than more. I have nothing against profitmaking of course, (I'm a Republican after all), but I think engineering and operational concerns need to take priority over getting the maximum number of people to attach to a given interconnect point. And in the end, if things don't work because you've oversold the system, people will find somewhere else to pass their traffic. This is too important an issue to not get done right. thanks, Milo PS Sorry for the long note.