re: AGIS Route Flaps Interrupting its Peering?
| These NetEdges seem to have three different possible operating states: | completely working (which doesn't happen often enough); broken (often, right | out of the box); and kind of working (which happens all too often). That these things work at all under load is nothing short of miraculous. There are plenty of mixed-media-bridging devices that have been tried and which have failed miserably in the past, most notably: -- Magnum 100s (ethernet<->funny framing<->ethernet), which don't meet the standard IFG specification; jam too many back-to-back packets at them, and it will lose most of a burst of traffic. This hurt the old MAE-EAST. -- various ADSUs (FR<->ATM<->FR), which lack buffering and perform SAR too slowly, which delayed the PAC*Bell and Ameritech NAPs for months and some of which still exhibit flakiness under load -- a horrible idea (ethernet<->MFS ATM<->ethernet) which didn't work under load; perhaps a veteran user of these neato little things could explain the failure mode to anyone interested. I remember one service provider who had a national "10Mbps" ethernet backbone who had various horrible problems including the breakdown of the LIS (such that some routers couldn't talk to others), connections going simplex, frame loss and other wonderful things -- another horrible idea (FR<->MFS ATM<->FR) this was pretty neat; some of the problems above plus a brand new problem: the ADSU would strip the FR frame checksum, perform SAR, send the cells, and the ADSU on the opposite end would reassemble the frame and produce a correct FR checksum. All fine and dandy, unless cells arrived out of order, SAR was done wrong, or there was data corruption under load in the DSU or in the network. This interesting technology advanced the state of IS:IS in one vendor's software rather considerably -- mixed-media bridging (NetEdges, FDDI/Ethernet bridges) these break in all sorts of interesting ways. In particular, NetEdges have an annoying habit of confusing FDDI stations in particularly toxic ways, and some FDDI/Ethernet bridges resemble roach motels: frames check in but they don't check out. Essentially, most of these things worked mostly perfectly under low load, but when faced with the kind of traffic one sees at a busy exchange point, most bridging technology has failed in really awkward ways. My advice is that if you can avoid talking to something across a bridge at an exchange point, you should do so. The keep-it-simple principle is a bunch more expensive, but probably not as expensive as a very public failure. Finally, why is it that most vendors never test their products in a serious battlefield environment like an ISP of size medium to huge? These places tend to be excellent worst-case testing grounds. Sean.
Hear, hear! Bridging is a bad idea, whether for LAN extension or home ISDN users or simulation of point to point links in WANs. I have one additional (minor) data point to add to Sean's message:
and some FDDI/Ethernet bridges resemble roach motels: frames check in but they don't check out.
I have a DEC PEswitch-900 in my own little "hub" hanging off of DEC's Palo Alto exchange. I have absolutely no complaints about this box; I have pushed 50Mb/s through it (it has a FDDI and 6 10BaseT's) without dropping anything (all my TCP timers on the test hosts remained unchanged, and nothing was lost or reordered on minimum spaced back to back UDP spams, either). My PEswitch-900 is the "Personal Ethernet" version of this product; it only allows a small number (8, perhaps?) of end stations per 10BaseT port. So if you were gluing together a bunch of moderately sized LANs, you'd need the more expensive version (probably called a DECswitch-900-EF but don't take my word for it if details matter). On the other hand, the only reasonable (IMHO) way to use one of these is with one host per port.
-- a horrible idea (ethernet<->MFS ATM<->ethernet) which didn't work under load; perhaps a veteran user of these neato little things could explain the failure mode to anyone interested. I remember one service provider who had a national "10Mbps" ethernet backbone who had various horrible problems including the breakdown of the LIS (such that some routers couldn't talk to others), connections going simplex, frame loss and other wonderful things
This actually works reasonably well as long as you don't try to cram 20Mbps of data through it in aggregate. The problem that one particular provider had was trying to do this full-mesh AND ignoring the fact that Ethernet isn't really 10Mbps; its more like 5-6Mbps when you take the duplexing issues into account. Now add the collision domain problems to this (you really don't have a 3000-mile collision domain; its "faked" in the translation) and limited buffering and you can see where some problems show up really quickly, especially when you sell resale T1s to people and don't upgrade the backbone to meet the increasing sales of circuits...... If you only mesh the places you need to talk to from one point to another and use a better exit technology at exchange points (allowing an aggregate data rate > 10mbps) it works quite well. Try to go "cheap" and use the AUI interface everywhere and you get a different result.
-- another horrible idea (FR<->MFS ATM<->FR) this was pretty neat; some of the problems above plus a brand new problem: the ADSU would strip the FR frame checksum
(don't use ADSUs; that will help a lot :-)
-- mixed-media bridging (NetEdges, FDDI/Ethernet bridges)
Netedge <> Ethernet works quite well (far better than RETIX<>Ethernet.) Netedge<>FDDI has been reported to have the problems you listed, and more.
Finally, why is it that most vendors never test their products in a serious battlefield environment like an ISP of size medium to huge? These places tend to be excellent worst-case testing grounds.
That's a good question...
Sean.
-- -- Karl Denninger (karl@MCS.Net)| MCSNet - The Finest Internet Connectivity Modem: [+1 312 803-3590] | T1 from $600 monthly; speeds to DS-3 available Voice: [+1 312 803-MCS1] | 23 Chicagoland Prefixes, 13 ISDN, much more Fax: [+1 312 248-9865] | Email to "info@mcs.net" WWW: http://www.mcs.net/ ISDN - Get it here TODAY! | Home of Chicago's only FULL Clarinet feed!
This actually works reasonably well as long as you don't try to cram 20Mbps of data through it in aggregate.
The problem that one particular provider had was trying to do this full-mesh AND ignoring the fact that Ethernet isn't really 10Mbps; its more like 5-6Mbps when you take the duplexing issues into account. Now add the collision domain problems to this (you really don't have a 3000-mile collision domain; its "faked" in the translation) and limited buffering and you can see where some problems show up really quickly, especially when you sell resale T1s to people and don't upgrade the backbone to meet the increasing sales of circuits......
If you only mesh the places you need to talk to from one point to another and use a better exit technology at exchange points (allowing an aggregate data rate > 10mbps) it works quite well. Try to go "cheap" and use the AUI interface everywhere and you get a different result.
And I know a provider that despite realistic expectations of the technology, still experience serious problems with everything that Sean mentioned. 50% of all problems were caused by inadequate buffering, and the other 50% were caused by PVC's either dissappearing, or being re-routed around the wrong end of the country. (Gee, 250ms from San Francisco to San Jose...it's taking the east coast route again...)
-- mixed-media bridging (NetEdges, FDDI/Ethernet bridges)
Netedge <> Ethernet works quite well (far better than RETIX<>Ethernet.)
No argument on the Retix's.
Netedge<>FDDI has been reported to have the problems you listed, and more.
not quite there yet, no.
Finally, why is it that most vendors never test their products in a serious battlefield environment like an ISP of size medium to huge? These places tend to be excellent worst-case testing grounds.
That's a good question...
What are you talking about? They do a series of stress tests designed to push the equipment to it's utmost capabilities in their state-of-the-art ...laboratory. heh. Dave -- Dave Siegel Sr. Network Engineer, RTD Systems & Networking (520)623-9663 x130 Network Consultant -- Regional/National NSPs dsiegel@rtd.com User Tracking & Acctg -- "Written by an ISP, http://www.rtd.com/~dsiegel/ for an ISP."
participants (4)
-
Dave Siegel
-
Karl Denninger, MCSNet
-
Paul A Vixie
-
Sean Doran