June 2000 - NANOG - lists.nanog.org

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Richard A. Steenbergen 19 Jun '00

19 Jun '00

On Sat, 17 Jun 2000, RJ Atkinson wrote: > Sounds like a filter on which product one buys. :-) Based on those who don't support a non-standard extension? At any rate, people will buy them, and problems will ensue. :P > >That fact aside, > >several OS drivers for GigE NICs do not currently support jumbo frames. > > Which OSs don't yet support this ? Not OS, drivers. Pick your favorite OS with GigE support, grep jumbo the drivers section. In a few cases the unix drivers support jumbo frames and the reference vendor drivers do not, in a couple its the other way around. I see its getting better though, there is more support then there used to be the last time I looked. > >The point is that unless everyone makes these changes, any attempt to run > >a higher MTU along a non-supporting path without a reliable PMTU-D > >mechanism will result in bloody horror. > > For content replication, as differentiated from content distribution, > the larger MTU should be safe. A larger than 1518 MTU won't be > safe for content distribution anytime soon because the Cable Modem > standards specify a 1518 MTU and cable modems are a leading way > to provide high performance networking to homes. > > > > The real number ought to be >= 9180 IP bytes + Ethernet overhead, > > > so that hosts can do page-flipping and other optimisations. > > > Without Jumbo-grams, neither NT nor any common UNIX can run full > > > line-rate over TCP/IP with GigE today. With Jumbo-Grams, both > > > NT and UNIX can achieve 1 Gbps of throughput over TCP/IP/GigE. > > > >I've been able to get DAMN NEAR line rate with sendfile(2) zero-copy > >transfer, and some optimizations for the entire packet output path, using > >FreeBSD and some AMD K7 800s, without using jumbo frames. > > Curious. > sendfile(2) is using TCP ? Any TCP extensions present ? > This was off-the-shelf FreeBSD ? Which version ? > Which GigE card (NetGear GA-620 ?) ? Based off 4.0-STABLE, using some work I'm doing on the FreeBSD TCP/IP stack (mainly cleaner code and a few trivial optimizations at this point, nothing earth shattering), some additional optimizations and shortcuts through the stack based on IP Flow which I'm writing, using back to back NetGear GA620s, 512k send/recvspace buffers and a 1MB socket buffer, and a really quick in-kernel ack & discard in the receiving end. The last time I tried it with standard userland transfers was on back to back p3 500s which pulled about 450Mbps between a GA620 and an Intel GE. > >It would > >certainly help to be doing less packets/rateoftime though, as this is a > >(the?) major bottleneck. > > Packet processing overhead and memory manipulation are generally > the bottlenecks in hosts. There is substantial literature to > this effect. Isn't that the truth. I think of a lot of it is poorly optimized and well-planned code though. -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

4 3

Re: GigE performance on BSD
by Richard A. Steenbergen 18 Jun '00

18 Jun '00

On Sat, 17 Jun 2000, RJ Atkinson wrote: > > > >I've been able to get DAMN NEAR line rate with sendfile(2) zero-copy > > > >transfer, and some optimizations for the entire packet output path, using > > > >FreeBSD and some AMD K7 800s, without using jumbo frames. > > > > > > Curious. > > > sendfile(2) is using TCP ? Any TCP extensions present ? > > > This was off-the-shelf FreeBSD ? Which version ? > > > Which GigE card (NetGear GA-620 ?) ? > > > >Based off 4.0-STABLE, using some work I'm doing on the FreeBSD TCP/IP > >stack (mainly cleaner code and a few trivial optimizations at this point, > >nothing earth shattering), some additional optimizations and shortcuts > >through the stack based on IP Flow which I'm writing, using back to back > >NetGear GA620s, 512k send/recvspace buffers and a 1MB socket buffer, and a > >really quick in-kernel ack & discard in the receiving end. > > Ah, so this was neither an off-the-shelf software configuration > nor a normal user application. I see. > > >The last time I tried it with standard userland transfers was on back to > >back p3 500s which pulled about 450Mbps between a GA620 and an Intel GE. > > This is consistent with what I would have expected from a normal > user application on a modern UNIX over BSD Socket API for that speed > of Intel CPU. Folks using CPUs with correct byte-ordering (e.g. > non-DEC MIPS, SPARC) get about 1 Mbps/MHz under UNIX. Folks seem > to get about 0.8 Mbps/MHz under NT/Intel (assuming a ~1518 byte MTU). sendfile(2) is a current off-the-shelf method to improve things dramatically. The other question is what you do with the data once you receive it (in the example of an ftp, there is no equivilent recvfile(2) to do zero copy transfers). And of course, mbufs and checksums are the rest of the problem (for BSD derivatives, linux has its own very special brand of problems :P). -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

1 1

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Richard A. Steenbergen 18 Jun '00

18 Jun '00

On Sat, 17 Jun 2000, RJ Atkinson wrote: > At 22:46 16/06/00 , Richard A. Steenbergen wrote: > > The common number that all > >those using jumbo frames should support is 9000 bytes (not 9k aka 9216). > > Disagree, for the reasons described in RFC-1626. The IP MTU > described in RFC-1626 has a number of advantages for hosts > using TCP, whether or not NFS is in use. That came out wrong, my point was that the number on all jumbo frame implementations should be no lower then 9000 bytes, not that this is the best possible number. Optimizing for NFS seems to be one of the lowest considerations on the list of important things though. :P > >This would probably need to be a condition of using GigE at the NAP, > >in order to achieve any kind of common support, and you're eliminating > >some vendor gige support (assuming their network can even carry the 9k > >packets TO the nap without requiring even more work and possible > >problems). > > Virtually all GigE vendors already do support the ~9K size > I've been talking about. Some don't, we've already found a couple on this list. That fact aside, several OS drivers for GigE NICs do not currently support jumbo frames. > POS has a configurable MTU. One could easily reconfigure one's POS > interfaces to have a ~9K MTU. The point is that unless everyone makes these changes, any attempt to run a higher MTU along a non-supporting path without a reliable PMTU-D mechanism will result in bloody horror. > The real number ought to be >= 9180 IP bytes + Ethernet overhead, > so that hosts can do page-flipping and other optimisations. > Without Jumbo-grams, neither NT nor any common UNIX can run full > line-rate over TCP/IP with GigE today. With Jumbo-Grams, both > NT and UNIX can achieve 1 Gbps of throughput over TCP/IP/GigE. I've been able to get DAMN NEAR line rate with sendfile(2) zero-copy transfer, and some optimizations for the entire packet output path, using FreeBSD and some AMD K7 800s, without using jumbo frames. It would certainly help to be doing less packets/rateoftime though, as this is a (the?) major bottleneck. -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

2 1

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Paul Vixie 18 Jun '00

18 Jun '00

sean(a)donelan.com (Sean Donelan) writes: > I think you are making an argument by definition. ... As far as I knew I was following the definition used elsewhere in the thread. (If the distinction isn't valid than I oughtn't have bothered to reply.) -- Paul Vixie <vixie(a)mibh.net> SVP for Internet Services, MFNX M.I.B.H. Inc. is a subsidiary of Metromedia Fiber Network, Inc.

1 0

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Paul Vixie 18 Jun '00

18 Jun '00

mark(a)exodus.net (Mark Tripod) writes: > I'm not sure that I'd use AboveNet as the benchmark for the hosting > industry. I think I agree. Abovenet's ISX sites (while world class) are really just points of sale for the company's real product, which is connectivity. A fair number of Abovenet's customers purchase connectivity without also purchasing colocation -- and I can't think of any "hosting" company for which that would be true. (No "hosting" company of which I'm aware has its own global optical network.) -- Paul Vixie <vixie(a)mibh.net> SVP for Internet Services, MFNX M.I.B.H. Inc. is a subsidiary of Metromedia Fiber Network, Inc.

4 3

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Sean Donelan 17 Jun '00

17 Jun '00

I think you are making an argument by definition. No "hosting" company has a global optical backbone, because if it had a global optical backbone its not a hosting company. No tier 1 provider buys transit, because if it bought transit it wouldn't be a tier 1 provider. On Sat, 17 June 2000, Paul Vixie wrote: > sean(a)donelan.com (Sean Donelan) writes: > > I don't understand your distinction. Several companies sell both hosting > > services and network services. GlobalCenter/Crossing, Genuity/BBN and > > Worldcom/UUNET have large "global" optical networks and hosting divisions. > > Those companies also sell circuit-based services, i.e., services that do > not require hosting in order to accept delivery of them. Genuity's home > page mentions DSL, for example. These are fine companies, with fine > networks, but they are not "hosting" companies any more than making tanks > in WW2 made Chrysler into a weapons company or making keyboards and mice > makes Microsoft a hardware company. If it's just one thing they do, then > it doesn't make sense to label the company by it.

1 0

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Paul Vixie 17 Jun '00

17 Jun '00

sean(a)donelan.com (Sean Donelan) writes: > I don't understand your distinction. Several companies sell both hosting > services and network services. GlobalCenter/Crossing, Genuity/BBN and > Worldcom/UUNET have large "global" optical networks and hosting divisions. Those companies also sell circuit-based services, i.e., services that do not require hosting in order to accept delivery of them. Genuity's home page mentions DSL, for example. These are fine companies, with fine networks, but they are not "hosting" companies any more than making tanks in WW2 made Chrysler into a weapons company or making keyboards and mice makes Microsoft a hardware company. If it's just one thing they do, then it doesn't make sense to label the company by it. -- Paul Vixie <vixie(a)mibh.net> SVP for Internet Services, MFNX M.I.B.H. Inc. is a subsidiary of Metromedia Fiber Network, Inc.

2 1

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Sean Donelan 17 Jun '00

17 Jun '00

On Sat, 17 June 2000, Paul Vixie wrote: > (No "hosting" company of which I'm aware has its own global optical network.) I don't understand your distinction. Several companies sell both hosting services and network services. GlobalCenter/Crossing, Genuity/BBN and Worldcom/UUNET have large "global" optical networks and hosting divisions.

1 0

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Richard A. Steenbergen 17 Jun '00

17 Jun '00

On Fri, 16 Jun 2000, Ted Schroeder wrote: > As one of the early evangelists for jumbo frames I can say with > some amount of authority that the number we (Alteon WebSystems) > were/are pushing for is 9000 bytes. There are other numbers bandied > about, but 8940 was never one of them. 9216 was the other usual > candidate as it matches up with ATM. Come to think of it I can't see any good reason for 8940. FDDI is 4352, FDDI*2 is 8704. 9216 should behave nicely w/AAL5. > The IEEE will simply say, "only up to 1500 bytes". I've fought this > battle with very little success. Rich's book will only talk about 1500 > bytes. I've spoken to him about this a number of times and he's > personally opposed to anything other than 1500 bytes. I know there is a discussion of jumbo frames there, and its not terribly negative from what I can recall. > And, for those of you who think that things might get better in the future, > let me be an omen for even worse times to come. For 10 Gb Ethernet, > there are proposals on the table that will, for the first time, make frames > larger than 1522 bytes physically impossible. These proposals have to > do with speed matching between the WAN 10GE and the LAN 10GE > and minimum buffer sizes used to accomplish this speed matching. > > As one of the lone voices at the 802.3 meetings calling for longer frame > support, I've pretty much given up on ever seeing a longer MTU officially > supported by the IEEE. Perhaps if more people were going there that > weren't hardware vendors with an "obvious interest in only pushing > their products" the IEEE members would be more likely to listen. But > I wouldn't necessarily bet on it. There was a push early on during PAR > discussions of 10 GE to allow larger frames but it was clear that this > was a losing battle, so the push ended. Ugh, from an all around perspective I believe we need an MTU offically longer then 15??. Its better for applications, better for operating systems, better for anything that must be interrupted per frame/packet, better for anything which needs to work paged memory, and better for anything which needs to do routing per packet. 9000-range works well for server to server or backbone to backbone connections where you're aiming for NFS optimization or being able to support large packets without fragmentation across a NAP for example, but I would suggest you're aiming for trouble trying to take server-traffic past what is supported by FDDI/PoS, and 4k works well for many of the above problems. The standardization of a higher number is needed, or we are just going to see more of these problems as people seek to improve their networks in incompatible and confusion-introducing ways. PMTU-D has obvious problems. It would seem to me the cleanest course of action would be to put a fragmentation encountered flag somewhere along the line. -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

2 1

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Richard A. Steenbergen 17 Jun '00

17 Jun '00

On 16 Jun 2000 michael.dillon(a)gtsip.net wrote: > On Fri, 16 June 2000, "Richard A. Steenbergen" wrote: > > >In a public exchange point where you're talking to multiple networks > >across a shared media it makes sense to do GigE, multiple GigE, 10GigE, > >etc, > > What is the largest MTU that you can use across a GigE fabric? I think this would be extremely foolhardy. The common number that all those using jumbo frames should support is 9000 bytes (not 9k aka 9216). This would probably need to be a condition of using GigE at the NAP, in order to achieve any kind of common support, and you're eliminating some vendor gige support (assuming their network can even carry the 9k packets TO the nap without requiring even more work and possible problems). Has anyone tested these maximium sized jumbo frames with 802.1q vlan tagging to see exactly how much (or how little) inter-vendor support really exists? -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

2 2