- Test - lists.nanog.org

Re: Jumbo Frames (was Re: MAE-EAST Moving? from Tysons corner to reston VA. )
by Richard A. Steenbergen 19 Jun '00

19 Jun '00

> On Mon, 19 June 2000, "Bora Akyol" wrote: > > > As long as most end users are running Ethernet, Fast Ethernet, DSL or Cable > > Modems, what is the point of jumbo frames/packets other than transferring > > BGP tables really fast. Did any one look into how many packets are moved > > through an OC-48 in 1 seconds. (approx. 6 million 40 byte packets). I think > > even without jumbo frames, this bandwidth will saturate most CPUs. > > > > Jumbo frames are pointless until most of the Internet end users switch to a > > jumbo frame based media. > > > > Yes, they look cool on the feature list (we support it as well). Yes they > > are marginally more efficient than 1500 byte MTUs ( 40/1500 vs 40/9000). But > > in reality, 99% or more of the traffic out there is less than 1500 bytes. In > > terms of packet counts, last time I looked at one, 50% of the packets were > > around 40 byte packets (ACKs) with another 40% or so at approx 576 bytes or > > so. > > > > What is the big, clear advantage of supporting jumbo frames? > > When 1500 byte frames from the customer's LAN enter the customer's > router and enter some form of IP tunnel, then a core fabric which > supports larger than 1500 byte frames will not cause fragmentation. > It's not necessary to do the full jumbo size frames. I suspect that > supporting two levels of encapsulation will be enough in 99.9% of the > cases. For the sake of argument, what would be the downside of using a > 2000 byte MTU as the minimum MTU in your core? When the next end user upgrade is deployed and everyone has devices which can support larger MTUs, wouldn't it be a shame if they said "if only the internet core ran at larger MTUs, we could negiotate higher MTUs and make everyone happier". Also, it is far more then "marginally more efficient". For every packet you deal with, there is a great amount of work doing routing lookups, dealing with memory management, and handling interrupts. Copying another few bytes of data is easy in comparison. Since we are asking GigE to act in a server and backbone role, we should acknowledge that the requirements will be different from the average end-user ethernet. One of those requirements is that the backbone should be able to pass larger packets it may encounter without resorting to fragmentation (which only gets harder and we start getting into higher speeds). Aside from that, and the fact that there is nothing harmful in supporting larger packets through your network, there is the fact that if we want people to support standards we KNOW are good for them (even if they don't), we have to actually ask for it. Imagine an internet with a reliable MTU negiotation mechinism, which can take advantage of improved thruput, much lower CPU usage, zero copy, page-flipping, DMA transfers, and all those other lovely things. These are important for many reasons. Without these techniques, we can't even do line rate GigE on "common place" servers, let alone have any CPU left over to do more then just send packets. Its easy to just say "we'll throw a server farm at it" or "we'll just get a faster processor", but as higher speed links become more common place, and as GigE becomes common in servers (when servers can actually use it effectively) and 10GigE becomes commonplace for backbone links, we'll start to see these things matter. Why engineer ourselves into a corner of shortsightedness which only gets harder and harder to fix, because its "easier" to do nothing? (sorry Michael, just using your msg as a good point to reply :P) -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

4 3

Where in the world is ?
by Sean Donelan 19 Jun '00

19 Jun '00

> Sean Donelan was not heard from, and we're all hoping the power gnomes > didn't get him :) California should be safe this week, I'm staying at CNN Center in Atlanta. Of course, if CNN goes off the air, you'll know why :-) Is the CNN tour worth the 45 minutes? With brewery tours, at least you get a sample of the product at the end. In unrelated news, the State Department lost all its phones on Thursday including phones in its operation center. An earlier fire damaged its backup batteries, and when rain knocked out utility power, no phones.

3 2

Private and Public Peering
by HANSEN CHAN 19 Jun '00

19 Jun '00

Hi, Can someone tell me the difference between private and public peering? A peering paper I have read explained the switch is shared in the case of shared and non-shared in the case of private. I still do not understand. Appreciate if someone could shed some light on this matter for me. My guess is that public peering is done in a commerical internet exchange while private peering is done in a facility owned by either one of the ISP. Thanks, Hansen

3 2

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by michael.dillon＠gtsip.net 19 Jun '00

19 Jun '00

On Sat, 17 June 2000, "Richard A. Steenbergen" wrote: > > What is the largest MTU that you can use across a GigE fabric? > > I think this would be extremely foolhardy. The common number that all > those using jumbo frames should support is 9000 bytes (not 9k aka 9216). > This would probably need to be a condition of using GigE at the NAP, > in order to achieve any kind of common support, and you're eliminating > some vendor gige support (assuming their network can even carry the 9k > packets TO the nap without requiring even more work and possible > problems). I really should have worded that question more carefully. What I meant was, what is the maximum MTU that is supported by all GigE manufacturers and thus can be guaranteed to work without fragging across any arbitrary GigE fabric? I was under the impression that the answer is 1500 because not all vendors support larger MTUs. And the implication is that GigE (in general) is not suitable for a public exchange fabric because it will introduce fragging in packets which are encapsulated (tunnelled). Of course, it would be possible to choose GigE equipment carefully and to configure it correctly to avoid this downside. But it seems to me that it is essential for any exchange point fabric or private peering link or IP backbone to support MTUs larger than 1500 edge to edge. --- Michael Dillon Phone: +44 (20) 7769 8489 Mobile: +44 (79) 7099 2658 Director of Product Engineering, GTS IP Services 151 Shaftesbury Ave. London WC2H 8AL UK

1 0

Re: Jumbo Frames (was Re: MAE-EAST Moving? from Tysons corner to reston VA. )
by Vadim Antonov 19 Jun '00

19 Jun '00

Bora Akyol <akyol(a)akyol.org> wrote: > What is the big, clear advantage of supporting jumbo frames? There are none, not without doing something about TCP congestion control. The bigger frames are, the more coarse the congestion control is, and the flows are short enough on average to make it barely effectual. Increasing frame size to 10kb makes most flows "mice" flows, which kills the slow-start algorithm. Of course, flows may start by sending smaller frames first, but, again, what's the point? With 1500 byte frames, the framing overhead is about 2.8%, however, given that the 40% of packets are only 40-41 bytes long, this seems to be a very minor effect. --vadim

1 0

Re: PMTU-D: remember, your load balancer is broken
by Paul Vixie 19 Jun '00

19 Jun '00

Valdis.Kletnieks(a)vt.edu writes: > Has this changed? Has "fragmentation" become a Great Evil, ... ? Yes. http://research.compaq.com/wrl/techreports/abstracts/87.3.html says: Research Report 87/3, December 1987 87.3 -- Fragmentation Considered Harmful Christopher A. Kent, Jeffrey C. Mogul PostScript Version gzipped PostScript Version Adobe Acrobat PDF Version Internetworks can be built from many different kinds of networks, with varying limits on maximum packet size. Throughput is usually maximized when the largest possible packet is sent; unfortunately, some routes can carry only very small packets. The IP protocol allows a gateway to fragment a packet if it is too large to be transmitted. Fragmentation is at best a necessary evil; it can lead to poor performance or complete communication failure. There are a variety of ways to reduce the likelihood of fragmentation; some can be incorporated into existing IP implementations without changes in protocol specifications. Others require new protocols, or modifications to existing protocols. wrl-techops(a)pa.dec.com Copyright © 2000 Compaq Computer Corporation I was there, I saw the research that went into this, I know the guys who did the work, and I agree completely with the conclusions thus presented.

3 2

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Richard A. Steenbergen 19 Jun '00

19 Jun '00

On Sat, 17 Jun 2000, RJ Atkinson wrote: > Sounds like a filter on which product one buys. :-) Based on those who don't support a non-standard extension? At any rate, people will buy them, and problems will ensue. :P > >That fact aside, > >several OS drivers for GigE NICs do not currently support jumbo frames. > > Which OSs don't yet support this ? Not OS, drivers. Pick your favorite OS with GigE support, grep jumbo the drivers section. In a few cases the unix drivers support jumbo frames and the reference vendor drivers do not, in a couple its the other way around. I see its getting better though, there is more support then there used to be the last time I looked. > >The point is that unless everyone makes these changes, any attempt to run > >a higher MTU along a non-supporting path without a reliable PMTU-D > >mechanism will result in bloody horror. > > For content replication, as differentiated from content distribution, > the larger MTU should be safe. A larger than 1518 MTU won't be > safe for content distribution anytime soon because the Cable Modem > standards specify a 1518 MTU and cable modems are a leading way > to provide high performance networking to homes. > > > > The real number ought to be >= 9180 IP bytes + Ethernet overhead, > > > so that hosts can do page-flipping and other optimisations. > > > Without Jumbo-grams, neither NT nor any common UNIX can run full > > > line-rate over TCP/IP with GigE today. With Jumbo-Grams, both > > > NT and UNIX can achieve 1 Gbps of throughput over TCP/IP/GigE. > > > >I've been able to get DAMN NEAR line rate with sendfile(2) zero-copy > >transfer, and some optimizations for the entire packet output path, using > >FreeBSD and some AMD K7 800s, without using jumbo frames. > > Curious. > sendfile(2) is using TCP ? Any TCP extensions present ? > This was off-the-shelf FreeBSD ? Which version ? > Which GigE card (NetGear GA-620 ?) ? Based off 4.0-STABLE, using some work I'm doing on the FreeBSD TCP/IP stack (mainly cleaner code and a few trivial optimizations at this point, nothing earth shattering), some additional optimizations and shortcuts through the stack based on IP Flow which I'm writing, using back to back NetGear GA620s, 512k send/recvspace buffers and a 1MB socket buffer, and a really quick in-kernel ack & discard in the receiving end. The last time I tried it with standard userland transfers was on back to back p3 500s which pulled about 450Mbps between a GA620 and an Intel GE. > >It would > >certainly help to be doing less packets/rateoftime though, as this is a > >(the?) major bottleneck. > > Packet processing overhead and memory manipulation are generally > the bottlenecks in hosts. There is substantial literature to > this effect. Isn't that the truth. I think of a lot of it is poorly optimized and well-planned code though. -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

4 3

Re: GigE performance on BSD
by Richard A. Steenbergen 18 Jun '00

18 Jun '00

On Sat, 17 Jun 2000, RJ Atkinson wrote: > > > >I've been able to get DAMN NEAR line rate with sendfile(2) zero-copy > > > >transfer, and some optimizations for the entire packet output path, using > > > >FreeBSD and some AMD K7 800s, without using jumbo frames. > > > > > > Curious. > > > sendfile(2) is using TCP ? Any TCP extensions present ? > > > This was off-the-shelf FreeBSD ? Which version ? > > > Which GigE card (NetGear GA-620 ?) ? > > > >Based off 4.0-STABLE, using some work I'm doing on the FreeBSD TCP/IP > >stack (mainly cleaner code and a few trivial optimizations at this point, > >nothing earth shattering), some additional optimizations and shortcuts > >through the stack based on IP Flow which I'm writing, using back to back > >NetGear GA620s, 512k send/recvspace buffers and a 1MB socket buffer, and a > >really quick in-kernel ack & discard in the receiving end. > > Ah, so this was neither an off-the-shelf software configuration > nor a normal user application. I see. > > >The last time I tried it with standard userland transfers was on back to > >back p3 500s which pulled about 450Mbps between a GA620 and an Intel GE. > > This is consistent with what I would have expected from a normal > user application on a modern UNIX over BSD Socket API for that speed > of Intel CPU. Folks using CPUs with correct byte-ordering (e.g. > non-DEC MIPS, SPARC) get about 1 Mbps/MHz under UNIX. Folks seem > to get about 0.8 Mbps/MHz under NT/Intel (assuming a ~1518 byte MTU). sendfile(2) is a current off-the-shelf method to improve things dramatically. The other question is what you do with the data once you receive it (in the example of an ftp, there is no equivilent recvfile(2) to do zero copy transfers). And of course, mbufs and checksums are the rest of the problem (for BSD derivatives, linux has its own very special brand of problems :P). -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

1 1

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Richard A. Steenbergen 18 Jun '00

18 Jun '00

On Sat, 17 Jun 2000, RJ Atkinson wrote: > At 22:46 16/06/00 , Richard A. Steenbergen wrote: > > The common number that all > >those using jumbo frames should support is 9000 bytes (not 9k aka 9216). > > Disagree, for the reasons described in RFC-1626. The IP MTU > described in RFC-1626 has a number of advantages for hosts > using TCP, whether or not NFS is in use. That came out wrong, my point was that the number on all jumbo frame implementations should be no lower then 9000 bytes, not that this is the best possible number. Optimizing for NFS seems to be one of the lowest considerations on the list of important things though. :P > >This would probably need to be a condition of using GigE at the NAP, > >in order to achieve any kind of common support, and you're eliminating > >some vendor gige support (assuming their network can even carry the 9k > >packets TO the nap without requiring even more work and possible > >problems). > > Virtually all GigE vendors already do support the ~9K size > I've been talking about. Some don't, we've already found a couple on this list. That fact aside, several OS drivers for GigE NICs do not currently support jumbo frames. > POS has a configurable MTU. One could easily reconfigure one's POS > interfaces to have a ~9K MTU. The point is that unless everyone makes these changes, any attempt to run a higher MTU along a non-supporting path without a reliable PMTU-D mechanism will result in bloody horror. > The real number ought to be >= 9180 IP bytes + Ethernet overhead, > so that hosts can do page-flipping and other optimisations. > Without Jumbo-grams, neither NT nor any common UNIX can run full > line-rate over TCP/IP with GigE today. With Jumbo-Grams, both > NT and UNIX can achieve 1 Gbps of throughput over TCP/IP/GigE. I've been able to get DAMN NEAR line rate with sendfile(2) zero-copy transfer, and some optimizations for the entire packet output path, using FreeBSD and some AMD K7 800s, without using jumbo frames. It would certainly help to be doing less packets/rateoftime though, as this is a (the?) major bottleneck. -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

2 1

Re: MAE-EAST Moving? from Tysons corner to reston VA.
by Paul Vixie 18 Jun '00

18 Jun '00

sean(a)donelan.com (Sean Donelan) writes: > I think you are making an argument by definition. ... As far as I knew I was following the definition used elsewhere in the thread. (If the distinction isn't valid than I oughtn't have bothered to reply.) -- Paul Vixie <vixie(a)mibh.net> SVP for Internet Services, MFNX M.I.B.H. Inc. is a subsidiary of Metromedia Fiber Network, Inc.

1 0