On Wed, 4 Feb 1998, Perry E. Metzger wrote:

...

Several people have noted to the Microsoft Support and Product groups that they want the Windows 95 PPP MTU to be set to 576 (down from 1500). this change is in Windows 98.

<snip>

accept a large MTU, no matter what Windows would like. The entire story sounds, to say the least, fishy.

I think what is really going on is that people tweaking on the MTU setting have discovered that for some unknown reason 576 just plain works better over a dialup PPP connection than ~1500 or any other value for that matter. My guess would be that it is in some way related to the packet latency generated by clocking in 1500 bytes over a ppp link (~500 ms PLUS the V.whatever overhead) A ~500 byte packet would be more like ~166 ms. I just did a real-world check of these numbers, and they seem pretty close to reality for a 28.8 connection. They're a little high for a 33.6. (real 33.6 numbers were 250 (total clock+v.?) and 412 for 500 and 1500 respectively) Now it's been a while since I looked at latency vs transfer rates, so maybe someone who works on this on an everyday basis would like to comment on what ~200 more ms of latency on a 28.8 link would do to throughput end-to-end across the net (totals of something like 350 and 512 ms end-to-end). I'm pretty certain though, that the common "myth" that 576 works better because of end-to-end MTU's is just that - a myth. My network is all 1500 or better - not sure if I've EVER seen anything less than 1500 or not in the last few years, but I doubt it. - Forrest W. Christian (forrestc@imach.com) ---------------------------------------------------------------------- iMach, Ltd., P.O. Box 5749, Helena, MT 59604 http://www.imach.com Solutions for your high-tech problems. (406)-442-6648 ----------------------------------------------------------------------

Could be related to dialup people wanting lower latencies in multiplayer games. Dirk On Wed, Feb 04, 1998 at 12:05:29PM -0500, Perry E. Metzger wrote:

Peter Ford writes:

...

Several people have noted to the Microsoft Support and Product groups that they want the Windows 95 PPP MTU to be set to 576 (down from 1500). this change is in Windows 98.

The reason for this change cited by many customers is that many ISPs have 576 MTUs set "inside" their networks and packets get fragmented.

I know of no modern networking media in common use at ISPs that has an MTU below 1000, and have not heard of ISPs with MTUs that low. Also, the ISP's PPP could simply negotiate the MTU down -- it need not accept a large MTU, no matter what Windows would like. The entire story sounds, to say the least, fishy.

BTW, if Windows 95 is doing path MTU discovery, it should all be irrelevant.

Perry

Peter Ford writes:

...

Several people have noted to the Microsoft Support and Product groups that they want the Windows 95 PPP MTU to be set to 576 (down from 1500). this change is in Windows 98.

The reason for this change cited by many customers is that many ISPs have 576 MTUs set "inside" their networks and packets get fragmented.

I know of no modern networking media in common use at ISPs that has an MTU below 1000, and have not heard of ISPs with MTUs that low.

I know of none either, but perhaps someone from AOL/Compuserv could comment on their architecture. Some of those older or original custom networks may have different configurations. If memory serves, John Goltz told me that he wrote his own network protocol for the Compuserv servers to interact with each other. No idea if this was ever a part of compuserv's Internet service... It's worth remembering that it only takes one extremely large nationally visible provider like AOL to say to Microsoft "ISPs like us use 576 size MTU's, so fix win95". Dave -- Dave Siegel dave@rtd.net Network Engineer dave@pager.rtd.com (alpha pager) (520)579-0450 (home office) http://www.rtd.com/~dsiegel/

On Wed, Feb 04, 1998 at 01:17:05PM -0500, Robert E. Seastrom wrote:

Unless your ISP uses BBN Butterflies and C30 IMPs in its backbone, I would discount the odds of running into a link with an MTU of 576.

How do I program my router to emulate one of those? :-) Cheers,-- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "Two words: Darth Doogie." -- Jason Colby, Tampa Bay, Florida on alt.fan.heinlein +1 813 790 7592 Managing Editor, Top Of The Key sports e-zine ------------ http://www.totk.com

Fletcher E Kittredge wrote:

BBN Butterflys were one of the worst pieces of hardware ever built.

Pardon my ruffled feathers, but IMHO this is not so! The Butterfly was a fine example of a massively parallel computer. It was, unfortunately, seldom deployed with the full complement of 128 processors. The only problem with the architecture was the "flexible-bus" technology which made maintenance a bit unweildy. BTW - I believe that the C/30 probably has the best uptime record of any packet processor. Many are documented as up 100% for years. Anybody who has been online for more than 5 years probably had their packets passed by them. <off my soapbox and back into my cave> -- David J. Bowie GTE Internetworking Powered by BBN

On Thu, 5 Feb 1998 11:28:04 -0500 (EST) Joseph Malcolm wrote:

...

Pardon my ruffled feathers, but IMHO this is not so! The Butterfly was a fine example of a massively parallel computer. It was, unfortunately, seldom deployed with the full complement of 128 processors.

Well, the Butterfly was used in many capacities. As a communications processor, it stunk and never was deployed in production with more than 4 processors (or am I wrong?). That is why BBN is dead, and Cisco is a 4 billion dollar per year company. BBN used to totally dominate the router business until they started in with POS like the Butterfly. Didn't sell many of them as massively parallel systems either. regards, fletcher (ex. fkittred@bbn.com, fkittred@das.harvard.edu)

...

The reason for this change cited by many customers is that many ISPs have 576 MTUs set "inside" their networks and packets get fragmented.

I really don't buy that. Many or most backbone links have MTU >1500, and MTUs <1500 outside of low-speed dialup connections aren't that common. They are there, yes. But not that common.

My understanding of why a lower MTU is demonstratable better under Win95 is because the Win95 TCP stack is broken, and it is a good workaround. Yep. And moreover; we mentioned the problem for the customers who use WIN95 and try to get information from some WIN-NT servers, in case if there is low-MTU (576) links somewhere between this client and server. I suppose Win95 TCP/IP stack is not implemented correctly for this issue.

There are all sorts of people spouting all sorts of lies around Windows newsgroups about why small MTUs are good; I think novice users are simply getting drawn in by supposed experts. In theory, small MUS and priority queuering can make delays less; on practice (and if you remember about crasy and broken Win95 stack) the best choise is to use MTU 1500 everywhere when some Win95 customers exist.

Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)

On Wed, 4 Feb 1998, Phil Howard wrote:

Marc Slemko fills my screen with:

...

...

The reason for this change cited by many customers is that many ISPs have 576 MTUs set "inside" their networks and packets get fragmented.

I really don't buy that. Many or most backbone links have MTU >1500, and MTUs <1500 outside of low-speed dialup connections aren't that common. They are there, yes. But not that common.

My understanding of why a lower MTU is demonstratable better under Win95 is because the Win95 TCP stack is broken, and it is a good workaround. Most of the people raving about it are saying they are getting 2-4 times speed increases from changing their MTU from 1500 to 576. Something is very wrong there. I thought I had heard details about exactly what is broken in the Win95 TCP stack that causes this problem, but can't recall them at the moment. It could have no basis in reality and just be a rumour.

I have my MTU set to 552 and it helps quite a bit. It's not an issue of the Win95 stack being broken. I'm running Linux.

The reason I chose to use a low MTU, and sometimes I knock it down even further, is to be able to improve my telnet interactive response over a 33.6k link that I also run as many as 4 concurrent downloads on.

Interactive use is a different issue. A lot of people are claiming that lowering their MTU in Win95 improves their speeds by several times when downloading over a single TCP connection. There appears to be something else going on with Win95 users who have trouble with this.

Here's what I suspect is happening:

With web surfing, a page loads with many images, each of which is often larger than a sliding window worth of packets. The browser will nearly concurrently connect and request for every image. Thus for N images you now get N sliding windows worth of packets slammed at you. This takes up a _lot_ of buffer space in the dialup routers for all these concurrent TCP connections all sending data at the same time over a high speed net to a low speed final link.

I am doubtful. TCP's congestion control should deal with this reasonably. Interactive performance can certainly be helped with a lower MTU, but that is really another issue.

With this happening, buffer space is exhausted and packets are discarded.

If you set the MTU smaller, then the size of all those packets is smaller and the chance of being discarded due to memory exhaustion is reduced, even if you're the only one on that server with small packets.

Anything is possible. I am doubtful of that being the case. It is more likely that servers are retransmitting too quickly for the situation and causing excess traffic. This would be helped by a smaller MTU, yes.

...

There are all sorts of people spouting all sorts of lies around Windows newsgroups about why small MTUs are good; I think novice users are simply getting drawn in by supposed experts.

Such as?

That the reason why you need to lower the MTU is because if the packet is too big the modem will fragment it because the modem can only handle packets of a certain size.

I assert that small MTUs are good because in real life practice it actually does improve the effective responsiveness and reduces the lag causing loss of packets.

Of course of something is broken, it needs fixing.

Like maybe a lack of buffer space in terminal server?

I doubt it. Personally, aside from a gain for interactive work over a loaded low bandwidth connection, especially with some sort of priority queueing used, I find no gain from a smaller MTU on a dialup link.

...

I guess systems receiving data from servers with broken retransmission timers (eg. how Solaris used to be) could be helped by lowering the MTU which would result in faster ACKs so bogus retransmissions won't happen all the time, but the fix for this really isn't to lower the MTU.

Perhaps an initial transmission pace on the first few packets sent over a connection would help, too. There is virtually no value to send packets 1-7 immediately behind packet 0. The problem is there isn't any definition of what the pace should initially be. If I were to redesign TCP I would probably make the sliding window be small initially and gradually widen as the turnaround time becomes obvious, then pace the packets to match what the turnaround time looks like.

Look at TCP slow start.

...

You also get the obvious improvements in interactive performance, and you start getting data more quickly.

I would suggest that you would be well advised to find a handy user or four where this effect is easily observable, and find out what is really going on.

I suspect the buffer overflow situation. At least it should be looked at.

I recall a few months ago a provider I was connected to was having some very horrible performance. My modem connection wasn't blocking at all so it was letting everything through. I started up a ping from a machine out on the net back to me, and noticed I was getting most packets through OK, yet my TCP connections were horrible. Well ping defaults to 64 bytes so I increased it to 1472 and virtually none of them ever got here. I tried a number of different sizes and found erratic performance but on average the loss was proportional to the packet size up to around 1k where there was near total loss. I dropped my MTU down to 64!! and reconnected the telnet session. Now I was getting through. I was getting a very obvious "small-chunk" kind of response performance, but it did let me through.

This is obviously a pathological case where something else is broken and I'm really not sure it carries over. [...]

Some ISP may in fact be setting the MTUs on their routers lower to 576 or some other number just to force MTU discovery to kick in and chop the size of the packets. While I might consider doing this myself, I hesitate due to the fact that MTU discovery is often broken due to a number of reasons. Some sites are (stupidly) blocking all ICMP and if they also have DF set, the connection is hosed. In other cases, like the early version of the Cisco Local Director, the ICMP did not get back to the machine that held the connection (since ICMP network-unreachable doesn't include port numbers and LD's tables surely were port indexed) so again MTU discovery was broken and if DF was set, the connection was hosed. I saw this because many web servers apparently push the HTTP header part of the response ahead and it

Yea, because if they don't then dumb clients like Navigator screw up sometimes. :( It can't even properly handle reads where the boundry of the read data falls in the wrong place. I hate dumb workaround for broken clients. If you ever see a "X-Pad: avoid browser bug" sent from Apache, that's why since Apache tries to keep responses in as few packets as possible.

was smaller than my link MTU at the time so it came through OK, but the data that followed filled the packets and never made it. I saw the DF on the packet with the HTTP part and that's how I figured what was going on.

So turning down the MTU in the router at the ISP can be a problem and should not be done, but turning down the MTU on the end machine will work, whether it is ultimately the correct solution or not.

Users also perceive a better response if all the images are loading in parallel as opposed to them loading one at a time, even if the MTU setting to accomplish this smoothly has a net effect of a longer time for a complete load of all images.

Maybe what we need is a multiplex-HTTP with a server smart enough to send the images over separate subchannels without the browser needing to request them.

HTTP-NG. I would say more but I don't know more because the W3C is a PITA when it comes to individuals. Heck, even as things are now, clients would often come out ahead to use a pipelined persistent connection rather than a bunch of other connections. Lowering the MTU may appear to fix a lot of things, but really is a bad thing for performance and network overhead. I would strongly recommend that Microsoft look at why people are complaining and exactly what they are complaining about before changing any defaults. There are lots of possible reasons for this behaviour, and while no one reason applies in all cases, I suspect that one does apply in many.

...

There are all sorts of people spouting all sorts of lies around Windows newsgroups about why small MTUs are good; I think novice users are simply getting drawn in by supposed experts.

Such as?

I don't think it's an issue of lies about why small MTUs are good, but the fact that if joe random user is told "576 is good", then they assume that anything different is _bad_. We all know that there's good reasons for all sorts of different MTUs, mainly because of the different types of traffic you could have - there's always going to be a tradeoff between efficient transfer of large blocks of data and immediaecy (sp?) of response time. The problem with anything Microsoft may put forth is that it'll read like "576 is good. It is the best. We do it. So should everybody. If 576 is good, all else must be bad." And that's simply not the case. eric

Eric Osborne writes:

The problem with anything Microsoft may put forth is that it'll read like "576 is good. It is the best. We do it. So should everybody. If 576 is good, all else must be bad." And that's simply not the case.

I must admit that I'm very worried about what will happen to the internet if they do this. Why? This will effectively triple the number of packets that routers have to do processing on, that's why. Frankly, the whole thing is stupid. I didn't say this earlier, but Microsoft has about the most abominable stacks out there. Their boxes STILL don't handle recieving fragmented packets correctly (a requirement of the RFCs!) and they have a host of flaws. I wish they'd get their act together. Perry

On Thu, 5 Feb 1998, Perry E. Metzger wrote:

...

Eric Osborne writes:

...

The problem with anything Microsoft may put forth is that it'll read like "576 is good. It is the best. We do it. So should everybody. If 576 is good, all else must be bad." And that's simply not the case.

I must admit that I'm very worried about what will happen to the internet if they do this. Why?

This will effectively triple the number of packets that routers have to do processing on, that's why.

Packets going through our border routers have an average packet size in the region of 200 to 250 bytes. How would this triple the average packet size?

I think that "triple" is perhaps an oversimplification. This assumes that without a 576-byte MTU, all packets would be 1500-byte MTUs. 1500/576 ~=~ 3. Remember, there's three kinds of average: mean, median, mode. While the mean packet size may be 200-250 bytes, the mode and median are probably different. I don't have any statistics on this, but I'd be willing to guess that if you plotted the packet sizes frequency you'd see something like a bimodal curve, with a small peak at around 64 (ping) and a larger one near 1500 (10Mbit Ethernet/T1 MTU). As to median packet size, I have no idea. It's probably somewhere in the middle. :) eric

My common sense tells me that your concepts are kind of half flawed, but, of course, they may not be >;). Let me explain why I think this. Basically, there are two types of connectivity/traffic; interactive and non-interactive. As examples, I would put telnet in the former and FTP & HTTP/WWW in the latter. Theory tells me that for both types of traffic it is probably better, for response times sake, to have an asymetrical MTU (send = smaller, receive = bigger from the clients perspective). Servers set big MTU's, clients set their's smaller. Irrespective of your MTU size, the file or web page, etc. size is always going to be the same, therefore, if you set a smaller MTU at the server or within the network, fragmentation occurs, meaning greater overhead for a file of a given size and due to this the end station will have to reconstitute the data stream out of smaller packets, meaning more CPU overhead. -Steve. ====================================================== Steve Carter scarter@genuity.net GTE Internetworking Phone: (602) 308 2386 http://www.genuity.net http://www.bbn.com finger steve@lynx.genuity.net for public key ====================================================== <snip>

I have my MTU set to 552 and it helps quite a bit. It's not an issue of the Win95 stack being broken. I'm running Linux.

The reason I chose to use a low MTU, and sometimes I knock it down even further, is to be able to improve my telnet interactive response over a 33.6k link that I also run as many as 4 concurrent downloads on.

Here's what I suspect is happening:

With web surfing, a page loads with many images, each of which is often larger than a sliding window worth of packets. The browser will nearly concurrently connect and request for every image. Thus for N images you now get N sliding windows worth of packets slammed at you. This takes up a _lot_ of buffer space in the dialup routers for all these concurrent TCP connections all sending data at the same time over a high speed net to a low speed final link.

With this happening, buffer space is exhausted and packets are discarded. If you set the MTU smaller, then the size of all those packets is smaller and the chance of being discarded due to memory exhaustion is reduced, even if you're the only one on that server with small packets. <snip>

Its really amazing how people who know very little seem to think they have so much to say. Do any of you rocket scientists have even the smallest inkling of the fact that there are people out there who actually DO know what they are talking about, and have produced standards explaining how to do the stuff? Perry Phil Howard writes:

Steve Carter writes...

...

Theory tells me that for both types of traffic it is probably better, for response times sake, to have an asymetrical MTU (send = smaller, receive = bigger from the clients perspective). Servers set big MTU's, clients set their's smaller.

Irrespective of your MTU size, the file or web page, etc. size is always going to be the same, therefore, if you set a smaller MTU at the server or within the network, fragmentation occurs, meaning greater overhead for a file of a given size and due to this the end station will have to reconstitute the data stream out of smaller packets, meaning more CPU overhead.

I still think there has to be some kind of better approach to what it is we are doing when we have such extreme ranges of bandwidth capacity and the resultant extremes of optimal MTU. One idea I'm thinking of, and I may well even give it a try between a couple of Linux boxes over a phone line, is what I call "cell multiplexed PPP". Basically this would be a channelized stream that can parallel multiple packets. Small ones can come right through while the big ones are still working. That may only help minimally for parallelizing image loading unless there is added logic that detects the TCP ports and ensures that only one port at a time is taking up a channel.

Steve Carter writes:

Without discussion and questioning where would we all be?

They laughed at Fulton. They also laughed, however, at Bozo the Clown. I find it embarassing that people who should know this stuff cold are, putting it VERY charitably, speculating (very badly) about stuff that is already well understood. If you just spent a little time reading, you guys wouldn't have to blather about "why did they do things this way? does the don't fragment bit cause your router to explode? what is this ICMP thing anyway? And I think we need MTUs to modulate according to the sequence of digits of PI".

Would we all be running ARPA nets, or even still hunting animals with clubs? Do you consider the standards to be the last word and that there is no room for discussion?

It would be neat if people actually had some familiarity with the contents of the RFCs before discussing them. I realize that actually reading is difficult for many in this mailing list, but I suspect that audio or pictographic versions of the documents might be made available for many of you. I suppose that's cruel, but to someone who actually understands how this stuff works, it sounds like, hmmm.... it sounds more or less like a couple of people saying back and forth "I bet you could make Oxygen atoms with more protons! That would make them taste better!" Sure, no one gets anywhere without discussion and inquiry, but there is informed discussion and there is drivel. The far, far sadder thing, by the way, is that many of you aren't nearly as ignorant and stupid as the guys at the vendors. "Oh, we disabled slow start because we benchmarked much higher performance that way! We don't handle fragmented packets because we were too busy! Oh, and we've got this neat new proprietary IP Option..." Perry