FWIW, our router at MAE-East says: IP packet size distribution (38569M total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .400 .046 .016 .018 .012 .008 .009 .011 .012 .006 .007 .005 .004 .004 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .010 .006 .120 .000 .099 .197 .000 .000 .000 .000 .000 It's been up for some weeks and the sample size is, well, OK. As expected, large number at the low end, but I'm not sure what to think about the 2048 slot. I would have thought that once packets are bigger than ethernet MTU, they would also be bigger than 2048. If the numbers are true, then close to 1/3 of all packets (1536 slot plus 2048 slot) are larger than 576. -- ------ ___ --- Per G. Bilse, Director Network Eng & Ops ----- / / / __ ___ _/_ ---- EUnet Communications Services B.V. ---- /--- / / / / /__/ / ----- Singel 540, 1017 AZ Amsterdam, NL --- /___ /__/ / / /__ / ------ tel: +31 20 5305333, fax: +31 20 6224657 --- ------- 24hr emergency number: +31 20 421 0865 --- Connecting Europe since AS286 --- http://www.EU.net e-mail: bilse at domain
IP packet size distribution (38569M total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .400 .046 .016 .018 .012 .008 .009 .011 .012 .006 .007 .005 .004 .004
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .010 .006 .120 .000 .099 .197 .000 .000 .000 .000 .000
Note that if every 1536 packet became 3 paskets and every 2048 packet became 4 packets that would increase the packet count by 80% !!! ...George ================================================================== George Swallow Cisco Systems (508) 244-8143 250 Apollo Drive Chelmsford, Ma 01824
On Thu, 5 Feb 1998, George Swallow wrote:
IP packet size distribution (38569M total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .400 .046 .016 .018 .012 .008 .009 .011 .012 .006 .007 .005 .004 .004
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .010 .006 .120 .000 .099 .197 .000 .000 .000 .000 .000
Note that if every 1536 packet became 3 paskets and every 2048 packet became 4 packets that would increase the packet count by 80% !!!
Unfortunately this will be rather impossible with the MTU currently set to 1536 the 2048 byte packets must be coming from another source (router to router connections?). So the number of 2048 packet will stay the same, but the 1536 ones will triple (adding ~20% more packets in this scenario). The big win (from the users perspective) is that web pages update quicker. Instead of sitting there waiting for a large packet to come in before a page is partially updated the user could have 2/3 of the data already onto the page. While it might not be "the right thing to do" from the perspective of helping the "internet" function the perceived speed by the end user is higher and that is part of the battle we are all fighting (witness the coming deployment of more localized caches). bye, ken emery
<lots of speculation about packet sizes and mtus and the occasional bash at pmtud and microsfot thrown in for good measure> recently i was doing a fair amount of image-intensive web browsing from home. the last-hop dialup line was 28.8. was on a win95 machine with ftp software's stack. i was getting real bad performance. aggregate receive 'goodput' on the order of 10s or 100s of bytes/second. but the modem lights were constantly on. so i fired up the ol packet monitor and watched the packets. i saw many tcp connections open all at once. i saw many many many retransmissions. i saw many duplicate acks going back. transmissions in both directions were very very bursty it was ugly. so, the line was running at full speed, but most of what came across was useless glop. my theory was that it was a classic case of connection fratricide. in short, i'd open a web page and get umpity-poo connections established, one for each image, etc, etc. they'd all be in slow start and not interfere with each other. they'd get a reasonable rtt and then start cranking out the data. they'd all do this at once. implosion of packets onto the poor dialup server. dialup server q-length grows. that reasonable rtt gets blown all to hell. the webserver misses some acks. it resends. even though all the connections start slowing up, there still are umpity-poo connections all (slowly) dumping packets into the dialup server. lots of retransmissions come across. which i constantly re-ack. then things go quiet a bit. then the cwnd opens up again. repeat. so, to test the theory, i crank down the number of connections that netscape will open up (an older version of navigator will let you do this. i haven't figured out how to do it in the newer browsers). i crank it down to 2. goodput goes _way_ up. i consistantly get over 3kbytes/sec(24+kb) on the 28.8 line (as opposed to maybe 500bytes/sec...). i look at the packet trace then. nice beautiful packet-in, ack-out, packet-in, ack-out, series. seq-nums going up nicely as they should. no retransmissions. note well that _none_ of this has to do with pmtu. none of this has to do with the oft-proclaimed poor quality of m.s. software. <soapbox> moral of the story theorizing is wonderful but a good packet trace beats a theory every time and understanding how the protocols are _supposed_ to work and then trying to figure out why they don't is a lot more productive than bashing mammoth-software-corporations, though a lot harder </soapbox> frank kastenholz
Frank Kastenholz writes...
note well that _none_ of this has to do with pmtu. none of this has to do with the oft-proclaimed poor quality of m.s. software.
Turn the MTU down, way down, and see how it affects things. At what point in number of connections does the problem happen with MTU=1500 and at what point does it happen with MTU=600 and even MTU=200. Of course changing MTU isn't the correct solution, nor is changing the number of connections. But these are workarounds that do work, for now. Perhaps the retransmission timeout is the real problem as suggested, but given there are different effects based on what client stack is used, as apparently see on the end results and packet traces with Windows 95 and Linux as the clients, there is some effect by the client. That may not be the true cause, but it should be discovered and logically ruled out if it isn't.
<soapbox> moral of the story theorizing is wonderful but a good packet trace beats a theory every time
and understanding how the protocols are _supposed_ to work and then trying to figure out why they don't is a lot more productive than bashing mammoth-software-corporations, though a lot harder </soapbox>
I'll admit to having never read the _whole_ protocol, so I will make a point to do that at some point. But having read the details on slow start as a result of this thread, I find it's not what I proposed, although I can now see that it, too, would have been explained as I explained my idea, so the explanation was flawed. I might try to implement my ideas via UDP and see what kinds of effects I can really get over real networks of the 1990's, understanding that the original design was done in the 1980's and before with slow links and slow routers. -- Phil Howard | stop0it8@anywhere.edu no3spam2@s9p6a8m8.com stop2012@anyplace.net phil | ads6suck@lame8ads.net w2x7y5z2@dumb6ads.com end8ads2@anyplace.edu at | blow1me4@spam0mer.edu crash416@no19ads1.com suck3it7@no2place.edu milepost | stop8149@nowhere6.edu no88ads5@no0where.org a5b7c4d9@noplace5.net dot | eat84me0@dumbads2.com stop9ads@lame1ads.net die0spam@spammer9.com com | end0ads2@nowhere7.edu a9b4c4d7@dumbads8.edu stop2ads@spammer9.org
Sez Phil Howard:
Turn the MTU down, way down, and see how it affects things. At what point in number of connections does the problem happen with MTU=1500 and at what point does it happen with MTU=600 and even MTU=200.
Of course changing MTU isn't the correct solution, nor is changing the number of connections. But these are workarounds that do work, for now.
Changing the number of connections IS the correct answer in this case. If you are a dialup user at 33.6k (or 56k even), you should be very acutely aware of how puny your pipe is. There is absolutely no reason to expect 64 simultaneously connections to an OC-12 connected server to behave normally. HTTP 1.0 is known to be severely flawed in this respect; it opens large numbers of connections and closes them before slow-start and congestion avoidance can kick in. If your primary complaint is interactive response while performing large downloads, changing the MTU is the correct answer. This is simply a matter of the transmission time on large packets. Does anyone have data to show if any terminal servers or client stacks will honor TOS bits and/or known interactive port numbers when ordering packets for transmission across slow links?
point to do that at some point. But having read the details on slow start as a result of this thread, I find it's not what I proposed, although I can now see that it, too, would have been explained as I explained my idea, so the explanation was flawed. I might try to implement my ideas via UDP and see what kinds of effects I can really get over real networks of the 1990's, understanding that the original design was done in the 1980's and before with slow links and slow routers.
I'm sure we'd all like to see a reference implementation of your ideas. Keep in mind that one of the primary features of PMTUD is that it requires no modifications to any hosts or routers, and that it uses only one protocol (ICMP) which is required to be implemented on all hosts. -S -- Stephen Sprunk "Oops." Email: stephen@sprunk.org Network Consultant -Albert Einstein ICBM: 33.00151N 96.82326W
On Fri, Feb 06, 1998 at 10:17:03AM -0600, Stephen Sprunk wrote:
I'm sure we'd all like to see a reference implementation of your ideas. Keep in mind that one of the primary features of PMTUD is that it requires no modifications to any hosts or routers, and that it uses only one protocol (ICMP) which is required to be implemented on all hosts.
But, practically, is _not_, completely, right now; wasn't this the complaint I heard? Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "Two words: Darth Doogie." -- Jason Colby, Tampa Bay, Florida on alt.fan.heinlein +1 813 790 7592 Managing Editor, Top Of The Key sports e-zine ------------ http://www.totk.com
On Fri, Feb 06, 1998 at 02:35:55PM -0500, Perry E. Metzger wrote:
"Jay R. Ashworth" writes:
But, practically, is _not_, completely, right now; wasn't this the complaint I heard?
I haven't had any problems with machines that implement PMTU discovery.
Didn't someone bitch yesterday about routers that drop over-MTU packets without returning the appropriate ICMP complaint message? Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "Two words: Darth Doogie." -- Jason Colby, Tampa Bay, Florida on alt.fan.heinlein +1 813 790 7592 Managing Editor, Top Of The Key sports e-zine ------------ http://www.totk.com
On Sat, Feb 07, 1998 at 08:56:33AM -0800, Douglas Tooley wrote:
Smells like a marketing opportunity.
What, marketing a replacement stack? (you really should quote, BTW)
No, because the _users_ don't know enough better to care. It's the _net_ that suffers.
Actually I think there is a fair bit of awareness among the "users" out there - at least among those that are actually changing their MTU's...
I recently discovered that some of my customers have been led to believe that they should employ a program called "Turbo MTU" (or some such), which apparently has knobs for every tiny detail of Windows 95's TCP/IP stack. Since these are often the same type of customers who simply change every setting they can at random, I should not be suprised that they encounter performance problems. -- Christopher Masto <chris@netmonger.net> Director of Operations, NetMonger Communications, Inc. +1-516-221-6664 http://www.netmonger.net/ info@netmonger.net
Actually I think there is a fair bit of awareness among the "users" out there - at least among those that are actually changing their MTU's...
I recently discovered that some of my customers have been led to believe that they should employ a program called "Turbo MTU" (or some such), which apparently has knobs for every tiny detail of Windows 95's TCP/IP stack. Since these are often the same type of customers who simply change every setting they can at random, I should not be suprised that they encounter performance problems.
IMHO, most of the people out there changing their windows 95 have no idea what it is. They were just told by someone to do it, and when they see it does make an improvement, they think they have an "underground" source to secret information used primarily be Internet experts. I've heard at least one CNE tell someone it was a "Microcomputer Timeout Unit". I wonder who is the most ignorant in some of these cases. -- Phil Howard | stop2it1@no1place.net stop3545@no52ads7.edu eat1this@dumb4ads.com phil | suck4it0@nowhere7.net no9spam7@dumbads6.org a1b9c8d8@anywhere.com at | no25ads3@no1where.com w0x0y7z9@no01ads7.org ads0suck@dumbads5.com milepost | a6b7c5d1@dumb5ads.edu stop9895@anywhere.net eat88me7@no5where.net dot | suck5it9@spammer0.edu no4way60@nowhere0.com end7ads8@no41ads2.org com | die2spam@dumb2ads.org no2spam6@anywhere.net stop9it8@lame2ads.net
Stephen Sprunk writes:
Turn the MTU down, way down, and see how it affects things. At what point in number of connections does the problem happen with MTU=1500 and at what point does it happen with MTU=600 and even MTU=200.
Of course changing MTU isn't the correct solution, nor is changing the number of connections. But these are workarounds that do work, for now.
Changing the number of connections IS the correct answer in this case.
If "this case" == "your computer" then of course whatever you want to be the correct solution is the correct solution. It would not be so in the case of my computers or computers I give advice for if the user (like I) wants to have all images show in parallel in order to have some idea of which to click without waiting for everything to load.
If you are a dialup user at 33.6k (or 56k even), you should be very acutely aware of how puny your pipe is. There is absolutely no reason to expect 64 simultaneously connections to an OC-12 connected server to behave normally. HTTP 1.0 is known to be severely flawed in this respect; it opens large numbers of connections and closes them before slow-start and congestion avoidance can kick in.
Isn't slow start supposed to be there at the beginning? I have 33.6k and am fully aware. I run my MTU at 552 and I know how to change it on the fly (for new connections) any time I want to. I have many times run it at 104.
If your primary complaint is interactive response while performing large downloads, changing the MTU is the correct answer. This is simply a matter of the transmission time on large packets.
That's one of them. The other is having multiple images on a page load in parallel instead of having to wait for some to finish all the way to high res before others can even get started. Low MTU fixes that, too. I won't say that a low MTU is the ultimate correct solution as long as every aspect of the protocol and every detail of each implementation is equally open to review and blame. The "correct" solution may even be as drastic is discarding the whole protocol and starting over. But a low MTU does work and does accomplish my goals where no other has. Another solution that could help is a better design of some web pages. Lots of pages have collections of buttons with each being a separate image. If they were to design them as a single image for all buttons that loaded low res first and filled in high res later, then one could at least get some idea early on about where the buttons are. And if one is familiar with the page they could even click on the proper button even before the text becomes obvious. This may not be the "correct" solution, but I know it would certainly help. And it would meet my needs (don't know about yours).
I'm sure we'd all like to see a reference implementation of your ideas. Keep in mind that one of the primary features of PMTUD is that it requires no modifications to any hosts or routers, and that it uses only one protocol (ICMP) which is required to be implemented on all hosts.
I understand this. And that's why I see it as a hack. It was done as a workaround to the problem that router vendors would not upgrade their routers. Maybe that was because the base standard was then frozen and could not be expanded on, or maybe it was because the vendors didn't want to go back to tweaking the router code (although I hardly believe they never do). If you were designing a whole new protocol from scratch without any of the vendor constraints, would you do it the same way? How would you do it? -- Phil Howard | ads8suck@no7where.edu eat59me0@spam1mer.org die9spam@lame4ads.org phil | ads6suck@s1p6a5m7.com end2it77@no19ads5.edu eat9this@noplace5.net at | stop1it8@no2place.com no94ads7@dumb8ads.edu ads4suck@s3p2a2m4.net milepost | stop8801@no29ads3.org no5way27@s5p1a4m1.edu stop5ads@anyplace.org dot | w2x0y3z0@no94ads6.net stop4it3@anyplace.com eat66me8@anywhere.com com | no5way51@no2where.edu crash285@spam2mer.org end4ads7@anywhere.net
At 10:17 AM 2/6/98 -0600, Stephen Sprunk wrote:
Does anyone have data to show if any terminal servers or client stacks will honor TOS bits and/or known interactive port numbers when ordering packets for transmission across slow links?
This is relatively trivial with Cisco terminal servers, but it does have to be done manually. I'd be really surprised if no other vendor can specifically order packets in the queue by port number or something. Cisco does automatically take specific steps to allow smaller packets (regardless of TOS or port number) through the box faster when coming from many slow interfaces to one fast interface. The idea is, why should a bunch of (completed) telnet packets wait for the ethernet/T1 while a gigantic 1500 byte FTP packet is still serializing from a 28.8 modem connection? This should help "perceived" response times on interactive (e.g. telnet w/ 64 byte packets) stuff while not actually decreasing the throughput of large file transfers on other peoples' links. Supposedly, some other vendors place a packet in the outgoing queue when the first byte arrives. If you have a bunch of small packets come in (and complete) just after the big packet starts, you could create delays and waste bandwidth on the uplink. Of course, I've never actually measured the effects this, nor do I have first hand knowledge of how other vendors handle the process. I'm just taking Cisco's word for it (which you are more than welcome to disprove with real data - if possible ;).
Stephen Sprunk "Oops." Email: stephen@sprunk.org
TTFN, patrick ************************************************************** Patrick W. Gilmore voice: +1-650-482-2840 Director of Operations, CCIE #2983 fax: +1-650-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "Tomorrow's Performance Today" **************************************************************
Curtis Villamizar's Feb 97 NANOG talk has some relevant and interesting info. See http://www.academ.com/nanog/feb1997/TCP-LOSS.html
<lots of speculation about packet sizes and mtus and the occasional bash at pmtud and microsfot thrown in for good measure>
To paraphrase a well-known quote, those who do not study history are condemned to re-invent it.
In message <3.0.3.32.19980206081340.0095ce90@shultz.argon.com>, Frank Kastenhol z writes:
recently i was doing a fair amount of image-intensive web browsing from home. the last-hop dialup line was 28.8. was on a win95 machine with ftp software's stack. i was getting real bad performance. ...
perhaps this is one of the not-so-obvious benefits of running a web proxy cache such as squid. the greater internet can have larger packets floating around, and the local proxy of the ISP can deal with horrible tcp stacks, retransmissions and client machine with small receive buffer sizes. certainly from on the other side of the world here, i'd say it would contribute to increasing the overall 'goodput' quite substancially, in terms of the behaviour you describe above. cheers, lincoln.
perhaps this is one of the not-so-obvious benefits of running a web proxy cache such as squid. the greater internet can have larger packets floating around, and the local proxy of the ISP can deal with horrible tcp stacks, retransmissions and client machine with small receive buffer sizes.
And imagine having 2 interfaces on this machine, one with MTU=1500 and one with MTU=576. -- Phil Howard | no33ads4@dumb5ads.org ads6suck@dumb2ads.org stop3it7@spammer9.com phil | no0way72@lame7ads.net die8spam@no99ads1.net eat7this@anyplace.com at | w5x8y4z5@anyplace.com stop2it1@spammer0.com a9b0c8d8@noplace4.com milepost | die6spam@noplace8.com end6it79@no2place.com a5b0c5d3@s0p8a2m1.edu dot | crash053@anyplace.org crash981@spam6mer.com end8it77@dumbads7.org com | no20ads6@anyplace.org stop6221@dumb3ads.com ads9suck@no7place.net
On Thu, Feb 05, 1998 at 04:47:36PM +0100, Per Gregers Bilse wrote:
FWIW, our router at MAE-East says:
IP packet size distribution (38569M total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .400 .046 .016 .018 .012 .008 .009 .011 .012 .006 .007 .005 .004 .004
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .010 .006 .120 .000 .099 .197 .000 .000 .000 .000 .000
It's been up for some weeks and the sample size is, well, OK. As expected, large number at the low end, but I'm not sure what to think
Bug. I believe counters a shifted by one slot from 1024. As you'll note that that slot is empty. -dorian
At 05:45 PM 2/5/98 , Dorian R. Kim wrote:
On Thu, Feb 05, 1998 at 04:47:36PM +0100, Per Gregers Bilse wrote:
FWIW, our router at MAE-East says:
IP packet size distribution (38569M total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .400 .046 .016 .018 .012 .008 .009 .011 .012 .006 .007 .005 .004 .004
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .010 .006 .120 .000 .099 .197 .000 .000 .000 .000 .000
It's been up for some weeks and the sample size is, well, OK. As expected, large number at the low end, but I'm not sure what to think
Bug. I believe counters a shifted by one slot from 1024. As you'll note that that slot is empty.
-dorian
I'll bet that is true, since histograms for the last couple of years have shown clusters at 40, 512, 576, 1024 and a tiny bit of 1500. Looks like it hasn't changed much since. --Kent
On Feb 5, 20:45, "Dorian R. Kim" <dorian@blackrose.org> wrote:
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .010 .006 .120 .000 .099 .197 .000 .000 .000 .000 .000
It's been up for some weeks and the sample size is, well, OK. As expected, large number at the low end, but I'm not sure what to think
Bug. I believe counters a shifted by one slot from 1024. As you'll note that that slot is empty.
Yes, that's what one would suspect, but it's been like that ever since flowswitching was first introduced (which, of course, doesn't necessarily mean anything). In any case, the interesting part is that 30% of all packets would still be larger than 576, with probably close to 20% being larger than 2*576, meaning that an "enforced" MTU of 576 would lead to a very substantial increase in packet count. This would be a big step in the wrong direction. (On that note, I'm actually not sure about the wisdom of MTU discovery in the Internet, given the large number of dial-up users. The way to go is to send big packets and fragment in the access server if the user has negotiated a small MRU.) -- ------ ___ --- Per G. Bilse, Director Network Eng & Ops ----- / / / __ ___ _/_ ---- EUnet Communications Services B.V. ---- /--- / / / / /__/ / ----- Singel 540, 1017 AZ Amsterdam, NL --- /___ /__/ / / /__ / ------ tel: +31 20 5305333, fax: +31 20 6224657 --- ------- 24hr emergency number: +31 20 421 0865 --- Connecting Europe since AS286 --- http://www.EU.net e-mail: bilse at domain
Bug. I believe counters a shifted by one slot from 1024. As you'll note that that slot is empty.
What a relief. Means changing the MTU down can at most increase the traffic by 50%. ...George ================================================================== George Swallow Cisco Systems (508) 244-8143 250 Apollo Drive Chelmsford, Ma 01824
participants (15)
-
Christopher Masto -
Dorian R. Kim -
Douglas Tooley -
Frank Kastenholz -
George Swallow -
Jay R. Ashworth -
ken emery -
Kent W. England -
Lincoln Dale -
Mark Boolootian -
Patrick W. Gilmore -
Per Gregers Bilse -
Perry E. Metzger -
Phil Howard -
Stephen Sprunk