- NANOG - lists.nanog.org

Re: Questions about Internet Packet Losses
by William Allen Simpson 14 Jan '97

14 Jan '97

Yes, Bakul, keeping a central RTT cache per destination is a good idea. Most good stacks use it already. I think it was recommended in Host Requirements circa 1989. Keeping a per destination cache of Path RTT, Path MTU, and a Quality measurement was required in my initial IPng Neighbor Discovery design several years ago, before that was destroyed in the rewrite by committee. > From: Bakul Shah <bakul(a)torrentnet.com> > [Thinking aloud here...] > Perhaps a part of the TCP congestion avoidance algorithm can be > factored out in some sort of a `traffic central' module that tries > to give you the best bandwidth/packet loss estimate it has for a > given route provided you keep it updated with what you learn (i.e. > TCP tells it when a packet is lost etc). A new TCP connection can > then immediately start off with a bigger window (and won't open the > window too wide too quickly). Multiple connections between two > hosts can avoid what would be largely redundant estimate > computation. Even a UDP app. can try to benefit from this (such as > for communication where bounded delay is more critical than packet > loss). Other `traffic conditions' input can also be fed into this > module [perhaps as part of some future routing protocol]. Combining > this `quality' of a route aspect into routing protocols may make > sense in the long run.... > WSimpson(a)UMich.edu Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32 BSimpson(a)MorningStar.com Key fingerprint = 2E 07 23 03 C5 62 70 D3 59 B1 4F 5E 1D C2 C1 A2

1 0

Re: Questions about Internet Packet Losses
by Vern Paxson 14 Jan '97

14 Jan '97

> So you'll note that there are very clear spikes at 552 and 576 total packet > size. Further, note that the curve falls off _extremely_ rapidly. From > that, I argue that case (2) _cannot_ be the case because any type of random > distribution would be much smoother. (2) is a random distribution plus some spikes, plus some modality due to TCP's that can only generate packets of <= N bytes for different popular MSS's. So I think it still works (but will save more detailed analysis till later ....). > 2) What OS is using a 512 MSS? >From a bunch of (~ 20,000) traces in my study, looks like Irix 4.0/5.3, BSDI 1.1/2.0, OSF/1 3.0/3.2, and often but not always SunOS use either 512 or 536, or sometimes 500, 524, or 548 (BSDI). Never anything higher. This doesn't mean they *can't* use anything higher, just that they never found an opportunity to do so. 512 is far and away the most common MSS in the traces. (All the TCP's in the traces are BSD variants except for Solaris and Linux.) > 256? My best guess is SLIP links. > 4) 41 bytes is pretty obviously interactive traffic. Is the intuition > correct? That's my guess too. > What's so special about 44, 52, 48 and 56? Ack/SYN/FIN's with options. > What do people do > with 4, 8, 12 and 16 bytes of data? And why not any of the odd values? Because option sizes are rounded up to multiples of 4 bytes. Vern

2 1

Re: Questions about Internet Packet Losses
by Vern Paxson 14 Jan '97

14 Jan '97

> indicate that 9% of the packets had a 512 byte *payload*. > > I already subtracted off the 40 bytes for the headers. I meant MSS. (By payload, I meant data, i.e., not including headers, same as you ...) > However, > the spikes in the packet percentages are clearly due to hosts using that > MSS. You would otherwise expect to see a "smooth" distribution across > packet sizes, which is not at ALL what's happening. Right, spikes correspond to MSS's (and other effects like acks & SYN/FIN's). Are you saying the data shows significant spikes at other, non-standard MSS values? If it does, then (modulo some thinking about the particular numbers) I agree that they're likely MSS candidates and support the notion that weird MSS's are being used. But if the data has a few spikes at things like 512 but no strong spikes at non-standard values, then that doesn't help distinguish between (1) lots of weird TCP's using weird MSS's, vs (2) lots of packets that are less than full-size because the mainstream TCP didn't have enough bytes to fill them. If your data is available, I'd love to take a gander at it. Vern

3 3

Re: Questions about Internet Packet Losses
by Vern Paxson 14 Jan '97

14 Jan '97

> Isn't it unsurpsrising that >40% of packets are small? ACKs? > > If one made the gross oversimplifying approximation that everything is > unidirectional TCP traffic, then you'd expect to see one ACK per two data > packets. This is actually not at all a gross oversimplying approximation. Much TCP traffic unidirectional or request/response. The latter looks a whole lot like two back-to-back unidirectional connections, with only one chance to piggyback acks at the switchover. > Thus, we'd expect to see 33% at 40 bytes. It is the additional > 7+% that's surprising. Throw in frequent SYN/FIN/RST's because of small Web connections, SYN retransmissions because busy Web servers have full listen queues (or are being SYN-flooded :-), TCP's that ack every packet (Linux 1.0, I believe, and maybe later ones), TCP's that wind up acking every packet because their delay timer is less than an MSS prop time across a slow link (Solaris, for links < 10KB/sec), and dup acks due to sequence holes, and you might handwave the 7%. Vern

2 1

Re: 30%, huh? (Re: Questions about Internet Packet Losses )
by Vadim Antonov 14 Jan '97

14 Jan '97

Paul A Vixie <paul(a)vix.com> wrote: >I wish I had a background in "fractal mathematics" or whatever, but instead >I've just got my gut feelings to go by. They say that 30% loss translates >loosely to 30% overcommitment. I wish it was as simple as that. My educated guess is that 30% packet loss corresponds to about 1000% overcommitment. TCP actively reduces packet loss by doing congestion avoidance, so the results are more than a bit skewed. Strictly speaking, your assumption would be correct if packet arrivals were independent (i.e. Poisson). >As Tony and others have remarked here today, >TCP does not overcommit enough once it's up and running to account for 30% >loss. Just a little bit to test water :) But that's steady state. We're talking about the system where TCP doesn't really have a chance to establich proper window, mostly because of HTTP silliness. >And given slow start, I'm not sure it could overcommit that much even >given HTTP 1.0's tendancy to start a lot of new connections. HTTP barf barf barf. That's the poster case of shortcomings of the computer science education which prefers to neglect the gory details of what happens in the hardware or "lower levels" in general. >So if we are seeing 30% overcommit, my gut feeling is that most TCP endpoints >aren't using slow start. No, 30% loss can easily be caused by the initial TCP SYNs. The average flow duration is something like 12 packets, yes? TCP works well only when there's enough slack to handle initial SYNs. Actually, that's one argument against RED. It is interesting to explore how the network will behave if SYNs are given priority over the "regular" packets. >IETF had a huge flame war over whether slow start should be a recommended >standard or not since it was just an implementation detail or so said some. I'd say IETF should be abolished. If there's an argument about _that_ there's a serious doubt about overall sanity of the organization. >TCP creates a prisoner's dilemma problem for implementors, and it's sad that >more than one has said "let's not cooperate so that we can get more bit times." Well, no. Non-compliant TCP implementation is like ping -f. It should be a sufficient reason for disconnecting from the network. >I saw an ad for one company who advertised that their TCP stack was 200% >faster than the rest of them. Yep. The big problem is the sad state of cluefulness of the general user society. This ad sounds awfully like "buy our gun, earn $1000 in 30 minutes!". >As a good friend likes to tell me: "The world is full of clowns." Indeed. --vadim

1 0

Re: It Continues...Sprint Is played the fool...
by Sean Donelan 14 Jan '97

14 Jan '97

> Suggested reading are the rules and regulations for common carriers, Good idea, I suggest everyone do this, or at least have a chat with their company lawyer. Common carrier laws and regulations are a bit dry, but I read a bunch of them dating back to the 1600's. All based on English Common Law, because my non-english language skills are lacking. > Joe does not like the fact that I call him on the > phone, so he calls my local phone operator and have > them to block my phone for outgoing calls. > > My mother is about to die and she needs and ambulance, > but the phone does not work. This is a strawman argument. Barry didn't asked Sprint to block all outgoing communication, or calls to emergency services, only calls to his network number. A different division of Sprint will block calls between certain telephone numbers at the request of the subscriber. They advertise the blocking as a 'feature.' So this isn't a company-wide policy, but rather a division-level business decision. Unless your mother is in the habit of calling Software Tool & Die for ambulances, I don't see the relevance. > All ISP's should have as a policy to move packets, or for some > atlest try to move packets as close they can to the destination. You could have this policy for your ISP. But it isn't a requirement of common carriers when the carrier knows the addressee refused the shipment. An old example, you hire a livery to ship horse manure to my house. I tell the livery I don't want the horse manure delivered to my house. The livery does not have to move the horse manure as close as they can to my front door after I tell the livery I refuse to accept it. A more modern US-centric example, attach a US Post Office Business Reply mailer on a brick, and you will discover the post office will discard the brick as close as they can to the origin. This does not mean you can't mail bricks through the mail. One man mailed an entire house to Alaska, one brick at a time, and the post office delivered them. But if the Post Office knew the addressee had refused to accept the bricks in Alaska, the Post Office isn't required to ship the bricks to Alaska just so the shipment could be refused. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

2 1

Re: Questions about Internet Packet Losses
by Vern Paxson 14 Jan '97

14 Jan '97

> Yeah, but I doubt that they're in the majority. Recent stats indicate that > about 9% of the packets out there are using an MSS of 512. Another 1% are > using an MSS of 256. Only 6% are using 536, and 7% are using 1460. This > indicates to me that there are a whole lot of broken boxes out there still. I believe you're misinterpreting the numbers. The raw data for those numbers (if they're the same ones I'm thinking of) indicate that 9% of the packets had a 512 byte *payload*. This is quite different than whether the TCP sending the packet had a 512 byte MSS, because often the TCP doesn't *have* 512 bytes to send. Crunching through the 340,000 HTTP connections in and out of LBNL last Friday, less than 10% had requests exceeding 512 bytes. Even 35% of the replies were <= 512 bytes. Toss in SYN/FIN and pure acks and you rapidly dilute the relationship between measured packet size and MSS. A much more solid way to estimate MSS's is to look at MSS options in SYN packets. Rich Stevens did this in TCP/IP Illustrated Vol 3. I don't have it here at home for precise figures, but he found that the large majority of the MSS's were from a few standard sizes. He also found an immense range of bizarre values (my personal favorite: 17,520 bytes), but they were rare. All that said, at the next NANOG I'll talk about some findings from a TCP behavior study I've been doing (it's part of the packet dynamics study). One of my main findings is that independently-written (i.e., non-BSD-derived) TCP's are much more likely to have serious performance and congestion problems. Vern

3 2

Re: Questions about Internet Packet Losses
by Vadim Antonov 14 Jan '97

14 Jan '97

Bob Metcalfe <metcalfe(a)infoworld.com> wrote: >Hello, and best wishes for what's left of 1997. Now, if you would, ... Same to you :) >Is Merit's packet loss data (NetNow) credible? As always, the packet loss data doesn't always measure loss in the network. > Do packet losses in the >Internet now average between 2% and 4% daily? The congestion is definitely not distributed evenly. There are hot spots, and to my knowledge there was no any significant research on how those hot spots are distributes, what percentage of paths is affected, etc. >Is there any evidence that Internet packet losses are >trending up or down? Up. On-going backbone upgrades are simply not sufficient to keep up. >Were Merit's data correct, what would be the impact of 30% packet losses on >opening up TCP connections? Close to none. >On TCP throughput, say through a 28.8Kbps >modem? TCP throughput generally goes down the drain with that kind of packet loss. >How big a problem is HTTP's opening of so many TCP connections? HTTP is broken as designed. That "feature" actively destroys the cooperative congestion control mechanism. >Does TCP >need to operate differently than it does now when confronted routinely with >30% packet losses and quarter-second transit delays? No. TCP is self-clocking and fairly loss resistant. This is not to say there's no room for improvement, but nothing spectacular. For example, selective ACKs would help to deal with higher loss rates. 0.25 sec delays is something routine with satellite communications. TCP works just fine with that. >What is the proper >response of an IP-based protocol, like TCP, as packet losses climb? Try >harder or back off or what? That's described in 1988 article by Van Jacobson, Mike Karels et al. The answer is -- exponential backoff is sufficient to keep system stable. >How robust are various widespread TCP/IP >implementations in the face of 30% packet loss and quarter-second transit >delays? There's a lot of broken implementations (mostly in messy-dossy world). The 4.3 Reno-derived TCPs (the majority) are fairly robust. >Is the Internet's sometimes bogging down due mainly to packet losses or >busy servers or what, or does the Internet not bog down? The most annoying problem is not packet loss, it's routing stability. >What fraction of Internet traffic still goes through public exchange points >and therefore sees these kinds of packet losses? What fraction of Internet >traffic originates and terminates within a single ISP? 70% or so packets cross exchange points. That's empirical figure based on hypothesis that rate of content providers to consumers is about constant for all backbone ISPs. There's no difference between public and private exchanges in terms of dealing with congestion. So mentioning _public_ part is a bit misguided. >Where is the data on packet losses experienced by traffic that does not go >through public exchange points? I guess there's no reliable data. In fact, there's no theory on how to collect that data. For example, average, variance, and other traditional statistical instruments are useless when you attempt to measure fractal traffic. You end up with meaningless results -- for example, interarrival time variance is infinite; average rates over similar intervals may differ by orders of magnitude even when nothing changed, etc, etc. In other words -- if anybody pretends to know what's going on he's a charlatan. >If 30% loss impacts are noticeable, what should be done to eliminate the >losses or reduce their impacts on Web performance and reliability? a) put more fibers into the ground b) build better routers (want to invest?) c) connect those routers to that fiber. There's no hard physical limit on the capacity of the network which can be built with available electronics. There are at least six orders of magnitude of the growth which can be handled just by throwing money at the problem. The QoS, tag switching and other today's buzzwords are only marginal improvements at best. The only real solution always was and will be: more bandwidth. >Are packet losses due mainly to transient queue buffer overflows of user >traffic or to discards by overburdened routing processors or something else? Mostly buffer overflows. >What does Merit mean when they say that some of these losses are >intentional because of settlement issues? They may be looking for some libel lawsuits if they would care to substantiate the claim. >Are ISPs cooperating >intelligently in the carriage of Internet traffic, or are ISPs competing >destructively, to the detriment of them and their customers? I do not think any of them would kill the hen laying golden eggs. The reality is that backbone ISPs are starting to turn out new customers. There's always a lot of noise from those who just don't get TANSTAAFL, though. >Any help you can offer on these questions would be appreciated. You're welcome. Particularly it that would help to bring more reality into press reports. --vadim

3 2

Re: Questions about Internet Packet Losses
by Vadim Antonov 14 Jan '97

14 Jan '97

> The 4.3 Reno-derived TCPs (the majority) are fairly robust. >Yeah, but I doubt that they're in the majority. Recent stats indicate that >about 9% of the packets out there are using an MSS of 512. Another 1% are >using an MSS of 256. Only 6% are using 536, and 7% are using 1460. This >indicates to me that there are a whole lot of broken boxes out there still. I'm wondering why. The easiest way to implement TCP stack always was to pull the pieces off BSD code. Even oldest MS-DOS TCPs (like Phil Karn's KA9Q) are essentially that. --vadim

1 0

Re: E.E. Times?
by Nick Hilliard 13 Jan '97

13 Jan '97

> Has anyone heard of E.E. Times that did a piece on the Internet and blamed > Cisco routers for all the bottlenecks? They quote www.ra.net/statistics and > www.academ.com/nanog as well as Craig Lebowitz and Fred Baker. I have a > locally translated copy but I would be interested in seeing the original. > > It's on their web site if you wanna dig for it. A very sad article... The URL is http://www.techweb.com/se/directlink.cgi?EET19970106S0051. The first paragraph really sets the tone for the rest of the article: : A report by a federally funded networking research group here suggests : that severe Internet congestion might be linked to "routing instability," : wherein heavy traffic emanating from routers bogs down the public network. : The researchers further claim to have turned up evidence that Internet : autonomous systems supplied by the dominant maker of routers, Cisco : Systems Inc., have inherent router flaws that may exacerbate the problem. The comment at the end is pretty good, too: : Cisco routers might have IP-compatibility problems These people ought to stick to roads and bridges. Nick

4 3