Paul A Vixie writes...

the whole idea of opening multiple tcp sessions "to get better performance" is just horrible in any case. http/1.1 can do serial multiplexing when talking to a particular server, and browsers have now got no excuse at all for opening more than one connection unless it's to more than one server.

I'm aware of the existance of a protocol for HTTP to persistent connections where one object/document can be downloaded, and after that is done, another can be requested and downloaded, and again as needed. But I know not of any protocol for multiplexing so that I can have concurrency of the transfers of multiple images at the same time. Is there an RFC for this or is it some other kind of document? -- Phil Howard | a4b7c9d8@dumb4ads.edu w5x2y1z8@dumb7ads.net ads6suck@anywhere.org phil | stop9738@anyplace.org w2x1y8z3@anyplace.org no12ads2@no6place.net at | suck1it1@anyplace.net end5ads8@dumb0ads.org blow6me7@anywhere.edu milepost | stop1939@no0place.edu no7spam8@nowhere9.net crash622@lame2ads.edu dot | no2spam0@nowhere3.com ads9suck@spam1mer.org eat51me4@dumbads4.edu com | eat38me6@nowhere7.com end8ads2@dumb8ads.edu end2ads7@spam1mer.net

that's not what i meant by "serial multiplexing". but can someone please explain why anyone would want to do this? you don't get the bits faster by opening multiple connections, now that persistent http allows you to avoid the inter-object delay that used to be induced by doing a full syn/synack/ack for each object. does seeing the GIFs fill in in parallel really make that much difference, if the total page fill time is going to be the same?

You're assuming that the goal is to get _all_ the bits in the shortest amount of time. And in some cases that is the goal. In others it is not. By loading the images in parallel, with the initial part of the image files being a fuzzy approximation, you get to see about where every button is located, and in many cases you know exactly what it is, and you can click on them as soon as you know where to go. The total page fill time isn't that much of an issue when people are not in a hurry to go on to the appropriate page. In about half of my surfing I never let the pages finish. Many pages are designed with this in mind because they build their images to load low resolution first. What might work better is to just make the whole page be one giant image and that way it can pre-load in low res _and_ you get it in one connection. But now that means much _more_ total bandwidth involved. There is also a psychological perception of speed when images load in parallel. When someone complains about the slowness, and is given the low MTU solution, they often end up being happier. I still say the solution is a way to transmit multiple images in parallel over a single connection. A multiplexer that allowed concurrency would be one way. An image format that allowed scattered objects could be another. -- Phil Howard | no7way61@anywhere.edu blow5me3@dumbads2.com no7way02@no21ads7.net phil | stop9ads@noplace9.edu die7spam@anywhere.net ads8suck@anywhere.com at | end0it44@no6place.net crash855@s8p3a0m9.org crash905@dumb1ads.com milepost | suck1it2@lame6ads.org suck6it5@dumbads0.com no60ads5@nowhere3.com dot | no5spam5@lame0ads.net no9way14@anywhere.edu no9way20@s7p1a2m0.net com | eat05me5@spam2mer.com stop5ads@nowhere2.com w9x4y3z6@spammer8.com

At 12:36 AM 2/8/98 -0600, Phil Howard wrote:

By loading the images in parallel, with the initial part of the image files being a fuzzy approximation, you get to see about where every button is located, and in many cases you know exactly what it is, and you can click on them as soon as you know where to go.

things like buttons tend to be very very small. the connections never really get out of slowstart. loading them all in parallel would help, somewhat, to fully utilize the b*d of the path. but it isn't very adaptive. better would be to use a single connection to load all the images. the tcp will rather quickly settle on the 'best' bandwidth of the link and your overall throughput will then be optimal.

On Sat, 7 Feb 1998, Paul A Vixie wrote:

...

I'm aware of the existance of a protocol for HTTP to persistent connections where one object/document can be downloaded, and after that is done, another can be requested and downloaded, and again as needed.

that's what i meant by "serial multiplexing".

Ok, there seems to be a lot of confusion around here from some people (no, not you Paul) on how HTTP works so a summary is in order. Network operators have to understand how common protocols impact the network. Once upon a time, after other things, there became HTTP/1.0. It requires a new connection for each transfer. At one time you were only requesting a couple of URLs for each document, so that was fine. Then there were more graphics on the web page. It was easily observed that something was very inefficient when making multiple requests one after the other. Clearly, if you have a lock-step protocol using only one simultaneous connection, you have at least a RTT between each request. After you add the TCP handshake, you actually end up with 2 RTTs between the response to each request. On a LAN, that isn't a big deal. On a higher latency connection, it is a huge problem and results in poor efficiency. So clients started making multiple connections in an attempt to more efficiently utilize bandwidth. This worked, both in theory and in practice. However, you still end up with the RTT related slowness of establishing (non-T/TCP) TCP connections. People also may have figured out that large numbers of short flows are not as easily controlled by TCP's congestion control as small numbers of large flows. Hence the idea of "keepalive" (or, using the later name, "persistent") connections. A standard of sorts was established to extend HTTP/1.0 to allow for these; it allowed a client to make multiple requests in a single TCP connection. Clients still made multiple connections, though, since many servers didn't support persistent connections and they were already doing it and there are other advantages. Now you chop the inter-request delay down to one RTT by avoiding the TCP handshake. This also gives a win by putting less load on the server (lower TCP connection rate) and a loss because the server will have connections hanging around waiting for a (relatively short) timeout just in case the client decides to make another request on that connection. Then comes HTTP/1.1. It formalized the idea of a persistent connection (with a slightly different syntax) into the spec. It also allows for "pipelined" connections; with appropriate care, the client can send requests for multiple documents before it finishes receiving the first. This requires careful attention in the protocol and server and client code to get right; out of the very small field of deployed HTTP/1.1 clients and servers, few have this completely right yet. Even Apache, with a (IMBiasedO) darn good HTTP/1.1 implementation with very few bugs compared to most initial deployments of HTTP/1.1 (eg. MSIE 4), isn't yet perfect in any released version. The HTTP/1.1 spec also includes some comments about limiting multiple connections. The state of the art in terms of deployed protocols right now is persistent pipelined connections; most clients don't implement them yet, but they are getting there. Yet even with this, there is a perceived value to multiple connections. Why is that? There are several reasons: - that is how it has been done, so we better keep doing it - can grab more bandwidth on congested links by using more flows. In real life, this is a significant factor, more so when you get to ISDN and higher speeds. It would probably be better for everyone if everyone stopped using multiple connections, but if some do and some don't, those that don't lose out. This advantage could be eliminated by various algorithms. - is not acceptable to have a bunch of small responses stuck behind one huge response. - if you are a proxy, this becomes even more critical. If you only use one connection to each origin server, if one client requests a huge document from that server, anyone else wanting to make requests to that server would have to wait for the transfer to finish. These reasons, both the technically valid and the invalid ones, can not be dismissed out of hand.

...

But I know not of any protocol for multiplexing so that I can have concurrency of the transfers of multiple images at the same time. Is there an RFC for this or is it some other kind of document?

that's not what i meant by "serial multiplexing". but can someone please explain why anyone would want to do this? you don't get the bits faster by opening multiple connections, now that persistent http allows you to avoid the inter-object delay that used to be induced by doing a full syn/synack/ack for each object. does seeing the GIFs fill in in parallel really make that much difference, if the total page fill time is going to be the same?

Yes, it does make a difference. A real world nanog example? Cisco docs. You may have a 300k HTML file with a bunch of graphics in it. You want to read something partway down the file and it has an image you need to see. Do you want to wait for the whole HTML to download, then for each image to download until it gets to yours or do you want to be able to start reading ASAP? The basic idea is that humans are reading documents, and they take some finite time to read and normally start at the top and work down. The faster you can display the start of it, complete with the graphics, etc. that are contained in that part, the faster the user can get on with reading the document. Persistent connections with pipelining come close to the peak of efficiency from the network perspective (well, the perspective of anything sitting on top of TCP), but not from the much harder user perspective. Dealing with this is one of the goals of HTTP-NG. See: http://www.w3.org/Protocols/HTTP-NG/ for some details. I can't comment with more details or on how HTTP-NG is going because I can't be involved, even as a lurker because it is the W3C doing it and I don't work for the right people. One other additional gain from a multiplexed protocol is the ability to do graceful aborts of a request in progress, instead of having to dump the TCP connection and make a new one. That is a big win, iff you don't have too much buffering going on at the sending host between the application and the network. NNTP is worse in this way, but it still impacts HTTP especially if you want to progress to the idea of a single longer-term connection for a whole string of requests traversing a web site. HTTP-NG with multiplexed connections (that isn't the only feature of it, just the relevant one) will fail to convince vendors to switch to using one multiplexed connection unless the Internet either has more bandwidth than anyone can use (not likely) or backbone bandwidth grows far faster than user bandwidth (more possible, but not likely right now) or ends up giving the same net bandwidth to n simultaneous TCP connections as it gives to one multiplexed TCP connection (not possible to do fairly without unfairly penalizing things like proxys). The solution to all this? Well, that (and a lot of more generlized stuff) is the whole point of a whole bunch of research. There are known methods to get better than this, but they are not deployed.

Paul A Vixie writes:

explorer and navigator have both done this since their respective 3.0's. however, they both still open N (where N defaults to 4) connections to the same server rather than depending on persistence and limiting themselves to 1 connection per server. so they will reuse their N connections for more than one fetch if there are more than N objects to be fetched from some server to fill out one "page". phil howard says this is a feature, but now that the size of a graphic can be given in its enclosing page it is no longer nec'y to get the graphic objects' headers before laying out the page, and i think that for the good of the network and the origin servers it would be nicer if these browsers could learn to serialize their fetching a LOT more.

Merely laying out the page isn't enough. It helps, and unfortunately some browsers (e.g. explorer) don't seem to even use this feature correctly. Having a bunch of blank positions where buttons are supposed to be doesn't give one any idea a button is supposed to be there, or even where "there" is, until some stuff, either the first pixels of that button, or the buttons around it, are getting loaded. Seeing only the top button of a table of buttons doesn't help to find where the active space is for the green one you know you always click on, or the one with the person's head. One approach that can be used to get around this is to make all the images be very small low res only files, and include some Javascript that will detect when those are loaded and then start the medium res loading, and after that, the high res loading. But I would not want to depend on people who have degrees in graphical arts to be in a position to manage network bandwidth controls, so other fundamental solutions are still needed. My multiplexing protocol would need a persistent TCP connection. There would be 254 different subchannels. A byte code of 255 is a flag to the multiplex control. 255,255 means a real 255 in the current subchannel. 255,0 means EOF in the current subchannel. 255,N means a switch to subchannel N (1-254). The sender can determine how many bytes can be sent in one subchannel before switching to another. If more features might be required, such as per subchannel flow control, then more codes could be reserved (it might be good to reserve all of 200-254 for now). AFAIK, no valid HTTP would begin with a 255,1 so this multiplexing would likely be detectable dynamically against HTTP. -- Phil Howard | suck8it5@no57ads3.com w1x8y9z0@no0place.com no95ads7@anyplace.org phil | eat90me5@noplace5.net w5x2y1z6@noplace0.net end5it87@lame5ads.net at | ads9suck@spammer5.com no67ads8@anyplace.net end1it03@spam0mer.edu milepost | ads1suck@nowhere3.net end5ads3@anyplace.org stop1727@nowhere8.com dot | stop6it4@anywhere.edu w5x2y9z4@spammer8.org end0it35@noplace8.org com | stop6ads@noplace2.com stop0it9@anyplace.edu stop9ads@dumbads1.net

At 12:15 PM 2/8/98 -0600, Stephen Sprunk wrote:

For a perfect example of HEIGHT, WIDTH, and ALT parameters, check out the cisco main page. I can't find a good LOWSRC example offhand.

http://www.netproductions.com/npmenu.html

Don't invent a new transfer protocol when simply creating better data will fix the problem.

Good idea.

Stephen Sprunk "Oops." Email: stephen@sprunk.org

TTFN, patrick ************************************************************** Patrick W. Gilmore voice: +1-650-482-2840 Director of Operations, CCIE #2983 fax: +1-650-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "Tomorrow's Performance Today" **************************************************************

On Sun, 8 Feb 1998, Perry E. Metzger wrote:

Phil Howard writes:

...

By loading the images in parallel, with the initial part of the image files being a fuzzy approximation, you get to see about where every button is located, and in many cases you know exactly what it is, and you can click on them as soon as you know where to go.

By loading the images in parallel over multiple TCP connections, you also totally screw the TCP congestion avoidance mechanisms, and hurt the net as a whole, especially given how prevalent HTTP is these days. Unfortunately, as has been seen here, very few people working with the net these days actually understand the details of things the net depends on, and TCP congestion avoidance is one of them.

HTTP 1.1 allows multiplexing in a single stream, and even (amazingly

Once again, HTTP/1.1 does _not_ allow multiplexing multiple transfers simlultaneously in a single TCP connection. Multiple responses are serialized.

enough) ends up working faster in practice than multiple TCP connections.

I have seen nothing supporting that assertion for high latency, medium (aka. the Internet on a good day) packet loss connections. The discussion at: http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html shows some interesting information about the wins of persistent connections and piplining, however their tests only went as far as including a 28.8k local dialup, which does not simulate the "average user" going to the "average web site". If you are dropping packets and congestion contol is coming into play, you may see more impact when using one connection that is temporarily stalled than multiple connections, with the hope that at least one will be sending at any time. I am not aware of any research to support (or deny, for that matter) this view, however AFAIK there is a general lack of published research on the interaction between HTTP w/pipelined and persistent connections and the Internet. As I noted before, total transfer time for all the responses that make up one document in the client is not the metric that client vendors are trying to optomize and is not what most users care about. On Sat, 7 Feb 1998, Paul A Vixie wrote:

...

The state of the art in terms of deployed protocols right now is persistent pipelined connections; most clients don't implement them yet, but they are getting there.

explorer and navigator have both done this since their respective 3.0's.

As I have already pointed out to Paul, but think it deserves to be emphasized because it is not apparent to many, they do _not_ do pipelined connections but only persistent connections. You can not do reliable pipelined connections with HTTP/1.0. The difference between pipelined persistent and non-pipelined persistent connections (in the case where there are multiple requests to the same server in a row) is one RTT per request plus a possible little bit from merging the tail of one response with the head of another into one packet. Also worthy of note is that the only widespread client that implements HTTP/1.1 is MSIE4, and even it is botched badly, although not as badly as 4.0b2 was. (eg. it sometimes sent 1.1 requests but would only accept 1.0 responses)

In a previous episode Phil Howard said... :: :: Marc Slemko writes: :: :: > Once again, HTTP/1.1 does _not_ allow multiplexing multiple transfers :: > simlultaneously in a single TCP connection. Multiple responses are :: > serialized. :: :: I think the confusion here is due to Paul's use of the term "serial :: multiplexing" where he qualified it with "serial" to indicate that :: one-at-a-time situation. When I read it I wasn't sure if "serial" :: meant to be that or meant to describe a kind of multiplexing over a :: serial stream. But given the HTTP 1.1 that I knew had a persistent :: connection that allowed additional requests, I suspected that he was :: referring to this. But the term "multiplexing" by itself implies :: concurrency. While at the microsecond level it is one at a time, :: but each channel isn't completed in those short durations. My worry :: was that others might have assumed there was some new true multiplexing :: protocol for HTTP. I've not heard of one, but even I wondered of one :: I might have not heard of (and I don't keep track of all the protocols :: out there). At the risk of introducing meaningful background literature: ftp://ds.internic.net/rfc/rfc2068.txt I direct folks to 14.36.1 "Byte Ranges" which when interleaved with pipelined requests comes very close to achieving client-driven multiplexing that I'd suggest from a UI pov will behave much better than the multiple connections method (eliminating the cost of tcp congestion control but at the cost of some application protocol overhead). -P

On Mon, 9 Feb 1998, Phil Howard wrote:

Patrick McManus writes:

...

At the risk of introducing meaningful background literature: ftp://ds.internic.net/rfc/rfc2068.txt

I direct folks to 14.36.1 "Byte Ranges" which when interleaved with pipelined requests comes very close to achieving client-driven multiplexing that I'd suggest from a UI pov will behave much better than the multiple connections method (eliminating the cost of tcp congestion control but at the cost of some application protocol overhead).

As a server implementor, let me simply say this makes no sense and is a perversion of what byte ranges are intended for. You will end up repeating the request headers (which can be sizable) far too often, will put far too much load on the server, will break whenever you get dynamic content, etc.

More than application overhead, I suspect the biggest problem with this otherwise good idea is that it won't be implemented corrently by the browsers or the servers.

For example on the server end, it would see multiple requests for the same object, at different byte ranges. If that object is being created on the fly by a program process (e.g. CGI) the browser won't have a clue of the size.

What is the correct behaviour of the server if the request is made for bytes 0-2047 of an object which invokes a CGI program to create that object? Obviously it can send the first 2048 bytes, but then what?

For dynamic content, it normally has to send the whole document. That is a legal response to a byte range request.

On Sat, Feb 07, 1998 at 11:52:02PM -0700, Marc Slemko wrote:

Yet even with this, there is a perceived value to multiple connections. Why is that? There are several reasons:

- that is how it has been done, so we better keep doing it - can grab more bandwidth on congested links by using more flows. In real life, this is a significant factor, more so when you get to ISDN and higher speeds. It would probably be better for everyone if everyone stopped using multiple connections, but if some do and some don't, those that don't lose out. This advantage could be eliminated by various algorithms. - is not acceptable to have a bunch of small responses stuck behind one huge response. - if you are a proxy, this becomes even more critical. If you only use one connection to each origin server, if one client requests a huge document from that server, anyone else wanting to make requests to that server would have to wait for the transfer to finish.

And yet, and yet... Marc overlooks the most important reason why browsers make multiple connections: functionally, this gets the page _usable_ by the user (which is, after all the _point_ of the whole affair) more quickly. Given the current design environment of web pages, it is useful to have more than one image loading simultaneously, because some of those images are controls, not just pretty pictures, and if you can whip the small stuff up while the big stuff is still loading, that's a win. Everyone remember: the whole point is making the net usable for the users. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "Two words: Darth Doogie." -- Jason Colby, Tampa Bay, Florida on alt.fan.heinlein +1 813 790 7592 Managing Editor, Top Of The Key sports e-zine ------------ http://www.totk.com

Mike Hedlund writes:

To me it doesnt really matter, cause i cant remember the last time i followed, or even took a second glace at an add banner. But telling somone whos paying hundreds of thousands of dollars on add space, even at most 10-20% more of the people who started downloading the image didnt view it.. thats makes it a little more important. :)

That's why most ads are at the top. -- Phil Howard | end8it06@nowhere9.com suck9it0@anyplace.org die6spam@no5place.edu phil | blow9me2@noplace8.com die9spam@spam7mer.org no4way56@no6place.com at | eat7this@noplace9.com crash508@spam8mer.net die2spam@spam6mer.edu milepost | blow7me0@s1p4a7m9.com stop0ads@no5place.net w9x2y7z3@no77ads6.edu dot | no9way50@dumb2ads.org ads3suck@noplace9.edu w2x8y3z1@no57ads2.net com | no00ads3@nowhere1.edu a4b2c6d1@dumbads6.org blow6me2@spammer0.com

On Sat, Feb 07, 1998 at 09:06:41PM -0800, Paul A Vixie wrote:

the whole idea of opening multiple tcp sessions "to get better performance"

Paul, I believe you mispelled "to get better user perceived performance". Anyone who's worked with SNA knows that if the 3270 paints the screen in 1 second, after waiting 8 seconds, then it's obviously better than the serial termina that starts painting at 3 seconds, and is done at 7: "why, obviously: the 3270 painted the screen 4 times faster!"

is just horrible in any case. http/1.1 can do serial multiplexing when talking to a particular server, and browsers have now got no excuse at all for opening more than one connection unless it's to more than one server.

You're suggesting that 1.1 can multiplex more than one item retrieval over a single connection? Interleaved, I mean, not in series. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "Two words: Darth Doogie." -- Jason Colby, Tampa Bay, Florida on alt.fan.heinlein +1 813 790 7592 Managing Editor, Top Of The Key sports e-zine ------------ http://www.totk.com