Latency, TCP ACKs and upload needs
As part of the ongoing CRTC hearings, the incumbents' claim that continued implementation of the current 5/1 standard would make Canada a world leader for broadband in the future. A satellite company who currently can't even deliver its advertised 5/1 now brags its next satellite will deliver 25/1. So I have a few questions: Considering a single download TCP connection. I am aware that modern TCP stacks will rationalize ACKs and send 1 ACK for every x packets received, thus reducing upload bandwidth requirements. Is this basically widespread and assumed that everyone has that ? Also, as you split available bandwidth between multiple streams, won't ack upload requirements increase because ACK rationalisation happens far less often sicne each TCP connection has its own context for ACKs? When one considers the added latency of satellite links, does 25/1 make sense ? (I need a sanity check to distinguish between marketing spin presented to the regulator and real life) I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and they are also on a VIA Sat satellite, presumably the same vehicle that Xplornet tries to deliver its measly 5/1 on. Would all beams be identical on a satellite or can they be configured differently with a ISP adjustable rate of upload/download inside the same spectrum ? Also, when you establish a TCP connection, do most stacks have a default window size that gives the sender enough "patience" to wait long enough for the ACK ? If sender sends packet 457, doesn't get ACK in time and resends 457, doesn't that also result in reduction in window size (the very opposite of what would be needed in high latency links) ? And when the first ACK finally arrives, won't the sender assume this ACK was for the resent 457 ? Or is satellite latency low enough that stacks all have enough default "patience" to wait for ACKs and this is a non issue ? (Note Xplornet refused to answer questions on whether they operate special proxies at their gound stations to manage TCP connections to appear "close"). What i am trying to get at here is whether 25/1 on satellite, in real life with a few apps exchanging data, would actually be able to make use of the 25 download speed or whether the limited 1mbps upload would choke the downloads ?
On Apr 19, 2016, at 9:29 PM, Jean-Francois Mezei <jfmezei_nanog@vaxination.ca> wrote:
As part of the ongoing CRTC hearings, the incumbents' claim that continued implementation of the current 5/1 standard would make Canada a world leader for broadband in the future.
A satellite company who currently can't even deliver its advertised 5/1 now brags its next satellite will deliver 25/1.
So I have a few questions:
Considering a single download TCP connection. I am aware that modern TCP stacks will rationalize ACKs and send 1 ACK for every x packets received, thus reducing upload bandwidth requirements. Is this basically widespread and assumed that everyone has that ?
Also, as you split available bandwidth between multiple streams, won't ack upload requirements increase because ACK rationalisation happens far less often sicne each TCP connection has its own context for ACKs?
There’s a lot of dynamics here, including how long lived the TCP session is, the windowing behavior of the host/servers and any acceleration done in the path. There is also TCP Selective ACK which means you don’t need to explicitly ACK each frame. I’m not specifically knowledgeable about how this performs with earth to orbit and back dynamics, and hope to never personally need to debug it :) I have used satellite based internet over the atlantic ocean before and it didn’t seem that bad aside from the pricing involved. I would expect that if all endpoints are doing the right TCP options you will see reasonable performance. Make sure nobody is masking the window scaling options and such. (I have heard some carriers do this to prevent the window from getting too large). - Jared
One of the things to consider is that geostationary satellite operators operate based entirely on the economics of oversubscription. If you were to purchase a full duplex 1 Mbps x 1 Mbps connection via VSAT terminal in North America (whether C, Ku or Ka-band) you'd be looking at $2000/month or more. You'd have a fixed FDD pipe at 495ms latency end to end. 32:1 oversubscription or more is normal. It costs close to $150 million to build and launch a 5000 kilogram satellite into geostationary orbit before you build any teleport infrastructure. The entire satellite has far less aggregate data throughput capacity than two strands of singlemode. Modern Ka-band satellites as used by consumer grade VSAT services in the United States use dozens of individual spot beams. The FDD capacity in each spot beam may be exhausted or significantly oversubscribed in one geographical region, yet relatively unused in others. Compare, for example, real world user reported speeds at 10pm on Exede service in western WA state vs. somewhere in a very rural part of Wyoming. Spot beam TDMA contention ratios are carefully managed by satellite operators - they're very much aware of the issue you describe, and do their best to mitigate it. Extensive massaging of TDMA parameters in spot beams is the only way that it's economical to offer service for between $75 to $150/month even with a 2 or 3 year contract. There are a number of physics, OSI layer 1 and 2 issues to consider with satellite before discussing anything TCP related. On Tue, Apr 19, 2016 at 6:29 PM, Jean-Francois Mezei < jfmezei_nanog@vaxination.ca> wrote:
As part of the ongoing CRTC hearings, the incumbents' claim that continued implementation of the current 5/1 standard would make Canada a world leader for broadband in the future.
A satellite company who currently can't even deliver its advertised 5/1 now brags its next satellite will deliver 25/1.
So I have a few questions:
Considering a single download TCP connection. I am aware that modern TCP stacks will rationalize ACKs and send 1 ACK for every x packets received, thus reducing upload bandwidth requirements. Is this basically widespread and assumed that everyone has that ?
Also, as you split available bandwidth between multiple streams, won't ack upload requirements increase because ACK rationalisation happens far less often sicne each TCP connection has its own context for ACKs?
When one considers the added latency of satellite links, does 25/1 make sense ? (I need a sanity check to distinguish between marketing spin presented to the regulator and real life)
I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and they are also on a VIA Sat satellite, presumably the same vehicle that Xplornet tries to deliver its measly 5/1 on. Would all beams be identical on a satellite or can they be configured differently with a ISP adjustable rate of upload/download inside the same spectrum ?
Also, when you establish a TCP connection, do most stacks have a default window size that gives the sender enough "patience" to wait long enough for the ACK ?
If sender sends packet 457, doesn't get ACK in time and resends 457, doesn't that also result in reduction in window size (the very opposite of what would be needed in high latency links) ?
And when the first ACK finally arrives, won't the sender assume this ACK was for the resent 457 ? Or is satellite latency low enough that stacks all have enough default "patience" to wait for ACKs and this is a non issue ?
(Note Xplornet refused to answer questions on whether they operate special proxies at their gound stations to manage TCP connections to appear "close").
What i am trying to get at here is whether 25/1 on satellite, in real life with a few apps exchanging data, would actually be able to make use of the 25 download speed or whether the limited 1mbps upload would choke the downloads ?
On 4/19/16 6:29 PM, Jean-Francois Mezei wrote:
As part of the ongoing CRTC hearings, the incumbents' claim that continued implementation of the current 5/1 standard would make Canada a world leader for broadband in the future.
A satellite company who currently can't even deliver its advertised 5/1 now brags its next satellite will deliver 25/1.
So I have a few questions:
Considering a single download TCP connection. I am aware that modern TCP stacks will rationalize ACKs and send 1 ACK for every x packets received, thus reducing upload bandwidth requirements. Is this basically widespread and assumed that everyone has that ?
with an mss of 1460 the inbound packets for reasonably well packed flow will be 36.5x larger than the acks which are 40 bytes. sack rfc 2018 means you don't have to acknowledge all of the inbound packets.
Also, as you split available bandwidth between multiple streams, won't ack upload requirements increase because ACK rationalisation happens far less often sicne each TCP connection has its own context for ACKs?
When one considers the added latency of satellite links, does 25/1 make sense ? (I need a sanity check to distinguish between marketing spin presented to the regulator and real life)
satellite vendors use various forms of tcp acceleration which may involve among other things having middle boxes that pre-emptively send the ack, play with the window size etc.
I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and they are also on a VIA Sat satellite, presumably the same vehicle that Xplornet tries to deliver its measly 5/1 on. Would all beams be identical on a satellite or can they be configured differently with a ISP adjustable rate of upload/download inside the same spectrum ?
Also, when you establish a TCP connection, do most stacks have a default window size that gives the sender enough "patience" to wait long enough for the ACK ?
retransmission timeouts are typically measure in seconds... https://tools.ietf.org/html/rfc6298 and geostationary orbits typically have RTTs under 800ms.
If sender sends packet 457, doesn't get ACK in time and resends 457, doesn't that also result in reduction in window size (the very opposite of what would be needed in high latency links) ?
And when the first ACK finally arrives, won't the sender assume this ACK was for the resent 457 ? Or is satellite latency low enough that stacks all have enough default "patience" to wait for ACKs and this is a non issue ?
(Note Xplornet refused to answer questions on whether they operate special proxies at their gound stations to manage TCP connections to appear "close").
seems likely
What i am trying to get at here is whether 25/1 on satellite, in real life with a few apps exchanging data, would actually be able to make use of the 25 download speed or whether the limited 1mbps upload would choke the downloads ?
a geostationary orbit based connection will have a minimum latency of 492-495ms in a dedicated-carrier configuration between two earth stations, varying very slightly with the modem overhead time for FEC. In a TDMA network all bets are off, if you're in wyoming on exede and everyone is asleep, you could see very close to 500ms, you could also see 1200ms during peak hours. Or worse. Consumer grade satellite operators have only the blunt force tool of GB quota/month to prevent the TDMA capacity in a particular spotbeam from becoming a complete wasteland of modems stepping on each other (effectively reducing everyone to 32kbps or worse, and 2000ms). Go over 8 or 10GB/month and you either need to pay a lot more money, or go in the TDMA penalty box for $TIME. As usage patterns tend to rise and fall in a sine wave pattern not very different than a 10GbE circuit feeding a huge number of downstream DSL customers, many operators offer 'free' late night/evening hours that don't count towards the quota. Example: http://www.exede.com/the-free-zone-internet-details/ On Tue, Apr 19, 2016 at 7:03 PM, joel jaeggli <joelja@bogus.com> wrote:
On 4/19/16 6:29 PM, Jean-Francois Mezei wrote:
As part of the ongoing CRTC hearings, the incumbents' claim that continued implementation of the current 5/1 standard would make Canada a world leader for broadband in the future.
A satellite company who currently can't even deliver its advertised 5/1 now brags its next satellite will deliver 25/1.
So I have a few questions:
Considering a single download TCP connection. I am aware that modern TCP stacks will rationalize ACKs and send 1 ACK for every x packets received, thus reducing upload bandwidth requirements. Is this basically widespread and assumed that everyone has that ?
with an mss of 1460 the inbound packets for reasonably well packed flow will be 36.5x larger than the acks which are 40 bytes.
sack rfc 2018 means you don't have to acknowledge all of the inbound packets.
Also, as you split available bandwidth between multiple streams, won't ack upload requirements increase because ACK rationalisation happens far less often sicne each TCP connection has its own context for ACKs?
When one considers the added latency of satellite links, does 25/1 make sense ? (I need a sanity check to distinguish between marketing spin presented to the regulator and real life)
satellite vendors use various forms of tcp acceleration which may involve among other things having middle boxes that pre-emptively send the ack, play with the window size etc.
I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and they are also on a VIA Sat satellite, presumably the same vehicle that Xplornet tries to deliver its measly 5/1 on. Would all beams be identical on a satellite or can they be configured differently with a ISP adjustable rate of upload/download inside the same spectrum ?
Also, when you establish a TCP connection, do most stacks have a default window size that gives the sender enough "patience" to wait long enough for the ACK ?
retransmission timeouts are typically measure in seconds...
https://tools.ietf.org/html/rfc6298
and geostationary orbits typically have RTTs under 800ms.
If sender sends packet 457, doesn't get ACK in time and resends 457, doesn't that also result in reduction in window size (the very opposite of what would be needed in high latency links) ?
And when the first ACK finally arrives, won't the sender assume this ACK was for the resent 457 ? Or is satellite latency low enough that stacks all have enough default "patience" to wait for ACKs and this is a non issue ?
(Note Xplornet refused to answer questions on whether they operate special proxies at their gound stations to manage TCP connections to appear "close").
seems likely
What i am trying to get at here is whether 25/1 on satellite, in real life with a few apps exchanging data, would actually be able to make use of the 25 download speed or whether the limited 1mbps upload would choke the downloads ?
Others have already answered with the technical details. Let me take a stab at some more, uh, variable items. In a message written on Tue, Apr 19, 2016 at 09:29:12PM -0400, Jean-Francois Mezei wrote:
Also, when you establish a TCP connection, do most stacks have a default window size that gives the sender enough "patience" to wait long enough for the ACK ?
Your question is phrased backwards. All will wait for the ACK, the timeouts are long (30-120 seconds). The issue is that you only get one window of data per RTT, so if the window is too small, it will choke the connection. 90%+ of the stacks deployed will be too small. Modern Unix generally has "autotuning" TCP stacks, but I don't think Windows or OS X has those features yet (but I'd be very happy to be wrong on that point). Regardless of satellite uplink/downlink speeds, boxes generally need to be tuned to get maximum performance on satellite.
What i am trying to get at here is whether 25/1 on satellite, in real life with a few apps exchanging data, would actually be able to make use of the 25 download speed or whether the limited 1mbps upload would choke the downloads ?
With a properly tuned stack what you're describing is not a problem. 1460 byte payloads down, maybe 64 byte acks on the return, and with SACK which is widely deployed an ACK every 2-4 packets. You would see about 2,140 packets/sec downstream (25Mbps/1460), and perhaps send 1070 ACKs back upstream, at 64 bytes each, or about 68Kbps. Well under the 1Mbps upstream bandwidth. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
Thanks to all for the sanity check. Always depressing when you think you may have a good argument but after much reading, you find out you don't :-( BTW, in case someone knows. With the recent "beam" satellites having a lot of different focused antennas, how does the uplink work ? Does all traffic pass through a central "switch" which then directs packets to the approperiate antenna ? Would a each beam directed at a served area be paired with its own dedicated beam directed at ground station ? Or does the uplink from ground station carry traffic for multiple beams and thus becomes the bottleneck ? (Xplornet bragged about its next satellite having 20gbps capacity, but IF the uplink from ground station is also at 20gps and serves 5 beams aimed at Canada, then on average each beam only gets 4gbps ?) With regards to the "dream" of having 350 low orbit satellites covering the globe for Internet, does anyone know how the uplink will be done? Won't there be a bottleneck if in serving Canada's north, satellites currently speeding over the region have to use satellite-to-satellite links to carry information until it reaches a satellite that is over a ground station in the south ? or is it expected that ground stations will be built "near" each area to be served ? (Am trying to justify that satellite should be reserved for people truly isolated and that Nunavut communities should get undersea fibre and work need to start now because of long construction times during which satellites will fall further back in terms of capacity needs).
On 4/20/16, Leo Bicknell wrote:
Others have already answered with the technical details. Let me take a stab at some more, uh, variable items.
[.. snip lots ..]
90%+ of the stacks deployed will be too small. Modern Unix generally has "autotuning" TCP stacks, but I don't think Windows or OS X has those features yet (but I'd be very happy to be wrong on that point).
Windows has had an autotuning stack since at least Vista. Regards, Lee
Leo Bicknell <bicknell@ufp.org> wrote:
1460 byte payloads down, maybe 64 byte acks on the return, and with SACK which is widely deployed an ACK every 2-4 packets. You would see about 2,140 packets/sec downstream (25Mbps/1460), and perhaps send 1070 ACKs back upstream, at 64 bytes each, or about 68Kbps. Well under the 1Mbps upstream bandwidth.
Note that with delayed ACKs (RFC 1122) there is an ACK for every other packet; SACK should do better than that. Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ - I xn--zr8h punycode Humber, Thames: Northwest, veering north or northeast, 4 or 5. Slight or moderate. Fair. Good.
Wed, Apr 20, 2016 at 07:27:53AM -0700, Leo Bicknell wrote:
90%+ of the stacks deployed will be too small. Modern Unix generally has "autotuning" TCP stacks, but I don't think Windows or OS X has those features yet
OS X since ~10.5 has autotuning, here are some hints from ESnet https://fasterdata.es.net/host-tuning/osx/ Personally never tried this on the sat-type RTT. As explained, scaling factor of 3 is a limiting one for high-performance transfers. For sat links may limit the things too, since 64K * 2^3 / 0.9 ms RTT * 8 ~ 4.5 Mbit/sec, but one's mileage may vary. And, of course, packet loss can turn down this BDP-derived speed drastically. -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
On 20/04/16 16:27, Leo Bicknell wrote:
90%+ of the stacks deployed will be too small. Modern Unix generally has "autotuning" TCP stacks, but I don't think Windows or OS X has those features yet (but I'd be very happy to be wrong on that point). Regardless of satellite uplink/downlink speeds, boxes generally need to be tuned to get maximum performance on satellite.
Windows also has TCP buffers auto-tuning since Windows 7/Vista up to 16MB, however only receiver-side tuning on their client versions, for sender-side tuning you will need a server version. That means uploading stuff from a regular windows "client" machine to a remote host with a large RTT will be very slow. -- Chris
On 4/19/16, Jean-Francois Mezei <jfmezei_nanog@vaxination.ca> wrote:
As part of the ongoing CRTC hearings, the incumbents' claim that continued implementation of the current 5/1 standard would make Canada a world leader for broadband in the future.
A satellite company who currently can't even deliver its advertised 5/1 now brags its next satellite will deliver 25/1.
Take a look at https://www.rfc-editor.org/rfc/rfc3449.txt TCP Performance Implications of Network Path Asymmetry
So I have a few questions:
Considering a single download TCP connection. I am aware that modern TCP stacks will rationalize ACKs and send 1 ACK for every x packets received, thus reducing upload bandwidth requirements. Is this basically widespread and assumed that everyone has that ?
The usual defaults are to ack every other packet & wait 200ms before acking a single packet.
Also, as you split available bandwidth between multiple streams, won't ack upload requirements increase because ACK rationalisation happens far less often sicne each TCP connection has its own context for ACKs?
Yes. And multiple streams will interfere with each other. It used to be popular to split a download into multiple streams but I thought that went out of style now that tcp window scaling is generally enabled by default.
When one considers the added latency of satellite links, does 25/1 make sense ? (I need a sanity check to distinguish between marketing spin presented to the regulator and real life)
I noticed that in the USA, EXEDE Satellite advertises 12/3 plans and they are also on a VIA Sat satellite, presumably the same vehicle that Xplornet tries to deliver its measly 5/1 on. Would all beams be identical on a satellite or can they be configured differently with a ISP adjustable rate of upload/download inside the same spectrum ?
Also, when you establish a TCP connection, do most stacks have a default window size that gives the sender enough "patience" to wait long enough for the ACK ?
There's an initial timeout that's on the order of 3-5 seconds. Once the xfer starts ... I've forgotten the details, but the retransmission timer is based on the "smoothed round trip time" (srtt).
If sender sends packet 457, doesn't get ACK in time and resends 457, doesn't that also result in reduction in window size (the very opposite of what would be needed in high latency links) ?
The window size is what the receiver advertises, the congestion window (cwnd) is what the sender computes based on the advertised window size & packet loss. cwnd is what gets reduced when there's packet loss. & yes, reducing cwnd is bad for thruput as is not having selective acks (sack) enabled on the receiver.
And when the first ACK finally arrives, won't the sender assume this ACK was for the resent 457 ?
The sender keeps track of the 'smoothed round trip time' (srtt). Since it can't tell if the ack is for the original or retransmitted pkt, it's not supposed to use that ack for updating the rtt
Or is satellite latency low enough that stacks all have enough default "patience" to wait for ACKs and this is a non issue ?
should be a non-issue
(Note Xplornet refused to answer questions on whether they operate special proxies at their gound stations to manage TCP connections to appear "close").
What i am trying to get at here is whether 25/1 on satellite, in real life with a few apps exchanging data, would actually be able to make use of the 25 download speed or whether the limited 1mbps upload would choke the downloads ?
dunno. Assuming the bandwidth is available, I suspect you could get 25Mb/s doing something like downloading a movie from archive.org but for anything interactive like web surfing / gaming I'd bet no - but because of latency, not the 1Mb/s uplink speed. Regards, Lee
On Tue, 19 Apr 2016, Jean-Francois Mezei wrote:
Considering a single download TCP connection. I am aware that modern TCP stacks will rationalize ACKs and send 1 ACK for every x packets received, thus reducing upload bandwidth requirements. Is this basically widespread and assumed that everyone has that ?
Typically you'll see one ACK per 2 packets, so you need approximately 1:50 bytes up/down ratio for the ACKs. It's possible to have middle boxes suppress some ACKs, please see thread here: https://www.ietf.org/mail-archive/web/aqm/current/msg01482.html -- Mikael Abrahamsson email: swmike@swm.pp.se
participants (10)
-
Chris Welti -
Eric Kuhnke -
Eygene Ryabinkin -
Jared Mauch -
Jean-Francois Mezei -
joel jaeggli -
Lee -
Leo Bicknell -
Mikael Abrahamsson -
Tony Finch