why upload with adsl is faster than 100M ethernet ?
Hi, I met a question with upload speed and network access speed. One of our customer lease two lines from us. One is 2Mbps ADSL line the other is 100Mbps fiber ethernet link. The customer needs to upload files to server in Japan usually. Now, the customer complaint that the upload speed of ADSL is much slower than fiber link. For a 5MB file, it takes 420 seconds with fiber link to finish uploading while the time for ADSL is 170 seconds. There is no difference in routing path between ADSL far end and fiber ethernet far end other than the access method. ( from the first acess router ). We measured the latency between our core router and customer's computer, and find there is no packet loss between with both line while latency on ADSL is 0.3ms higher than fiber ethernet. And, no link along the path inside our network is over burdened. That is, bottleneck locates somewhere outside our network. And there is asymetric route between our network and Japan server. But, why TCP throughput experience so much difference between ADSL acess and fiber link access? Thanks. Joe __________________________________________________ Do You Yahoo!? Log on to Messenger with your mobile phone! http://sg.messenger.yahoo.com
On Fri, 15 Oct 2004, Joe Shen wrote:
But, why TCP throughput experience so much difference between ADSL acess and fiber link access?
Ask the customer to install Ethereal on the computer sending the file and dump all data on the TCP session and send it to you, then use the Ethereal function "speed over time" or what it's called. It'll use the TCP information in the packet dump to determine over time what speed the transfer was. If the speed is saw-toothing over time you might have a very aggressive policer on the way which is hit by the faster line, but not with the slower one. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Fri, 15 Oct 2004, Joe Shen wrote:
Hi,
I met a question with upload speed and network access speed.
One of our customer lease two lines from us. One is 2Mbps ADSL line the other is 100Mbps fiber ethernet link. The customer needs to upload files to server in Japan usually. Now, the customer complaint that the upload speed of ADSL is much slower than fiber link. For a 5MB file, it takes 420 seconds with fiber link to finish uploading while the time for ADSL is 170 seconds. There is no difference in routing path between ADSL far end and fiber ethernet far end other than the access method. ( from the first acess router ).
There is something wrong with both scenarios. A 5 Mbyte file is 40 megabits. With overhead, it should transfer in about one-half second over a 100 Mbps ethernet connection and somewhat less than 30 seconds on a 2Mbps connection. Look for duplex mismatch or something similar.
We measured the latency between our core router and customer's computer, and find there is no packet loss between with both line while latency on ADSL is 0.3ms higher than fiber ethernet.
The ADSL should be substantially higher considering just serialization delay.
And, no link along the path inside our network is over burdened. That is, bottleneck locates somewhere outside our network. And there is asymetric route between our network and Japan server.
Oh! There's another WAN link in the picture! What are the MTU settings? Are the packets being fragmented? Iis a firewall blocking all ICMP somewhere including path MTU discovery?
But, why TCP throughput experience so much difference between ADSL acess and fiber link access?
We need more details as to the entire network. Ethereal captures at both ends would be a good start. What is the connection in Japan? Note that this isn't exactly within the realm of the NA(North American) Network Operators Group, but the photons don't respect political boundaries so you may get appropriate answers here. -- Jay Hennigan - CCIE #7880 - Network Administration - jay@west.net WestNet: Connecting you to the planet. 805 884-6323 WB6RDV NetLojix Communications, Inc. - http://www.netlojix.com/
Hi, the network path is: |-(ADSL)--------\ customer/ --Edge_router---...---Japan Server \-(100Methernet)-/ So, from edge_router to Japan server the path is identical.
There is something wrong with both scenarios.
A 5 Mbyte file is 40 megabits. With overhead, it should transfer in about one-half second over a 100 Mbps ethernet connection and somewhat less than 30 seconds on a 2Mbps connection.
Yes. But, for ftp TCP control real end-to-end transmission speed. I attached a monitor computer to our core router along the path between customer's site and server. Monitoring computer ping customer's site by targeting both ends of ADSL line and ethernet line. The measuring is scheduled 20packet per 20seconds, we also ping each hop address along the path to server. The result shows there is no packet loss along from monitoring computer to customer site, but packet loss increase at a special hop along the path to server in japan. So, we think the bottleneck is not inside our network. And, TCP connection between customer's computer and its server should be affacted by same bottleneck. So, the uploading speed should be similar (?), but it shows so much difference!
Look for duplex mismatch or something similar.
I disable autonegotiation of ethernet. So, there is no such situation.
Oh! There's another WAN link in the picture! What are the MTU settings? Are the packets being fragmented? Iis a firewall blocking all ICMP somewhere including path MTU discovery?
the measurement is taken without firewall in customer site. And, no packet filtering is enabled inside our network.
Note that this isn't exactly within the realm of the NA(North American) Network Operators Group, but the photons don't respect political boundaries so you may get appropriate answers here.
Thanks for all your helps. To me, knowledge should be shared around world no matter where people live. Luckly, NANOG/RIPE and some other group open to the wolrd. regards Joe __________________________________________________ Do You Yahoo!? Log on to Messenger with your mobile phone! http://sg.messenger.yahoo.com
On 15-okt-04, at 10:14, Joe Shen wrote:
The measuring is scheduled 20packet per 20seconds, we also ping each hop address along the path to server. The result shows there is no packet loss along from monitoring computer to customer site, but packet loss increase at a special hop along the path to server in japan.
Pinging with 1 second intervals only gives you part of the picture. Some types of problems only show up when the traffic load increases. Try ping -f or ping -l. If you have packet loss, TCP performance will suffer. A possible explanation for your situation is that the packet loss increases as bandwidth use increases (this can easily happen with duplex problems or when rate limiting is in effect) and since ethernet can deliver more packets within a shorter time, it's more likely to experience several lost packets in a row. When this happens, TCP goes into "slow start" and your performance is gone. With ADSL, the packets don't come in as fast so it's less likely that several successive packets are lost, and only a single packet means "congestion avoidance" which also slows down your session but not quite as badly.
Look for duplex mismatch or something similar.
I disable autonegotiation of ethernet. So, there is no such situation.
It's generally a bad idea to turn of ethernet autonegotiation unless the equipment at the other side doesn't support it. Did you set the link to full duplex? Full duplex and no autonegotiation on one end with autonegotiation turned on at the other will create the situation where one end uses full duplex and the other half duplex. (Because the other end sees no autonegotiation, it thinks it's connected to a dumb hub = half duplex.) With a simple ping you generally don't see this, but when traffic increases and both sides start to send packets at the same time you get packet loss. (Excessive collions errors on one side, CRC errors on the other side.)
It's generally a bad idea to turn of ethernet autonegotiation unless the equipment at the other side doesn't support it.
Yes, we've checked the configuration, both access router interface and customer's ethernet interface are forced to be (100Mbsp, full duplex). And, there is no CRC records shows. Your explanation on TCP behavior seems reasonable, but why TCP over fast access line express so much packet loss than slow access line ? Do WindowsXP/Win2k determine its startup sending window according to access speed or path MTU ? regards Joe __________________________________________________ Do You Yahoo!? Log on to Messenger with your mobile phone! http://sg.messenger.yahoo.com
On 15-okt-04, at 12:04, Joe Shen wrote:
Your explanation on TCP behavior seems reasonable, but why TCP over fast access line express so much packet loss than slow access line ? Do WindowsXP/Win2k determine its startup sending window according to access speed or path MTU ?
I don't think there is much of a difference in the actual window size. But assuming 1500 byte packets, you can transmit a packet over 100 Mbps every 121 microseconds, while over 2 Mbps this is every 6 milliseconds. Now suppose that somewhere along the path there is a link that has some congestion. This means that most of the time, that link is busy so when a packet comes in, it must wait until it can be transmitted. This isn't much of an issue, unless so many packets come in that the buffers fill up. After that, any additional packets will be dropped (tail drop) until more packets have been transmitted and there is buffer space again. In this situation, it's likely that the 100 Mbps link will deliver several packets while the buffer is full, so that several successive packets are dropped. But the ADSL 6 ms spacing makes sure that there is always time for the buffer to drain between two packets. Tail drops will still happen, but it's less likely that several successive packets are dropped. And as I wrote in my previous message, TCP will slow down slightly for each individual packet dropped, but it will slow down dramatically when three successive packets are dropped. If congestion is indeed the cause, it would be helpful to increase the buffer space and turn on random early discard (or detect, RED) so packet drops increase gradually as the buffers fill up and tail drops are avoided. However, the cause can also be rate limiting. Rate limiting is deadly for TCP performance so it shouldn't be used on TCP traffic.
--On 15 October 2004 13:33 +0200 Iljitsch van Beijnum <iljitsch@muada.com> wrote:
However, the cause can also be rate limiting. Rate limiting is deadly for TCP performance so it shouldn't be used on TCP traffic.
Add "unless appropriate shaping is performed prior to the rate-limiting with the parameters carefully tuned to the rate-limiting" You can also see an effect similar to rate-limiting from inadequate buffering where there is a higher input media speed than output. I can't remember what the tool is now, but there used to be a tool which worked like ping but sent a udp stream at a given rate per second and told you about packet drops, and also allowed for some parameter to be tweaked to give stochastic variation in interpacket delay (i.e. massive jitter). You could use this to show inadequate buffering on gigabit interfaces where a 2Mb/s stream would get through, but if you wound up the jitter sufficiently, a whole burst of packets would arrive together and a gigabit interface with (deliberately) misconfigured buffers would then drop packets. Alex
I've used QCheck (freeware) for testing response time, TCP and UDP performance, and packet loss. Works well. You need to install the enpoint on the target machine. Check out: http://www.ixiacom.com/products/performance_applications/pa_display.php?skey =pa_q_check -Mike McSpedon Arrow Global Data Communications -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Alex Bligh Sent: Friday, October 15, 2004 8:25 AM To: Iljitsch van Beijnum; Joe Shen Cc: NANGO; Alex Bligh Subject: Re: why upload with adsl is faster than 100M ethernet ? <snip> I can't remember what the tool is now, but there used to be a tool which worked like ping but sent a udp stream at a given rate per second and told you about packet drops, and also allowed for some parameter to be tweaked to give stochastic variation in interpacket delay (i.e. massive jitter). You could use this to show inadequate buffering on gigabit interfaces where a 2Mb/s stream would get through, but if you wound up the jitter sufficiently, a whole burst of packets would arrive together and a gigabit interface with (deliberately) misconfigured buffers would then drop packets. Alex
Alex Bligh wrote:
--On 15 October 2004 13:33 +0200 Iljitsch van Beijnum <iljitsch@muada.com> wrote:
However, the cause can also be rate limiting. Rate limiting is deadly for TCP performance so it shouldn't be used on TCP traffic.
Add "unless appropriate shaping is performed prior to the rate-limiting with the parameters carefully tuned to the rate-limiting"
You can also see an effect similar to rate-limiting from inadequate buffering where there is a higher input media speed than output.
The following link provided a great overview of tcp throughput http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.htm
I can't remember what the tool is now, but there used to be a tool which worked like ping but sent a udp stream at a given rate per second and told you about packet drops, and also allowed for some parameter to be tweaked to give stochastic variation in interpacket delay (i.e. massive jitter). You could use this to show inadequate buffering on gigabit interfaces where a 2Mb/s stream would get through, but if you wound up the jitter sufficiently, a whole burst of packets would arrive together and a gigabit interface with (deliberately) misconfigured buffers would then drop packets.
Alex
netiq's qcheck (the artist formerly known as ganymede) is a utility we commonly use to prove to customers that they are indeed receiving correct throughput. http://www.ixiacom.com/products/performance_applications/pa_display.php?skey... -Gordon
On Oct 15, 2004, at 14:24 Uhr, Alex Bligh wrote:
I can't remember what the tool is now, but there used to be a tool which worked like ping but sent a udp stream at a given rate per second and told you about packet drops,
iperf? (works for both, tcp and udp) Regards, Marc -- Marc Binderberger <marc@sniff.de>
On Fri, 15 Oct 2004, Iljitsch van Beijnum wrote:
However, the cause can also be rate limiting. Rate limiting is deadly for TCP performance so it shouldn't be used on TCP traffic.
Hmm...I'd have to disagree. Are you perhaps assuming a certain threshold (>100mbps, for instance)? I use rate limiting for some of my customers, and when correctly configured (you _must_ use the right burst sizes), you will get the exact rate specified, TCP or not. However, I've never had to rate-limit above 30mbps, so perhaps you have some experience that I don't. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
--On 15 October 2004 11:46 -0400 Andy Dills <andy@xecu.net> wrote:
Hmm...I'd have to disagree. Are you perhaps assuming a certain threshold (>100mbps, for instance)?
I use rate limiting for some of my customers, and when correctly configured (you _must_ use the right burst sizes), you will get the exact rate specified, TCP or not. However, I've never had to rate-limit above 30mbps, so perhaps you have some experience that I don't.
I can support what Iljisch said. In a former life I ran extensive tests on the effect of CAR on TCP (no longer have the data to publish, but it's out there), and it's "just plain broken" - if your purpose is to simulate a lower amount of bandwidth with or without a burst. In a nutshell the problem is that the sliding window algorithm expects RTT to gradually increase with congestion, to find the optimum window size - the increased RTT stops the window growing. With rate-limiting that does not also shape (i.e. buffer the packets - this is true of token based systems such as CAR), the window size just keeps on expanding in leaps and bounds until there's a packet drop, whereupon it shrinks right down, rinse and repeat, so you get a sawtooth effect. Adding burst sizes just moves the problem around - you don't see the effect until later in the stream - because the excess of traffic over committed rate just sits there using up the burst and there is no signal to slow down; it /somewhat/ hides the effect in a lab if you are using short single requests (e.g. short HTTP) but not if you aggregate multiple parallel requests. If you want to simulate lower bandwidths through a high bandwidth interface, and you want to be TCP friendly, you HAVE to use shaping. That means buffering (delaying) packets, and at gigabit line rates, with multiple clients, you need BIG buffers (but set sensible buffer limits per client). You can reasonably trivially do the above test with ftp, ethereal, a bit of perl, and something to graph sequence numbers and throughput. There certainly used to be very few devices that did this properly, AND cope with a saturated GigE of small packet DDoS without dying spectacularly. This may or may not have changed. Alex
On Fri, 15 Oct 2004, Alex Bligh wrote:
I can support what Iljisch said.
In a former life I ran extensive tests on the effect of CAR on TCP (no longer have the data to publish, but it's out there), and it's "just plain broken" - if your purpose is to simulate a lower amount of bandwidth with or without a burst. In a nutshell the problem is that the sliding window algorithm expects RTT to gradually increase with congestion, to find the optimum window size - the increased RTT stops the window growing. With rate-limiting that does not also shape (i.e. buffer the packets - this is true of token based systems such as CAR), the window size just keeps on expanding in leaps and bounds until there's a packet drop, whereupon it shrinks right down, rinse and repeat, so you get a sawtooth effect. Adding burst sizes just moves the problem around - you don't see the effect until later in the stream - because the excess of traffic over committed rate just sits there using up the burst and there is no signal to slow down; it /somewhat/ hides the effect in a lab if you are using short single requests (e.g. short HTTP) but not if you aggregate multiple parallel requests.
If you want to simulate lower bandwidths through a high bandwidth interface, and you want to be TCP friendly, you HAVE to use shaping. That means buffering (delaying) packets, and at gigabit line rates, with multiple clients, you need BIG buffers (but set sensible buffer limits per client).
You can reasonably trivially do the above test with ftp, ethereal, a bit of perl, and something to graph sequence numbers and throughput.
In the testing I have done, it works great. You get the bandwidth configured. Our customers would be outside with burning stakes and pitchforks otherwise. The vast majority of people who complain about CAR have horribly misconfigured limits. If your burst size is off, you get nothing resembling the configured throughput. Yes, there is sawtoothing, but the excessive burst accomodates that to a large degree, allowing for an overall correct transfer rate. The time spent in slow-start is accomodated by dropping packets later than you normally would, resulting in a correct rate over time. The sawtoothing doesn't really matter if you're selling a 512kbps ethernet connection, and the customer can always get both 512kbps in a single stream and aggregated across multiple streams. If the desire is to provide a simulated circuit with "x" bandwidth, CAR does a great job, IFF you correctly size the burst: 1.5x/8 for the normal burst, 3x/8 for the max burst. The aggregate rate of the transfer is "x" in all the testing I've done. How can you ask for more than the configured line rate? In my testing, I noticed a pronounced saw-tooth effect with incorrectly configured bursts, but with correctly configured bursts, the saw-toothing affect did not prevent delivery of the configured throughput. This holds up with multiple concurrent transfers. The customer gets whatever bandwidth is available under their cap, and when I look at their bandwidth graphs, they have the ability to saturate their bandwidth to 100% of the configured rate. Works for me and mine. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
--On 15 October 2004 12:31 -0400 Andy Dills <andy@xecu.net> wrote:
If the desire is to provide a simulated circuit with "x" bandwidth, CAR does a great job, IFF you correctly size the burst: 1.5x/8 for the normal burst, 3x/8 for the max burst.
The aggregate rate of the transfer is "x" in all the testing I've done. How can you ask for more than the configured line rate? In my testing, I noticed a pronounced saw-tooth effect with incorrectly configured bursts, but with correctly configured bursts, the saw-toothing affect did not prevent delivery of the configured throughput.
It's a fair while ago now, we did a pretty full range of tweaking (both of max burst, burst size, and indeed of committed rate). We observed the following problems: a) The fudge factor that you needed to apply to get the right bandwidth depended heavily on (from memory) (i) TCP stacks either end, whether slowstart configured etc. (ii) path MTU (iii) Number of simultaneous connections (iv) Protocol type (e.g. TCP vs. UDP), and content (HTTP was for reasons to do with persistent connections typically different from FTP) We did indeed (until we found a better solution) manage to come up with a fudge factor that minimized customer complaints under this head (which was most of them), but it was essentially "let's wind everything up high enough that in the worst case of the above they get throughput not less than they have bought"; however, this meant we were giving away rather more bandwidth than we meant to, which made upgrades a hard sell. b) It *STILL* didn't work like normal TCP. We had customers with web servers behind these things who expected (say) a 2Mb service running constantly flatlined to operate like a 2Mb/s pipe running full (but not overfull) - i.e. they'd expect to go buy a level of service roughly equal to their 95th percentile / busy hour rate. When they were even slightly congested, their packet loss substantially exceeded what you'd see on the end of properly buffered (say) 2Mb/s serial link. If their traffic was bursty, the problem was worse. Even if you could then say "well our tests show you are getting 2Mb/s (or rather more than that)" the fact a disproportionate number of packets were being lost caused lots of arguments about SLA. c) The problem is worst when the line speed and the ratelimit speed are most mismatched. Thus if you are ratelimiting at 30Mb/s on a 100Mb/s, you won't see too much of a problem. If you are ratelimiting at (say) 128kbps on a 1Gb/s port, you see rather more problems. In theory, this should have been fixed by sufficient buffering and burst, but at least on Cisco 75xx (which is what this was on several years ago), it wasn't - whilst we found a mathematical explanation, it wasn't sufficient to explain the problems we saw (I have a feeling it was due to something in the innards of CEF switching). I know several others who had similar problems both before this and after it (one solving it by putting in a Catalyst with an ATM blade running LANE and a fore ATM switch - yuck - there are better ways to do it now). I am told that PXE stuff which does WFQ etc. in hardware is now up to this (unverified). But that's shaping, not rate-limiting. Alex
CAR does not work like a regular link; no way. It works like a link with 0 buffer size. Problem is that CAR drops packets which override bandwidth, do not query them (and do not prioritize them), so it use TCP adaptation to the packet drop, not to the delay/rtt. This thing works fine with drop levels about 1% - 5%, not more. Traffic shaping works better (it query packets) but it's implementation @ Cisco makes it very unstable by the nature. ----- Original Message ----- From: "Alex Bligh" <alex@alex.org.uk> To: "Andy Dills" <andy@xecu.net> Cc: "Iljitsch van Beijnum" <iljitsch@muada.com>; "NANOG" <nanog@merit.edu>; "Alex Bligh" <alex@alex.org.uk> Sent: Friday, October 15, 2004 9:51 AM Subject: Re: why upload with adsl is faster than 100M ethernet ?
--On 15 October 2004 12:31 -0400 Andy Dills <andy@xecu.net> wrote:
If the desire is to provide a simulated circuit with "x" bandwidth, CAR does a great job, IFF you correctly size the burst: 1.5x/8 for the
normal
burst, 3x/8 for the max burst.
The aggregate rate of the transfer is "x" in all the testing I've done. How can you ask for more than the configured line rate? In my testing, I noticed a pronounced saw-tooth effect with incorrectly configured bursts, but with correctly configured bursts, the saw-toothing affect did not prevent delivery of the configured throughput.
It's a fair while ago now, we did a pretty full range of tweaking (both of max burst, burst size, and indeed of committed rate). We observed the following problems: a) The fudge factor that you needed to apply to get the right bandwidth depended heavily on (from memory) (i) TCP stacks either end, whether slowstart configured etc. (ii) path MTU (iii) Number of simultaneous connections (iv) Protocol type (e.g. TCP vs. UDP), and content (HTTP was for reasons to do with persistent connections typically different from FTP) We did indeed (until we found a better solution) manage to come up with a fudge factor that minimized customer complaints under this head (which was most of them), but it was essentially "let's wind everything up high enough that in the worst case of the above they get throughput not less than they have bought"; however, this meant we were giving away rather more bandwidth than we meant to, which made upgrades a hard sell. b) It *STILL* didn't work like normal TCP. We had customers with web servers behind these things who expected (say) a 2Mb service running constantly flatlined to operate like a 2Mb/s pipe running full (but not overfull) - i.e. they'd expect to go buy a level of service roughly equal to their 95th percentile / busy hour rate. When they were even slightly congested, their packet loss substantially exceeded what you'd see on the end of properly buffered (say) 2Mb/s serial link. If their traffic was bursty, the problem was worse. Even if you could then say "well our tests show you are getting 2Mb/s (or rather more than that)" the fact a disproportionate number of packets were being lost caused lots of arguments about SLA. c) The problem is worst when the line speed and the ratelimit speed are most mismatched. Thus if you are ratelimiting at 30Mb/s on a 100Mb/s, you won't see too much of a problem. If you are ratelimiting at (say) 128kbps on a 1Gb/s port, you see rather more problems. In theory, this should have been fixed by sufficient buffering and burst, but at least on Cisco 75xx (which is what this was on several years ago), it wasn't - whilst we found a mathematical explanation, it wasn't sufficient to explain the problems we saw (I have a feeling it was due to something in the innards of CEF switching).
I know several others who had similar problems both before this and after it (one solving it by putting in a Catalyst with an ATM blade running LANE and a fore ATM switch - yuck - there are better ways to do it now). I am told that PXE stuff which does WFQ etc. in hardware is now up to this (unverified). But that's shaping, not rate-limiting.
Alex
tcptrace with xplot or jplot could help you out debuging this issue. http://jarok.cs.ohiou.edu/software/tcptrace/useful.html _sergei On Fri, 15 Oct 2004 10:11:36 -0700, Alexei Roudnev <alex@relcom.net> wrote:
CAR does not work like a regular link; no way. It works like a link with 0 buffer size.
Problem is that CAR drops packets which override bandwidth, do not query them (and do not prioritize them), so it use TCP adaptation to the packet drop, not to the delay/rtt. This thing works fine with drop levels about 1% - 5%, not more.
Traffic shaping works better (it query packets) but it's implementation @ Cisco makes it very unstable by the nature.
----- Original Message ----- From: "Alex Bligh" <alex@alex.org.uk> To: "Andy Dills" <andy@xecu.net> Cc: "Iljitsch van Beijnum" <iljitsch@muada.com>; "NANOG" <nanog@merit.edu>; "Alex Bligh" <alex@alex.org.uk> Sent: Friday, October 15, 2004 9:51 AM Subject: Re: why upload with adsl is faster than 100M ethernet ?
--On 15 October 2004 12:31 -0400 Andy Dills <andy@xecu.net> wrote:
If the desire is to provide a simulated circuit with "x" bandwidth, CAR does a great job, IFF you correctly size the burst: 1.5x/8 for the
normal
burst, 3x/8 for the max burst.
The aggregate rate of the transfer is "x" in all the testing I've done. How can you ask for more than the configured line rate? In my testing, I noticed a pronounced saw-tooth effect with incorrectly configured bursts, but with correctly configured bursts, the saw-toothing affect did not prevent delivery of the configured throughput.
It's a fair while ago now, we did a pretty full range of tweaking (both of max burst, burst size, and indeed of committed rate). We observed the following problems: a) The fudge factor that you needed to apply to get the right bandwidth depended heavily on (from memory) (i) TCP stacks either end, whether slowstart configured etc. (ii) path MTU (iii) Number of simultaneous connections (iv) Protocol type (e.g. TCP vs. UDP), and content (HTTP was for reasons to do with persistent connections typically different from FTP) We did indeed (until we found a better solution) manage to come up with a fudge factor that minimized customer complaints under this head (which was most of them), but it was essentially "let's wind everything up high enough that in the worst case of the above they get throughput not less than they have bought"; however, this meant we were giving away rather more bandwidth than we meant to, which made upgrades a hard sell. b) It *STILL* didn't work like normal TCP. We had customers with web servers behind these things who expected (say) a 2Mb service running constantly flatlined to operate like a 2Mb/s pipe running full (but not overfull) - i.e. they'd expect to go buy a level of service roughly equal to their 95th percentile / busy hour rate. When they were even slightly congested, their packet loss substantially exceeded what you'd see on the end of properly buffered (say) 2Mb/s serial link. If their traffic was bursty, the problem was worse. Even if you could then say "well our tests show you are getting 2Mb/s (or rather more than that)" the fact a disproportionate number of packets were being lost caused lots of arguments about SLA. c) The problem is worst when the line speed and the ratelimit speed are most mismatched. Thus if you are ratelimiting at 30Mb/s on a 100Mb/s, you won't see too much of a problem. If you are ratelimiting at (say) 128kbps on a 1Gb/s port, you see rather more problems. In theory, this should have been fixed by sufficient buffering and burst, but at least on Cisco 75xx (which is what this was on several years ago), it wasn't - whilst we found a mathematical explanation, it wasn't sufficient to explain the problems we saw (I have a feeling it was due to something in the innards of CEF switching).
I know several others who had similar problems both before this and after it (one solving it by putting in a Catalyst with an ATM blade running LANE and a fore ATM switch - yuck - there are better ways to do it now). I am told that PXE stuff which does WFQ etc. in hardware is now up to this (unverified). But that's shaping, not rate-limiting.
Alex
Hi Joe, "divide et impera" :-) Put a (FTP-)server in _your_ network to get an idea if the problem is with the edge device (or somewhere in your network) or on the WAN to Japan. The PING may not tell you much about short-term queue problems in a device. As Mikael Abrahamsson wrote use a sniffer (Ethereal, tcpdump, ...) and search for retransmissions, i.e. some "breaks" in a monotonic increasing sequence number sequence or such. We had a similar issue when E1 customers had been fine but customers on the same router with 10Mbit/s and above complained. Took us some memory/queue fine-tuning of the 7500 box to get it right, traffic on the higher speed link probably was more "bursty" and this flooded some buffers then. On Oct 15, 2004, at 10:14 Uhr, Joe Shen wrote:
Hi,
the network path is:
|-(ADSL)--------\ customer/ --Edge_router---...---Japan Server \-(100Methernet)-/
So, from edge_router to Japan server the path is identical.
Regards, Marc -- Marc Binderberger <marc@sniff.de>
On Fri, 15 Oct 2004, Joe Shen wrote:
Hi,
the network path is:
|-(ADSL)--------\ customer/ --Edge_router---...---Japan Server \-(100Methernet)-/
So, from edge_router to Japan server the path is identical.
Yes. But, for ftp TCP control real end-to-end transmission speed.
I attached a monitor computer to our core router along the path between customer's site and server. Monitoring computer ping customer's site by targeting both ends of ADSL line and ethernet line. The measuring is scheduled 20packet per 20seconds, we also ping each hop address along the path to server. The result shows there is no packet loss along from monitoring computer to customer site, but packet loss increase at a special hop along the path to server in japan.
So, we think the bottleneck is not inside our network. And, TCP connection between customer's computer and its server should be affacted by same bottleneck. So, the uploading speed should be similar (?), but it shows so much difference!
I can think of three possible things: 1. MTU over the ethernet and ADSL lines is different and there is fragmentation occuring when the ethernet link is used. Look at packet sizes with your sniffer and/or try sweeping a range of ping sizes. 2. Somewhere in the path there are parallel load-balanced paths with variable delay resulting in packets arriving out-of-order more frequently when sent over the ethernet link, due to the packets arriving more frequently in bursts when originating from the faster link. Do you have a group of load-sharing per-packet links in your core over which the traffic flows? Could also be beyond your control. Ethereal at the receiving end might show this. 3. As was previously suggested, aggresive rate-limiting or policing happening along the long-haul. -- Jay Hennigan - CCIE #7880 - Network Administration - jay@west.net WestNet: Connecting you to the planet. 805 884-6323 WB6RDV NetLojix Communications, Inc. - http://www.netlojix.com/
On Fri, 15 Oct 2004 00:14:11 -0800, Joe Shen wrote:
|-(ADSL)--------\ customer/ --Edge_router---...---Japan Server \-(100Methernet)-/
it is probably worth doing an experiment, by placing a target host just before the edge router, inside your net, and verify that you do not get the (bad) differential performance there. d/ -- Dave Crocker Brandenburg InternetWorking +1.408.246.8253 dcrocker a t ... www.brandenburg.com
Thanks. I've done the experiments. The reason is: the 100Mbps ethernet is so fast that it could fill the buffer of bottleneck link very quickly ( Path_mtu, burstness of traffic). There may also exist ACK compression in reverse path . Joe --- Dave Crocker <dhc2@dcrocker.net> wrote:
On Fri, 15 Oct 2004 00:14:11 -0800, Joe Shen wrote:
|-(ADSL)--------\ customer/
--Edge_router---...---Japan Server
\-(100Methernet)-/
it is probably worth doing an experiment, by placing a target host just before the edge router, inside your net, and verify that you do not get the (bad) differential performance there.
d/ -- Dave Crocker Brandenburg InternetWorking +1.408.246.8253 dcrocker a t ... www.brandenburg.com
__________________________________________________ Do You Yahoo!? Log on to Messenger with your mobile phone! http://sg.messenger.yahoo.com
participants (12)
-
Alex Bligh -
Alexei Roudnev -
Andy Dills -
Dave Crocker -
Gordon -
Iljitsch van Beijnum -
Jay Hennigan -
Joe Shen -
Marc Binderberger -
Mikael Abrahamsson -
Mike McSpedon -
sergei