Bandwidth issues in the Sprint network
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else. -- Brian Raaen Network Engineer braaen@zcorum.com
On Mon, 07 Apr 2008 15:06:21 EDT, Brian Raaen said:
have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
TCP window size tuning? I'd look there first...
Has this circuit ever run clean(normal)? -M< On Mon, Apr 7, 2008 at 1:06 PM, Brian Raaen <braaen@zcorum.com> wrote:
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
Currently there is not a proxy server in the network, although when using some of the test on dslreports.com there is a message about compression being used for the upload and to remove proxy settings. I have also been testing using FTP on a *nix server as well. Both the server and PC are connect to a Cisco 2960 switch in the headend that is connected to the 7200 router. I can transfer ftp at about 80Mbps between the PC and the server, so they are not IO bound. The Site I am testing with is a ftp server located in a colo facility that we use and has sufficient bandwidth. This circuit is clean in the sense of not having CRC, framing or other errors but this is a new circuit and we have never gotten more than 5Mbps out of a single session (flow/ip) across the wan. I would have to double check the mtu, but it is currently the default. -- Brian Raaen Network Engineer braaen@zcorum.com On Monday 07 April 2008, Brian Raaen wrote:
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
Try using the Java test on DSLReports rather than the Flash based test. I've found it to be much more accurate. I also receive the message about compression being used when I test with the flash test. I think it may be a bug. Matthew Evans, MCSA Alpha Theory | “the right decision, every time.” 2201 Coronation Blvd., Suite 140 Charlotte, NC 28227 (704) 307-2914 x205 www.alphatheory.com ALPHA THEORY QUICK DEMO (click here) -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Brian Raaen Sent: Tuesday, April 08, 2008 9:49 AM To: nanog@merit.edu Subject: Re: Bandwidth issues in the Sprint network Currently there is not a proxy server in the network, although when using some of the test on dslreports.com there is a message about compression being used for the upload and to remove proxy settings. I have also been testing using FTP on a *nix server as well. Both the server and PC are connect to a Cisco 2960 switch in the headend that is connected to the 7200 router. I can transfer ftp at about 80Mbps between the PC and the server, so they are not IO bound. The Site I am testing with is a ftp server located in a colo facility that we use and has sufficient bandwidth. This circuit is clean in the sense of not having CRC, framing or other errors but this is a new circuit and we have never gotten more than 5Mbps out of a single session (flow/ip) across the wan. I would have to double check the mtu, but it is currently the default. -- Brian Raaen Network Engineer braaen@zcorum.com On Monday 07 April 2008, Brian Raaen wrote:
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
Could be your TCP window size? A 17520 byte TCP window (Windows 2000) will cause a single flow to top out at 5Mbps at about 50ms. What is the latency on the link? Try some figures here and see what limit you might be hitting: http://www.wand.net.nz/~perry/max_download.php?bits_per_second=155000000&ack_size=40&no_delayed_acks=2&mss=1460&rtt=35&wsize=17520&ploss=0 Sam Brian Raaen wrote:
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
I have been using the Java based versions of the speed test. At this point I have had some Sprint people get in contact with me so I will see what they find. Thank you for all your help to everyone. -- Brian Raaen Network Engineer braaen@zcorum.com
On Monday 07 April 2008, you wrote: I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
On Tue, Apr 8, 2008 at 9:19 AM, Brian Raaen <braaen@zcorum.com> wrote:
I have been using the Java based versions of the speed test. At this point I have had some Sprint people get in contact with me so I will see what they find. Thank you for all your help to everyone.
-- Brian Raaen Network Engineer braaen@zcorum.com
On Monday 07 April 2008, you wrote: I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
Most of the speed test sites on the Internet basically issue a HTTP GET request to a server and time the download. For upload they utilize a HTTP POST via a CGI script and time that. The main issue I have with these speed tests is that they only use a single TCP session for data transfer, which is fine if you have a large or self adjusting TCP window size and a relatively low latency link. However for high capacity links, it is unlikely (but possible) that you are planning to use a single TCP session and consume all the available capacity. Realistically you will have a few dozen server/applications/users and produce hundreds/thousands of TCP sessions which will fully utilize the link. For our PtP customers that have concerns regarding capacity, I generally they suggest setup iperf at both ends and run a few tests with multiple TCP sessions so they can independently verify. Hopefully Sprint will take your concerns to heart and assist you with testing. -Mike Gonnason
I have been using the Java based versions of the speed test. At this
have had some Sprint people get in contact with me so I will see what
Does anyone know of bootable Linux CD with iperf on it? Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Mike Gonnason Sent: Wednesday, April 09, 2008 9:05 AM To: nanog@merit.edu Subject: Re: Bandwidth issues in the Sprint network On Tue, Apr 8, 2008 at 9:19 AM, Brian Raaen <braaen@zcorum.com> wrote: point I they
find. Thank you for all your help to everyone.
-- Brian Raaen Network Engineer braaen@zcorum.com
On Monday 07 April 2008, you wrote: I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
Most of the speed test sites on the Internet basically issue a HTTP GET request to a server and time the download. For upload they utilize a HTTP POST via a CGI script and time that. The main issue I have with these speed tests is that they only use a single TCP session for data transfer, which is fine if you have a large or self adjusting TCP window size and a relatively low latency link. However for high capacity links, it is unlikely (but possible) that you are planning to use a single TCP session and consume all the available capacity. Realistically you will have a few dozen server/applications/users and produce hundreds/thousands of TCP sessions which will fully utilize the link. For our PtP customers that have concerns regarding capacity, I generally they suggest setup iperf at both ends and run a few tests with multiple TCP sessions so they can independently verify. Hopefully Sprint will take your concerns to heart and assist you with testing. -Mike Gonnason
A quick search comes up with Scientific Linux, but I cannot provide any claims to suitability. I have never even heard of it before, but it is provided as a LiveCD. http://linux.web.psi.ch/livecd/software.html -Mike Gonnason On Wed, Apr 9, 2008 at 6:28 AM, Frank Bulk <frnkblk@iname.com> wrote:
Does anyone know of bootable Linux CD with iperf on it?
Frank
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Mike Gonnason Sent: Wednesday, April 09, 2008 9:05 AM To: nanog@merit.edu Subject: Re: Bandwidth issues in the Sprint network
I have been using the Java based versions of the speed test. At this
have had some Sprint people get in contact with me so I will see what
On Tue, Apr 8, 2008 at 9:19 AM, Brian Raaen <braaen@zcorum.com> wrote: point I they
find. Thank you for all your help to everyone.
-- Brian Raaen Network Engineer braaen@zcorum.com
On Monday 07 April 2008, you wrote: I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
Most of the speed test sites on the Internet basically issue a HTTP GET request to a server and time the download. For upload they utilize a HTTP POST via a CGI script and time that. The main issue I have with these speed tests is that they only use a single TCP session for data transfer, which is fine if you have a large or self adjusting TCP window size and a relatively low latency link.
However for high capacity links, it is unlikely (but possible) that you are planning to use a single TCP session and consume all the available capacity. Realistically you will have a few dozen server/applications/users and produce hundreds/thousands of TCP sessions which will fully utilize the link.
For our PtP customers that have concerns regarding capacity, I generally they suggest setup iperf at both ends and run a few tests with multiple TCP sessions so they can independently verify. Hopefully Sprint will take your concerns to heart and assist you with testing.
-Mike Gonnason
Does anyone know of bootable Linux CD with iperf on it?
Knoppix STD (security tools distro) http://www.knoppix-std.org/tools.html Cheers, Michael Holstein Cleveland State University
Not 100% sure about iperf but I2 has a nice Network Performance Toolkit that runs on top of Knoppix and they have a downloadable ISO image... Get the ISO here... http://e2epi.internet2.edu/network-performance-toolkit.html Interesting doc on configuring toolkit from SLAC... http://confluence.slac.stanford.edu/display/IEPM/Network+Performance+Too lkit Bill Murphy Senior Network Analyst University of Texas Health Science Center - Houston -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Michael Holstein Sent: Wednesday, April 09, 2008 10:00 AM To: frnkblk@iname.com Cc: nanog@merit.edu Subject: Re: Bandwidth issues in the Sprint network
Does anyone know of bootable Linux CD with iperf on it?
Knoppix STD (security tools distro) http://www.knoppix-std.org/tools.html Cheers, Michael Holstein Cleveland State University
Some people wanted to know what I found the problem to be. I have discovered. the problem for a fact is the TCP window size on uploads. I have a Linux box that I changed the Window sizes to match and I still get 32k on a upload window and 64k on a download window. With a ping time of 50ms I have a max theoretical throughput of 5.2Mbps Which is about what I was getting. The formula to calculate this is the following. (((Ts/Tw)*Rtd)/1000)+((Ts*8)/(Lr*1000))) Where the following are Ts = Transfer size in Bytes Tw = Tcp Window size in Bytes Rtd = Round trip Delay in milliseconds Lr = Line rate in bps At this point I am still trying to locate the offending device that is changing the window size. After I determine for sure whether the problem is with my router, the sprint network, or another upstream system I will let everybody know what I find. -- Brian Raaen Network Engineer braaen@zcorum.com On Monday 07 April 2008, Brian Raaen wrote:
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
On Thu, Apr 17, 2008 at 1:00 PM, Brian Raaen <braaen@zcorum.com> wrote:
Some people wanted to know what I found the problem to be. I have discovered. the problem for a fact is the TCP window size on uploads. I have a Linux box that I changed the Window sizes to match and I still get 32k on a upload window and 64k on a download window. With a ping time of 50ms I have a max theoretical throughput of 5.2Mbps Which is about what I was getting. The formula to calculate this is the following.
(((Ts/Tw)*Rtd)/1000)+((Ts*8)/(Lr*1000)))
Where the following are
Ts = Transfer size in Bytes Tw = Tcp Window size in Bytes Rtd = Round trip Delay in milliseconds Lr = Line rate in bps
At this point I am still trying to locate the offending device that is changing the window size. After I determine for sure whether the problem is with my router, the sprint network, or another upstream system I will let everybody know what I find.
-- Brian Raaen Network Engineer braaen@zcorum.com
On Monday 07 April 2008, Brian Raaen wrote:
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
-- Brian Raaen Network Engineer braaen@zcorum.com
Thanks for reporting back to curious minds. Mike Gonnason
even with tuned TCP window sizes, make sure you don't have TCP syncookies enabled on either endpoint. many syncookie implementations have implications on supporting RFC1323 options. cheers, lincoln.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Brian Raaen Sent: Friday, 18 April 2008 7:00 AM To: nanog@merit.edu Subject: Re: Bandwidth issues in the Sprint network
Some people wanted to know what I found the problem to be. I have discovered. the problem for a fact is the TCP window size on uploads. I have a Linux box that I changed the Window sizes to match and I still get 32k on a upload window and 64k on a download window. With a ping time of 50ms I have a max theoretical throughput of 5.2Mbps Which is about what I was getting. The formula to calculate this is the following.
(((Ts/Tw)*Rtd)/1000)+((Ts*8)/(Lr*1000)))
Where the following are
Ts = Transfer size in Bytes Tw = Tcp Window size in Bytes Rtd = Round trip Delay in milliseconds Lr = Line rate in bps
At this point I am still trying to locate the offending device that is changing the window size. After I determine for sure whether the problem is with my router, the sprint network, or another upstream system I will let everybody know what I find.
-- Brian Raaen Network Engineer braaen@zcorum.com
On Monday 07 April 2008, Brian Raaen wrote:
I am currently having problems get upload bandwidth on a Sprint circuit. I am using a full OC3 circuit. I am doing fine on downloading data, but uploading data I can only get about 5Mbps with ftp or a speedtest. I have tested against multiple networks and this has stayed the same. Monitoring Cacti graphs and the router I do get about 30Mbps total traffic outbound, but individual (flows/ip?) test always seem limited. I would like to know if anyone else sees anything similar, or where I can get help. The assistance I have gotten from Sprint up to this point is that they find no problems. Due to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know if I was overlooking something else.
Once upon a time, Lincoln Dale <ltd@interlink.com.au> said:
even with tuned TCP window sizes, make sure you don't have TCP syncookies enabled on either endpoint.
IIRC Linux (at least) syncookies only come into play when you are being syn-flooded (i.e. when the kernel has to start dropping syns). Having them enabled at other times has no impact, so there's rarely (if ever) a reason to disable them. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Chris Adams wrote:
Once upon a time, Lincoln Dale <ltd@interlink.com.au> said:
even with tuned TCP window sizes, make sure you don't have TCP syncookies enabled on either endpoint.
IIRC Linux (at least) syncookies only come into play when you are being syn-flooded (i.e. when the kernel has to start dropping syns). Having them enabled at other times has no impact, so there's rarely (if ever) a reason to disable them.
whats up with sprint today? http://www.internetpulse.net/ seems like they keep having issues.
participants (12)
-
Brian Raaen -
Chris Adams -
FISIX -
Frank Bulk -
Lincoln Dale -
Martin Hannigan -
Matthew Evans -
Michael Holstein -
Mike Gonnason -
Murphy, William -
Sam Stickland -
Valdis.Kletnieks＠vt.edu