Chuck, The question posed by Chris, "how long is your access line"?, is a rather important data point before answering your question. I have a T3 that is about 15 miles long. It runs between two 7500 routers. Its minimum ping round trip time with 100 byte pings is 2 ms. It is not very heavily loaded with peaks of about 10 Mb/s today and the max round trip time was 21 ms. So that should give you some minimal bounds of what you might expect. As the saying goes your mileage may vary. Speed of light does play in here, if the circuit is longer. Also intervening electronics such as a frame cloud or telco muxes also add latency. Walt
The rule of thumb I use is that the speed of light in fiber-optic cable is roughly 2x10^8 m/sec. 2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec I once worked with a customer whose first hop out was ~30ms, regardless of the load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring that travelled the north-south length of the US roughly twice before his traffic went elsewhere. ......Matthew ---------- M. F. Ringel Network Engineer Akamai Technologies, Inc. ringel@akamai.com On Sat, Feb 17, 2001 at 07:37:48AM +0000, Walter Prue wrote:
Chuck,
The question posed by Chris, "how long is your access line"?, is a rather important data point before answering your question. I have a T3 that is about 15 miles long. It runs between two 7500 routers. Its minimum ping round trip time with 100 byte pings is 2 ms. It is not very heavily loaded with peaks of about 10 Mb/s today and the max round trip time was 21 ms.
So that should give you some minimal bounds of what you might expect. As the saying goes your mileage may vary. Speed of light does play in here, if the circuit is longer. Also intervening electronics such as a frame cloud or telco muxes also add latency.
Walt
Matthew: Appears to be a typo in your final number of 130 mi/sec, but I get where you're going with this. I'm just having a problem trying to figure out how I end up with a couple thousand fiber miles from Northern Michigan to Chicago. Should be interesting to sort this one out. Thanks, Chuck On Sat, 17 Feb 2001, Matthew F. Ringel wrote:
The rule of thumb I use is that the speed of light in fiber-optic cable is roughly 2x10^8 m/sec.
2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec
I once worked with a customer whose first hop out was ~30ms, regardless of the load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring that travelled the north-south length of the US roughly twice before his traffic went elsewhere.
......Matthew
Yup. Dropped a letter there. 130mi/msec. .....Matthew On Sat, Feb 17, 2001 at 09:33:02AM -0500, Charles Scott wrote:
Matthew: Appears to be a typo in your final number of 130 mi/sec, but I get where you're going with this. I'm just having a problem trying to figure out how I end up with a couple thousand fiber miles from Northern Michigan to Chicago. Should be interesting to sort this one out.
Thanks,
Chuck
On Sat, 17 Feb 2001, Matthew F. Ringel wrote:
The rule of thumb I use is that the speed of light in fiber-optic cable is roughly 2x10^8 m/sec.
2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec
I once worked with a customer whose first hop out was ~30ms, regardless of the load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring that travelled the north-south length of the US roughly twice before his traffic went elsewhere.
......Matthew
Or, perhaps a more simplified and easily remembered figure... RTT on a straight line run from sfo to dc would be ~63ms. (Seems to be roughly 100 to 120 for most real circuits.) We all know, however, that telcos rarely use straight lines. Still, I would not expect more than 6 to 7ms. Perhaps your telcos equipment, through some fluke, has you operating on the backup path? On Sat, Feb 17, 2001 at 09:33:02AM -0500, Charles Scott wrote:
Matthew: Appears to be a typo in your final number of 130 mi/sec, but I get where you're going with this. I'm just having a problem trying to figure out how I end up with a couple thousand fiber miles from Northern Michigan to Chicago. Should be interesting to sort this one out.
Thanks,
Chuck
On Sat, 17 Feb 2001, Matthew F. Ringel wrote:
The rule of thumb I use is that the speed of light in fiber-optic cable is roughly 2x10^8 m/sec.
2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec
I once worked with a customer whose first hop out was ~30ms, regardless of the load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring that travelled the north-south length of the US roughly twice before his traffic went elsewhere.
......Matthew
---------------------------------------------------------------------- Wayne Bouchard [Imagine Your ] web@typo.org [Company Name Here] Network Engineer http://www.typo.org/~web/resume.html ----------------------------------------------------------------------
On Sat, 17 Feb 2001, Wayne Bouchard wrote:
Or, perhaps a more simplified and easily remembered figure...
RTT on a straight line run from sfo to dc would be ~63ms. (Seems to be roughly 100 to 120 for most real circuits.)
We all know, however, that telcos rarely use straight lines. Still, I would not expect more than 6 to 7ms. Perhaps your telcos equipment, through some fluke, has you operating on the backup path?
Wayne: Now there's a thought that didn't occur to me. They may be running me all over creation for some reason rather than taking some more direct route. Perhaps they ran out of somthing here in Michigan and had to run it down to Texas and back to get it connected correctly. I think I'll have them inventory the path and let me know what it is. Thanks to all the responses on this. Chuck Scott
I know one of my circuits runs from Seattle to Los Angeles to El Paso (fiber cut a few weeks ago) to Washington DC and finally to New York City. Nice to know my packets get to visit almost 15 states before hitting their destination. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Charles Scott Sent: Saturday, February 17, 2001 8:19 AM To: Wayne Bouchard Cc: Matthew F. Ringel; nanog@merit.edu Subject: Re: T3 Latency On Sat, 17 Feb 2001, Wayne Bouchard wrote:
Or, perhaps a more simplified and easily remembered figure...
RTT on a straight line run from sfo to dc would be ~63ms. (Seems to be roughly 100 to 120 for most real circuits.)
We all know, however, that telcos rarely use straight lines. Still, I would not expect more than 6 to 7ms. Perhaps your telcos equipment, through some fluke, has you operating on the backup path?
Wayne: Now there's a thought that didn't occur to me. They may be running me all over creation for some reason rather than taking some more direct route. Perhaps they ran out of somthing here in Michigan and had to run it down to Texas and back to get it connected correctly. I think I'll have them inventory the path and let me know what it is. Thanks to all the responses on this. Chuck Scott
Chuck, should read 130mi/msec I guess. Which would end up with ~7msec per 1000miles. Arnold ----- Original Message ----- From: "Charles Scott" <cscott@gaslightmedia.com> To: "Matthew F. Ringel" <ringel@akamai.com> Cc: <nanog@merit.edu> Sent: Saturday, February 17, 2001 3:33 PM Subject: Re: T3 Latency
Matthew: Appears to be a typo in your final number of 130 mi/sec, but I get where you're going with this. I'm just having a problem trying to figure out how I end up with a couple thousand fiber miles from Northern Michigan to Chicago. Should be interesting to sort this one out.
Thanks,
Chuck
On Sat, 17 Feb 2001, Matthew F. Ringel wrote:
The rule of thumb I use is that the speed of light in fiber-optic cable
is
roughly 2x10^8 m/sec.
2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec
I once worked with a customer whose first hop out was ~30ms, regardless of the load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring that travelled the north-south length of the US roughly twice before his traffic went elsewhere.
......Matthew
Packet size of your pings may have something to do with it, but assuming you are pinging with the same size packet across the board, the data should be reliable. If you are using unix boxes your pings will have a different level of resolution than say from a windows box. One test that should be reasonably conclusive is the following. The hop that is 22ms and 7 hops away from your upstream should ping you on the other side of the new T3. If it approaches 42 ms, you have a 20ms T3. Is it ATM? Your upstream could be running on a congested ATM cloud. If the latency drops in the wee hours of the morning, even for a few minutes, its congestion (which is obvious). I have a 27 mile (measured by air or telco :) ) DS3 that even under 30mb/s load pings minimum 1 ms average 1-2 ms. (cisco<->cisco). At 1400 bytes the average goes to 3ms. Your upstream may be putting you on a channelized interface on a router that has a very busy VIP (if its a cisco). Or is running you through a lot of electronics on the way. The point here is that any kind of aggregation he is doing could eat up 10-20ms, by design or oversubscription. Bigger providers are more likely to aggregate DS3s into bigger access methods whereas small providers usually don't have the operational necessity. Deepak Jain AiNET On Sat, 17 Feb 2001, Nipper, Arnold wrote:
Chuck,
should read 130mi/msec I guess. Which would end up with ~7msec per 1000miles.
Arnold
----- Original Message ----- From: "Charles Scott" <cscott@gaslightmedia.com> To: "Matthew F. Ringel" <ringel@akamai.com> Cc: <nanog@merit.edu> Sent: Saturday, February 17, 2001 3:33 PM Subject: Re: T3 Latency
Matthew: Appears to be a typo in your final number of 130 mi/sec, but I get where you're going with this. I'm just having a problem trying to figure out how I end up with a couple thousand fiber miles from Northern Michigan to Chicago. Should be interesting to sort this one out.
Thanks,
Chuck
On Sat, 17 Feb 2001, Matthew F. Ringel wrote:
The rule of thumb I use is that the speed of light in fiber-optic cable
is
roughly 2x10^8 m/sec.
2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec
I once worked with a customer whose first hop out was ~30ms, regardless of the load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring that travelled the north-south length of the US roughly twice before his traffic went elsewhere.
......Matthew
On Sat, 17 Feb 2001, Deepak Jain wrote:
Packet size of your pings may have something to do with it, but assuming you are pinging with the same size packet across the board, the data should be reliable. If you are using unix boxes your pings will have a different level of resolution than say from a windows box. One test that should be reasonably conclusive is the following.
The hop that is 22ms and 7 hops away from your upstream should ping you on the other side of the new T3. If it approaches 42 ms, you have a 20ms T3. Is it ATM? Your upstream could be running on a congested ATM cloud. If the latency drops in the wee hours of the morning, even for a few minutes, its congestion (which is obvious).
Have been testing from our 7204 router and from various linux boxes. No matter how small the packet size, we always see the 20 ms. Yes, the total ping time to that other system is a minumum of 42 ms. Not running ATM and have not seen any times when the latency drops below 20ms.
The point here is that any kind of aggregation he is doing could eat up 10-20ms, by design or oversubscription. Bigger providers are more likely to aggregate DS3s into bigger access methods whereas small providers usually don't have the operational necessity.
This may be the key. More of an effect of the way they're bringing me in. I guess I need to ask them some specific questions. Chuck
Also sprach Charles Scott
Have been testing from our 7204 router and from various linux boxes. No matter how small the packet size, we always see the 20 ms. Yes, the total ping time to that other system is a minumum of 42 ms. Not running ATM and have not seen any times when the latency drops below 20ms.
This may be the key. More of an effect of the way they're bringing me in. I guess I need to ask them some specific questions.
I've been watching this thread with some interest...from reading it all, my gut instinct is that some circuit provider has your circuit routed all over creation, or there's some other sub-optimal aspect of your circuit (well...I guess that's pretty obvious). The question then really becomes...how important is the latency *to you*. Clearly, 20ms latency on a 300-400 mile (as the crow flies) T3 is sub-optimal somehow. You might very well have customers and potential customers question that, because it *does* affect performance, however slightly, and it might be an indicator of other, more in depth problems. Also, if you're looking at several thousand fiber miles on a 300 mile line of sight run...that gives more opportunity for backhoe fade and other problems. Honestly, if I were in your situation, I would give some very serious thought to refusing the circuit. I would be very likely to demand that they get the circuit down to 10ms, and might even consider a demand of around 7ms before I would accept it. That, of course, is my thinking, and I don't know all the details of your situation. I believe you posted looking for input and what others thought on it though, so, FWIW, there's my $.02. Another possibility is to look into what the provider gives as far as SLA's are concerned. (I know some on the list have heard about this before...forgive me for sharing an old, though still ongoing, story again) I have a T3 between Louisville, KY and Indianapolis, IN (around 3ms latency, unloaded) that is carried by 6 different telco's. It took 7 months from time of order to get that circuit turned up. When it has gone down, it has taken no less than 18 hours to get it fixed, and the conference calls involved have typically been to the point of outright hilarity (depending on your perspective I guess). The circuit was ordered by UU.Net, and when we found out how the circuit was routed, we were sure to sign up for their SLA agreements. We have had very good luck getting SLA credits when the circuit goes down...and with the circuit typically being down for 24-30 hours...we typically gets a free month of service when it dies. I'm dumbfounded that UU.Net hasn't kicked WCOM's butt into getting this circuit groomed onto WCOM facilities in order to cut down the number of carriers to 2, but they haven't done it yet, and we keep collecting the SLA credits when it goes down, so I can deal with that...I'd rather it stay up and get fixed quicker, but a day of credit per hour of outage does tend to take the sting out of having the circuit down. Anyway...my point in that story...if the provider has SLA's that include latency provisions, it might be worthwhile to try to get it fixed by pounding on the SLA issue until they realize that its in *their* best interest to get the circuit fixed. :) -- Jeff McAdams Email: jeffm@iglou.com Head Network Administrator Voice: (502) 966-3848 IgLou Internet Services (800) 436-4456
On Sat, 17 Feb 2001, Jeff Mcadams wrote:
The question then really becomes...how important is the latency *to you*. Clearly, 20ms latency on a 300-400 mile (as the crow flies) T3 is sub-optimal somehow. You might very well have customers and potential customers question that, because it *does* affect performance, however slightly, and it might be an indicator of other, more in depth problems. Also, if you're looking at several thousand fiber miles on a 300 mile line of sight run...that gives more opportunity for backhoe fade and other problems.
I fully agree. Aside from the marketing handycap this introduces, the "backhoe fade" factor (love that term) is more significant. Did I mention that this line has 13 cross connects and they first tried to turn it up on the 13th of this month? Yes, I'm concerned about reliability and repairability of the line.
Honestly, if I were in your situation, I would give some very serious thought to refusing the circuit. I would be very likely to demand that they get the circuit down to 10ms, and might even consider a demand of around 7ms before I would accept it. That, of course, is my thinking, and I don't know all the details of your situation.
I have already posted a message to the marketing and provisioning people requesting that they at least explain why the latency is there. Something better than low priority responses to ICMP. At this point we're not in a position to refuse the circuit, we need it, but I can probably work this diplomatically with them to get this worked out. Considering this line was ordered in July, and has missed repeated deadlines, they owe me a few favors.
Another possibility is to look into what the provider gives as far as SLA's are concerned.
There are some service guarantees, but it's been so long since I placed the order, I don't remember what they are. I'll have to go pull the file and look. Good point though. Thanks, Chuck
What is your confidence level that your test packets are taking the path that you expect? Are you multihomed? Does your addressing and routing setup allow for the return packets to be taking a different path other than the new DS3? What address are your test pings being sourced with, and what is the destination's return path to that address? dave At 11:05 AM 2/17/01 -0500, Deepak Jain wrote:
Packet size of your pings may have something to do with it, but assuming you are pinging with the same size packet across the board, the data should be reliable. If you are using unix boxes your pings will have a different level of resolution than say from a windows box. One test that should be reasonably conclusive is the following.
The hop that is 22ms and 7 hops away from your upstream should ping you on the other side of the new T3. If it approaches 42 ms, you have a 20ms T3. Is it ATM? Your upstream could be running on a congested ATM cloud. If the latency drops in the wee hours of the morning, even for a few minutes, its congestion (which is obvious).
I have a 27 mile (measured by air or telco :) ) DS3 that even under 30mb/s load pings minimum 1 ms average 1-2 ms. (cisco<->cisco). At 1400 bytes the average goes to 3ms.
Your upstream may be putting you on a channelized interface on a router that has a very busy VIP (if its a cisco). Or is running you through a lot of electronics on the way.
The point here is that any kind of aggregation he is doing could eat up 10-20ms, by design or oversubscription. Bigger providers are more likely to aggregate DS3s into bigger access methods whereas small providers usually don't have the operational necessity.
Deepak Jain AiNET
On Sat, 17 Feb 2001, Nipper, Arnold wrote:
Chuck,
should read 130mi/msec I guess. Which would end up with ~7msec per 1000miles.
Arnold
----- Original Message ----- From: "Charles Scott" <cscott@gaslightmedia.com> To: "Matthew F. Ringel" <ringel@akamai.com> Cc: <nanog@merit.edu> Sent: Saturday, February 17, 2001 3:33 PM Subject: Re: T3 Latency
Matthew: Appears to be a typo in your final number of 130 mi/sec, but I get
where
you're going with this. I'm just having a problem trying to figure out how I end up with a couple thousand fiber miles from Northern Michigan to Chicago. Should be interesting to sort this one out.
Thanks,
Chuck
On Sat, 17 Feb 2001, Matthew F. Ringel wrote:
The rule of thumb I use is that the speed of light in fiber-optic cable
is
roughly 2x10^8 m/sec.
2x10^8 m/sec = 200,000,000 m/sec = 200,000 km/sec = 200 km/msec =~ 130 mi/sec
I once worked with a customer whose first hop out was ~30ms, regardless of the load on the line (a t3, iirc). Sure enough, he was on a very large SONET ring that travelled the north-south length of the US roughly twice before his traffic went elsewhere.
......Matthew
On Sat, 17 Feb 2001, dave o'leary wrote:
What is your confidence level that your test packets are taking the path that you expect?
Are you multihomed? Does your addressing and routing setup allow for the return packets to be taking a different path other than the new DS3? What address are your test pings being sourced with, and what is the destination's return path to that address?
That was something I was considering, but when I also get 20 ms pinging from my 7204 to their router on the other end of the line (both addresses in the same /30), I don't think the responses are going anywhere but straight back to my interface. No, this location is not multi-homed. chuck
On Sat, 17 Feb 2001, Nipper, Arnold wrote:
Chuck,
should read 130mi/msec I guess. Which would end up with ~7msec per 1000miles.
Arnold
Arnold: Yep, that's about what I figured, which means that we're seeing something like 3000 miles of fiber, or somewhat less depending on the latency of any interconnecting equipment. Funny for a path that's about 300 miles line of site and perhaps 500-600 they way I think they have it running. Chuck
participants (9)
-
Aaron Moreau-Cook -
Charles Scott -
dave o'leary -
Deepak Jain -
Jeff Mcadams -
Matthew F. Ringel -
Nipper, Arnold -
Walter Prue -
Wayne Bouchard