Say like this traceroute. This is from TW to a Broadwing DS3. 5 tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13) 153.267 ms 207.125 ms tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9) 218.920 ms 6 ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206) 36.976 ms 26.923 ms 57.770 ms 7 ge-11-0.core2.Chicago1.Level3.net (4.68.101.37) 254.145 ms ge-11-1.core2.Chicago1.Level3.net (4.68.101.101) 258.522 ms ge-11-2.core2.Chicago1.Level3.net (4.68.101.165) 227.223 ms 8 broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10) 231.451 ms 9 so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1) 53.269 ms 35.568 ms 22.511 ms 10 216.140.14.17 (216.140.14.17) 34.751 ms 39.008 ms 46.644 ms 11 p5-0-0.e0.cncn.broadwing.net (216.140.15.78) 32.065 ms 60.797 ms 54.766 ms 12 67.98.17.122 (67.98.17.122) 44.772 ms 27.631 ms 30.655 ms 13 * * * ---------------------------------------------------------------------- Elijah Savage | AOL IM:layer3rules Senior Network Engineer | When it has to be switched or routed. http://www.digitalrage.org | The Information Technology News Center ----- http://www.digitalrage.org/?page_id=46 for pgp public key-------- On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:

Can you be more specific?

-brandon

On 10/26/06, Elijah Savage <esavage@digitalrage.org> wrote: Looks like level3 is having issues. Anyone know what is going on? ---------------------------------------------------------------------- Elijah Savage | AOL IM:layer3rules Senior Network Engineer | When it has to be switched or routed. http://www.digitalrage.org | The Information Technology News Center ----- http://www.digitalrage.org/?page_id=46 for pgp public key--------

-- Brandon Galbraith Email: brandon.galbraith@gmail.com AIM: brandong00 Voice: 630.400.6992 "A true pirate starts drinking before the sun hits the yard-arm. Yarrrr. --thelost"

Yes sir I did. This is now resolved. But thank you for noticing. On Oct 26, 2006, at 7:11 PM, Aaron Glenn wrote:

On 10/26/06, Elijah Savage <esavage@digitalrage.org> wrote:

...

Say like this traceroute. This is from TW to a Broadwing DS3.

5 tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13) 153.267 ms 207.125 ms tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9) 218.920 ms 6 ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206) 36.976 ms 26.923 ms 57.770 ms 7 ge-11-0.core2.Chicago1.Level3.net (4.68.101.37) 254.145 ms ge-11-1.core2.Chicago1.Level3.net (4.68.101.101) 258.522 ms ge-11-2.core2.Chicago1.Level3.net (4.68.101.165) 227.223 ms 8 broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10) 231.451 ms 9 so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1) 53.269 ms 35.568 ms 22.511 ms 10 216.140.14.17 (216.140.14.17) 34.751 ms 39.008 ms 46.644 ms 11 p5-0-0.e0.cncn.broadwing.net (216.140.15.78) 32.065 ms 60.797 ms 54.766 ms 12 67.98.17.122 (67.98.17.122) 44.772 ms 27.631 ms 30.655 ms 13 * * *

Uhh, you do realize the end to end latency there (to hop 12, at least) is ~30ms...not the 250ms+ you see on intermediate hops, right?

Adam, Because of contractual issues it makes it very hard for me to participate on this list hence the vague original post. I was just asking a general question to see if anyone else was having issues. I have peering points with Broadwing(now level3), Sprint, AT&T and MCI (now Verizon) that I can test for throughput from. This was not just about home cable connectivity though when frontline starts to get calls I often use wget (very low overhead) to test throughput between my sites or to home my home box often times simulating the same sort of connectivity that a customer may have. There were customers that could not even get to level3.net yesterday which is their home page, but it is always nice to get the refresher course on ICMP though :). As for html posted messages truly my mistake I know better and thank you for mentioning it. The new duo core 2 mac mail client which I am still trying to get use to under preferences says it is set to plain text hmmm something I need to look into. Thank you On Oct 27, 2006, at 12:22 AM, Adam Rothschild wrote:

Elijah,

On 2006-10-26-16:34:18, Elijah Savage <esavage@digitalrage.org> wrote: [HTML mail stripped]

...

It seems anything traversing level3 has very high latency along with what seems overloaded capacity as if they are running in a degraded mode I have connections with Time Warner, AT&T, and MCI [...]

On 2006-10-26-16:48:15, Elijah Savage <esavage@digitalrage.org> wrote: [HTML mail stripped]

...

Say like this traceroute. This is from TW to a Broadwing DS3.

5 tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13) 153.267 ms 207.125 ms tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9) 218.920 ms 6 ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206) 36.976 ms 26.923 ms 57.770 ms 7 ge-11-0.core2.Chicago1.Level3.net (4.68.101.37) 254.145 ms ge-11-1.core2.Chicago1.Level3.net (4.68.101.101) 258.522 ms ge-11-2.core2.Chicago1.Level3.net (4.68.101.165) 227.223 ms 8 broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10) 231.451 ms 9 so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1) 53.269 ms 35.568 ms 22.511 ms

Your postings appear to be missing two key pieces of information which would help with the community diagnosis requested: source and destination IP addresses. From the information you did provide, one can deduce that you're behind a TW/RoadRunner cable modem:

13.216.78.4.IN-ADDR.ARPA domain name pointer tenge-3-2.car1.Cincinnati1.Level3.net 14.216.78.4.IN-ADDR.ARPA domain name pointer ROADRUNNER.car1.Cincinnati1.Level3.net 9.216.78.4.IN-ADDR.ARPA domain name pointer tenge-3-1.car1.Cincinnati1.Level3.net 10.216.78.4.IN-ADDR.ARPA domain name pointer ROADRUNNER.car1.Cincinnati1.Level3.net

Now, the jitter and high latency you're seeing could be a result of one or more factors, including but not limited to RF/plant issues, TWC running their transport and/or Level(3) transit hot (which seems to be a common occurrence these days), ECMP across two circuits of uneven loading, or your neighbor might be jacking wifi and downloading a bunch of torrents -- we, the readers, just don't know.

Of note when performing armchair troubleshooting across Level(3)'s network: the 'ebr's (PTR record of ebr*.{pop}.level3.net == Force10 E1200; Experimental Backbone Router?) tend to drop a lot of diagnostic traffic (such as, say, 'ping' and 'traceroute') as a part of overly aggressive control-plane policers. This loss is, of course, strictly cosmetic, and has no bearing on end-to-end performance. Hence, the old "to it, not through it" rule applies.

smokeping[1] and iperf[2] (to end hosts) are your friends.

As an aside, I've noticed your string of postings today were all HTML-tagged. While not expressly forbidden (or even discouraged) by the current Mailing List AUP, this is generally regarded as bad form; you might wish to reconfigure your mail client accordingly...

Hope this helps, -a

[1] <http://oss.oetiker.ch/smokeping/> [2] <http://dast.nlanr.net/Projects/Iperf/>

On 2006-10-27-07:37:37, Elijah Savage <esavage@digitalrage.org> wrote:

Because of contractual issues it makes it very hard for me to participate on this list hence the vague original post.

I can understand you might have various NDAs in place limiting what you can and can't disclose. Unfortunately, without full information, it is difficult to provide a full and proper diagnosis.

I was just asking a general question to see if anyone else was having issues.

See, therein lies the problem. As pointed out in recent congressional testimony by the esteemed Senator from Alaska, the internets are comprised of very many tangled-up tubes. At any given time, something just isn't working. Without source and destination IP addresses, it's difficult to determine whether a problem is global in scope (entirely appropriate for this list), or an end-user issue (inappropriate for this list, though some folk may beg to differ :-), as suggested by your snippet of 'traceroute' output -- and ultimately take corrective action.

I have peering points with Broadwing(now level3), Sprint, AT&T and MCI (now Verizon) that I can test for throughput from.

This phraseology is also a bit confusing, though sadly, all too common these days. Unless you're settlement-free, a better idea might be to word this as "I buy transit from..." or perhaps more appropriately, "My cable MSO buys transit from..."

This was not just about home cable connectivity though when frontline starts to get calls I often use wget (very low overhead) to test throughput between my sites or to home my home box often times simulating the same sort of connectivity that a customer may have. There were customers that could not even get to level3.net yesterday which is their home page

Be that as it may, a little information would have helped greatly. Had you said this sooner, and backed it up with some supporting data such as IP addresses and perhaps 'wget' output, chances are we wouldn't be having this discussion. On the other hand, if you can't trust us, perhaps a better course of action would be to open trouble tickets with your provider(s)... -a

Seems to be all cleared up now. I had a couple of my customers even try to pull up their home site and could not get to it. For FYI :) I realize that ICMP is not the best way to test and it is not a true indication of slowness or the presence of a problem. On Oct 26, 2006, at 5:14 PM, Elijah Savage wrote:

Here is one from that browdwing ds3 to MCI well Verizon now.

5 tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9) 157.795 ms 179.050 ms tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13) 205.087 ms 6 * * ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206) 50.134 ms 7 * ae-1-100.ebr1.Chicago1.Level3.net (4.69.132.41) 45.873 ms * 8 ae-2.ebr2.NewYork1.Level3.net (4.69.132.66) 66.346 ms 72.509 ms *

---------------------------------------------------------------------- Elijah Savage | AOL IM:layer3rules Senior Network Engineer | When it has to be switched or routed. http://www.digitalrage.org | The Information Technology News Center ----- http://www.digitalrage.org/?page_id=46 for pgp public key--------

On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:

...

Can you be more specific?

-brandon

On 10/26/06, Elijah Savage <esavage@digitalrage.org> wrote: Looks like level3 is having issues. Anyone know what is going on? --------------------------------------------------------------------- - Elijah Savage | AOL IM:layer3rules Senior Network Engineer | When it has to be switched or routed. http://www.digitalrage.org | The Information Technology News Center ----- http://www.digitalrage.org/?page_id=46 for pgp public key--------

-- Brandon Galbraith Email: brandon.galbraith@gmail.com AIM: brandong00 Voice: 630.400.6992 "A true pirate starts drinking before the sun hits the yard-arm. Yarrrr. --thelost"

Which begs the same question I've asked in the recent past: then what *is* a good diagnostic tool? If ICMP "is not the best way to test", then what is? What other globally-implemented layer 3 or below protocols do we have available for troubleshooting?

Sure, UDP-based traceroute still relies on ICMP TTL exceeded responses to work. I've no idea what TCP traceroute relies on, as I haven't looked at it.

I love it when people answer their own questions and tell us that they are lazy, to boot. For the record, TCP traceroute and similar TCP based tools rely on the fact that if you send a TCP SYN packet to a host it will respond with either a TCP RST (if the port is NOT listening) or a TCP SYN/ACK. The round trip time of this provides useful information which is unaffected by any ICMP chicanery on the part of routers or firewalls. A polite application such as TCP traceroute will reply to the SYN/ACK with an RST packet so it is reasonably safe to use this tool with live services. Of course, even TCP packets can be blocked or dropped for various reasons so this is not a 100% solution. However, if you want to avoid ICMP filtering or low precedence, then TCP traceroute will help. --Michael Dillon

On Thu, 2006-10-26 at 18:01 -0400, Elijah Savage wrote:

For FYI :) I realize that ICMP is not the best way to test and it is not a true indication of slowness or the presence of a problem.

Two questions for everybody...(any and all responses appreciated, even if the reply mentions botnets or hammers ;-) ) 1) What value is ICMP if everybody pretty much considers it's accuracy suspect? 2) How does ICMP's suspect nature affect Path MTU? -Jim P.

* Jim Popovitch:

Two questions for everybody...(any and all responses appreciated, even if the reply mentions botnets or hammers ;-) )

1) What value is ICMP if everybody pretty much considers it's accuracy suspect?

The problem with ICMP-based traceroutes is that it doesn't necessarily test the path you are interested in. Use tcptraceroute or traceproto instead. Of course, this doesn't solve the problem that TTL Exceeded messages might be generated with very low priority, which is in generally a very good idea.

2) How does ICMP's suspect nature affect Path MTU?

In this case, you're interested in the ICMP payload, not the fact whether an ICMP packet goes through or not. (You lose if someone filters ICMP, though.)