Greetings NANOGers, Yesterday we starting noticing long delays on an ADSL connection. I spent most of the day trying to track down the problem and getting no where. Telco says they do not detect any problem on the line... so I am kind of lost. Anyone here have any ideas? Here are the specifics: This connection uses a Cisco 827 ADSL router and has several static IPs. All IPs show identical delays. Using other circuits between the same two locations, we do not see any delays. Normally on this DSL connection, local can ping remote with packet transit times around 60-70ms. Here is what we are seeing now: # ping -s SOMEHOST 68 25; sleep 1; ping -s SOMEHOST 68 25 PING SOMEHOST: 68 data bytes 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=0. time=105. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=1. time=9132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=2. time=8132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=3. time=7132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=4. time=6132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=5. time=5133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=6. time=4133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=7. time=3133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=8. time=2133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=9. time=1133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=10. time=133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=11. time=104. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=12. time=110. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=13. time=109. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=14. time=112. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=15. time=106. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=16. time=114. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=17. time=107. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=18. time=109. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=19. time=106. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=20. time=112. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=21. time=106. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=22. time=108. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=23. time=106. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=24. time=110. ms ----SOMEHOST PING Statistics---- 25 packets transmitted, 25 packets received, 0% packet loss round-trip (ms) min/avg/max = 104/1918/9132 PING SOMEHOST: 68 data bytes 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=0. time=112. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=1. time=9131. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=2. time=8132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=3. time=7132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=4. time=6132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=5. time=5132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=6. time=4133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=7. time=3132. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=8. time=2133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=9. time=1133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=10. time=133. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=11. time=111. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=12. time=106. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=13. time=109. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=14. time=116. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=15. time=108. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=16. time=107. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=17. time=113. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=18. time=106. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=19. time=107. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=20. time=108. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=21. time=108. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=22. time=105. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=23. time=109. ms 76 bytes from SOMEHOST (w.x.y.z): icmp_seq=24. time=106. ms ----SOMEHOST PING Statistics---- 25 packets transmitted, 25 packets received, 0% packet loss round-trip (ms) min/avg/max = 105/1918/9131 What really has me bugged is the pattern shown by the first dozen packets... why the relatively quick first time, followed by a long but decreasing delay which repeats every time you restart the ping (that's why I provided 2 samples)? Despite the fact that Telco says there are not any line problems, we are seeing a change in DSL performance compared to our benchmark. When we first started noticing the problem yesterday, both in and out connections were using the Fast path, but compared to the benchmark, the inbound speed had dropped to 576 and the Capacity had jumped to 99%, plus we had some RS and CRC errors on both in and out connections. Later in the day, the connection switched from using the Fast path to the Interleave path (we did nothing on our end to cause this to change) and the performance settled down to what is shown below under "DSL NOW." DSL BENCHMARK: ============== ATU-R (DS) ATU-C (US) Capacity Used: 72% 21% Interleave Fast Interleave Fast Speed (kbps): 0 960 0 256 Reed-Solomon EC: 0 0 0 0 CRC Errors: 0 0 0 0 Header Errors: 0 0 0 0 Bit Errors: 0 0 BER Valid sec: 0 0 BER Invalid sec: 0 0 DSL NOW: ======== ATU-R (DS) ATU-C (US) Capacity Used: 94% 63% Interleave Fast Interleave Fast Speed (kbps): 736 0 256 0 Reed-Solomon EC: 99 0 4 0 CRC Errors: 4 0 1 0 Header Errors: 3 0 0 0 Bit Errors: 0 0 BER Valid sec: 0 0 BER Invalid sec: 0 0 This leaves me with a few questions... 1) What exactly is the difference between the Fast and Interleave paths? 2) At the same data rates, should both Fast and Interleave show similar throughput? 3) What would cause a switch from Fast to Interleave to occur? 4) If the Telco cannot find any line problems, what can account for the increased capacity use, the decreased data rates, and the data errors? And the bottom line is: Why the bizarre ping times, and could the changes observed in DSL performance account for that... plus the near doubling of average ping transit times? TIA for everyone's expert thoughts on this problem! Jon R. Kibler A.S.E.T., Inc. Charleston, SC USA (843) 849-8214 ================================================== Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email.
Jon.Kibler@aset.com wrote:
What's the weather like? ;-) See if you can get the ADSL router to give you upstream/downstream noise margins and any other userful reporting ... AR Driver Counters Display : TX :|packets: 8597915 = direct: 2923483 + qued: 5674434 | = oamF4: 0 + oamF5: 0 + others |fail count = chNoEr: 0 + dropped: 0 |txMissIsr= 0, queCnt= 0, txOnGoing= 0 RX :|packets: 8924470 = toATM: 8919249 + loopback: 0 + errors | , where oamF4: 0, oamF5: 0 |errors = crc: 5069 + mbuf: 0 + len: 0 + pad: 0 + strayed: 151 |rxMissIsr= 0, queCnt= 0, nonAA= 0, sramErr= 0, reqSramMax= 6 |dummyIsr = 256833, fpgaIsr = 14826785 VC( 0 to 3 ) : 08924319 00000000 00000000 00000000 VC( 4 to 7 ) : 00000000 00000000 00000000 00000000 VC( 8 to 11 ) : 00000000 00000000 00000000 00000000 VC( 12 to 15 ) : 00000151 Upstream Noise Margin relative capacity occupation: 78% noise margin upstream: 11.0 db output power downstream: 16.0 dbm attenuation upstream: 31.5 db carrier load: number of bits per symbol(tone) tone 0- 31: 00 00 00 04 67 77 66 65 66 66 66 66 55 54 43 00 tone 32- 63: 00 00 00 44 55 66 66 66 66 66 66 66 66 66 26 66 tone 64- 95: 66 65 55 54 45 55 55 44 44 44 44 44 44 43 33 22 tone 96-127: 22 22 02 22 22 20 00 00 00 00 00 00 00 00 00 00 tone 128-159: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 tone 160-191: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 tone 192-223: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 tone 224-255: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Downstream Noise Margin relative capacity occupation: 95% noise margin downstream: 6.5 db output power upstream: 12.0 dbm attenuation downstream: 66.5 db carrier load: number of bits per symbol(tone) tone 0- 31: 00 00 00 04 67 77 66 65 66 66 66 66 55 54 43 00 tone 32- 63: 00 00 00 44 55 66 66 66 66 66 66 66 66 66 26 66 tone 64- 95: 66 65 55 54 45 55 55 44 44 44 44 44 44 43 33 22 tone 96-127: 22 22 02 22 22 20 00 00 00 00 00 00 00 00 00 00 tone 128-159: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 tone 160-191: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 tone 192-223: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 tone 224-255: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jon.Kibler@aset.com wrote:
Greetings NANOGers,
Yesterday we starting noticing long delays on an ADSL connection.
<snip> Assuming it is not your ISP or that the telco is the ISP. Dont believe them. Tell them to reset the port. Tell them to change the pairs. Tell them to switch your line to a different port on the dslam. Tell them to put you into a different CO. Tell them to dispatch a technician to test your line "at the nid". Get a FTP server with good connectivity on the internet and upload/download to it, measuring your speed. Show the telco low bandwidth and packet loss. Do some flood pinging (carefully). Test the line with a cheap linksys or netgear or smc or dlink or similar "broadband" residential router with ADSL modem (or even software [google for raspppoe for windows, linux has pppoe software available as well - if thats what your setup uses]). Spend a few dollars and get ADSL on another phone line if that all does not work. For the money they make off a ADSL line, a Telco is unlikely to do more than run the standard automated web testing thingy and say "Everything fine here!" and hope you dont call back and cost them more. That makes sense. The more support time and expertise expended on you, the less profit generated for them by your business. I cant count the number of "Tests perfectly!" that get resolved mysteriously inside the telco after some more harrasment. Furthermore, our experience on average is that the more the line costs per month, the better service you get on it. Typicaly with any large amount of circuits, you will find the right people in the telco who actually give a damn about you and can "get things done" Joe
On Tue, 30 Mar 2004, Joe Maimon wrote: : Jon.Kibler@aset.com wrote: : >Greetings NANOGers, : > : >Yesterday we starting noticing long delays on an ADSL connection. : <snip> : Assuming it is not your ISP or that the telco is the ISP. : Dont believe them. Tell them to reset the port. Tell them to change the NETAT! Never Ever Trust A Telco! test, test and test some more on your side and then demand they do the same. I have even had to troubleshoot their network. I did the above and then when it still didn't work everyone (my boss, my boss' boss, data center techs and the same level of telco folks all got on a conference call for The Big Blame Party. It was, once again, their fault. scott : pairs. Tell them to switch your line to a different port on the dslam. : Tell them to put you into a different CO. Tell them to dispatch a : technician to test your line "at the nid". Get a FTP server with good : connectivity on the internet and upload/download to it, measuring your : speed. Show the telco low bandwidth and packet loss. Do some flood : pinging (carefully). : : Test the line with a cheap linksys or netgear or smc or dlink or similar : "broadband" residential router with ADSL modem (or even software [google : for raspppoe for windows, linux has pppoe software available as well - : if thats what your setup uses]). : : Spend a few dollars and get ADSL on another phone line if that all does : not work. : : For the money they make off a ADSL line, a Telco is unlikely to do more : than run the standard automated web testing thingy and say "Everything : fine here!" and hope you dont call back and cost them more. That makes : sense. The more support time and expertise expended on you, the less : profit generated for them by your business. : : I cant count the number of "Tests perfectly!" that get resolved : mysteriously inside the telco after some more harrasment. Furthermore, : our experience on average is that the more the line costs per month, the : better service you get on it. Typicaly with any large amount of : circuits, you will find the right people in the telco who actually give : a damn about you and can "get things done" : : Joe : :
Jon, I saw something like this once, and it was a router bug. Your ping trace seems to be saying that after the first ping something gets stuck, and ten more pings accumulate in a queue somewhere until a ten-second timer expires, or maybe ten messages in a queue is a maximum that triggers some action. Then the queued-up pings are all released at once and after that things run normally. Very strange, but I'll bet it's some peculiarity deep in the guts of whatever box you're pinging. Maybe a process stops running and gets restarted by an inactivity timer. The ISP might be able to find something in a log file or system trace that would be a clue to someone familiar with the router's internals. Did they just have an equipment change or a software upgrade? It doesn't feel like a link outage or quality problem because it always starts after the first ping and lasts for exactly ten more. I also wonder if the ping anomaly is even the same problem as the one that's causing the poor performance you're observing on your benchmarks. Cheers, and good luck! John Renwick
You've gone from fast path to interleaved. Interleaved can inject up to 64ms of latency, in each direction, ontop of the normal line latency. (IE say 12ms loop time, interleaved can bump that up to 140ms latency.) Interleaved is used to trade latency for line stability. I'm not sure of the specifics on that however. Basically, you set your latency tolerance on the dslam, up to 64ms for up and downstream, and dependant on line conditions, your latency will vary between base loop latency and the max allowed by your tolerance. On a good line, you won't see any latency injected, a poor line will run right up to the tolerance and still retrain due to errors. You need to ask the telco why they've changed you from fast path, and request that you get put back to a fast path config. You MAY be able to restrict your dsl modem to training fast path only if they have your line set to auto for signaling. Joshua Coombs
--On 30 March 2004 07:08 -0500 Jon.Kibler@aset.com wrote:
Yesterday we starting noticing long delays on an ADSL connection
I am betting you are running ping under linux. There is some linux bugette (I think cosmetic) that occasionally causes Linux to count down (effectively) 9, 8, 7, 6, 5, 4, 3, 2, 1 secs. I once tracked it down and it was something boring like DNS resolution in ping. Check with tcpdump you are /really/ seeing that delay. Oh and make sure you are pinging something unloaded, not a router that has better things to do. I have no doubt you are seeing some delay (or you wouldn't have started looking) but I would bet ping is putting you off the scent. Alex
participants (7)
-
Alex Bligh -
Chris Brookes -
Joe Maimon -
John Renwick -
Jon.Kibler＠aset.com -
Joshua Coombs -
Scott Weeks