Hi all, do you have any recommended tools that can measure latency/delay hop by hop basis? Preferable the tools can measure the running (live) traffic. Regards, Steven Lee
On Tue, Dec 2, 2008 at 10:47 PM, Lee, Steven (NSG Malaysia) < kin-wei.lee@hp.com> wrote:
Hi all, do you have any recommended tools that can measure latency/delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
mtr ? - http://www.bitwizard.nl/mtr/ its an ncurses app.. output like to: Host Loss% Snt Last Avg Best Wrst StDev 1. 89.202.212.124 0.0% 7 0.4 0.5 0.4 0.6 0.0 2. xe-7-1-0-0.ams-koo-score-1-re0.i 0.0% 7 0.3 2.9 0.2 18.9 7.0 3. po1.ccr01.ams03.atlas.cogentco.c 0.0% 6 1.3 3.8 1.3 16.0 6.0 4. te2-4.3490.ccr01.ams03.atlas.cog 0.0% 6 1.1 1.1 1.1 1.2 0.0 5. te2-3.mpd02.lon01.atlas.cogentco 0.0% 6 8.0 9.9 7.9 19.6 4.7 6. te9-3.ccr02.jfk02.atlas.cogentco 0.0% 6 91.9 91.9 91.8 91.9 0.0 7. te8-3.ccr01.dca01.atlas.cogentco 0.0% 6 92.0 95.0 91.7 111.2 7.9 8. te9-1.ccr02.dca01.atlas.cogentco 0.0% 6 92.1 92.2 92.1 92.4 0.1 9. vl3896.na32.b001806-1.dca01.atla 0.0% 6 92.3 93.1 92.3 94.1 0.7 10. 38.104.30.102 0.0% 6 92.0 92.0 92.0 92.1 0.1 11. www.joost.com 0.0% 6 92.0 91.9 91.8 92.0 0.1 thanks Andrew
Regards, Steven Lee
On Tue, 2 Dec 2008, Andrew Mulholland wrote:
On Tue, Dec 2, 2008 at 10:47 PM, Lee, Steven (NSG Malaysia) < kin-wei.lee@hp.com> wrote:
Hi all, do you have any recommended tools that can measure latency/delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
mtr ? - http://www.bitwizard.nl/mtr/
FWIW, Mtr measures latency/delay and loss based on ICMP messages heard back from the routers on path. As a result, in almost all cases, the real hop-by-hop latency of actual end-to-end data packets is better than it can report. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
I don't know if it helps, but anyways. There is another tool called Iperf ( http://sourceforge.net/projects/iperf ). It's usually used to report bandwidth between two hops which you define, but it can be also used to measure jitter, datagram loss, and a lot of other things, one in all it's very handy if you want to get some idea regarding the network performance. On Wed, Dec 3, 2008 at 8:52 AM, Pekka Savola <pekkas@netcore.fi> wrote:
On Tue, 2 Dec 2008, Andrew Mulholland wrote:
On Tue, Dec 2, 2008 at 10:47 PM, Lee, Steven (NSG Malaysia) < kin-wei.lee@hp.com> wrote:
Hi all, do you have any recommended tools that can measure latency/delay
hop by hop basis? Preferable the tools can measure the running (live) traffic.
mtr ? - http://www.bitwizard.nl/mtr/
FWIW, Mtr measures latency/delay and loss based on ICMP messages heard back from the routers on path. As a result, in almost all cases, the real hop-by-hop latency of actual end-to-end data packets is better than it can report.
-- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
On Wed, 3 Dec 2008, Pekka Savola wrote:
FWIW, Mtr measures latency/delay and loss based on ICMP messages heard back from the routers on path. As a result, in almost all cases, the real hop-by-hop latency of actual end-to-end data packets is better than it can report.
mtr has a recently added '-u' option to use UDP instead of ICMP echo requests. -- Antonio Querubin whois: AQ7-ARIN
On Tue, 2 Dec 2008, Antonio Querubin wrote:
On Wed, 3 Dec 2008, Pekka Savola wrote:
FWIW, Mtr measures latency/delay and loss based on ICMP messages heard back from the routers on path. As a result, in almost all cases, the real hop-by-hop latency of actual end-to-end data packets is better than it can report.
mtr has a recently added '-u' option to use UDP instead of ICMP echo requests.
But that doesn't change the gist of my message: it's still relying on ICMP ttl exceeded messages sent by the routers on the path to check the delays etc. As such it suffers from basically the same limitations as ICMP probing. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
Mtr is even less usefull then that, in its default mode it does a traceroute and then proceeds to ICMP Ping flood each IP in the list generated by the traceroute, the result is usually completly useless on WAN topologies due to asym-routing, ICMP node protections by carriers and punting etc.. And using UDP will not really provide better results due to the same thing, and IIRC Cisco from 12.0 has a standard setting of no more then 1 ICMP Unreach per 500ms.. ------------------------------ Anders Lindbäck anders.lindback@dnz.se On 3 dec 2008, at 12.00, Pekka Savola wrote:
On Tue, 2 Dec 2008, Antonio Querubin wrote:
On Wed, 3 Dec 2008, Pekka Savola wrote:
FWIW, Mtr measures latency/delay and loss based on ICMP messages heard back from the routers on path. As a result, in almost all cases, the real hop-by-hop latency of actual end-to-end data packets is better than it can report.
mtr has a recently added '-u' option to use UDP instead of ICMP echo requests.
But that doesn't change the gist of my message: it's still relying on ICMP ttl exceeded messages sent by the routers on the path to check the delays etc. As such it suffers from basically the same limitations as ICMP probing.
-- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
If the question is to measure hop by hop latency from source to destination, perhaps across routers you don't manage, how can this be done without using the ICMP time exceeded messages? End to end latency is easily done with Smokeping and the use of TCP (SYN, SYN ACK, ACK, RST) and them timestamp it. This gives you a very clear idea on application latency on any tcp port. Hop by hop is a different story and the only option I know of is with the reliance on the ICMP mechanism. One additional question on this; how do you measure hop by hop when the path dynamically changes, and then record the path change (time/date and the differences in latency on the new path)? Mike -----Original Message----- From: Anders Lindbäck [mailto:list-only@dnz.se] Sent: Wednesday, December 03, 2008 10:02 AM To: Pekka Savola Cc: nanog@nanog.org Subject: Re: Recommendation of Tools Mtr is even less usefull then that, in its default mode it does a traceroute and then proceeds to ICMP Ping flood each IP in the list generated by the traceroute, the result is usually completly useless on WAN topologies due to asym-routing, ICMP node protections by carriers and punting etc.. And using UDP will not really provide better results due to the same thing, and IIRC Cisco from 12.0 has a standard setting of no more then 1 ICMP Unreach per 500ms.. ------------------------------ Anders Lindbäck anders.lindback@dnz.se On 3 dec 2008, at 12.00, Pekka Savola wrote:
On Tue, 2 Dec 2008, Antonio Querubin wrote:
On Wed, 3 Dec 2008, Pekka Savola wrote:
FWIW, Mtr measures latency/delay and loss based on ICMP messages heard back from the routers on path. As a result, in almost all cases, the real hop-by-hop latency of actual end-to-end data packets is better than it can report.
mtr has a recently added '-u' option to use UDP instead of ICMP echo requests.
But that doesn't change the gist of my message: it's still relying on ICMP ttl exceeded messages sent by the routers on the path to check the delays etc. As such it suffers from basically the same limitations as ICMP probing.
-- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
-- THIS E-MAIL MESSAGE AND ANY FILES TRANSMITTED HEREWITH, ARE INTENDED SOLELY FOR THE USE OF THE INDIVIDUAL(S) ADDRESSED AND MAY CONTAIN CONFIDENTIAL, PROPRIETARY OR PRIVILEGED INFORMATION. IF YOU ARE NOT THE ADDRESSEE INDICATED IN THIS MESSAGE (OR RESPONSIBLE FOR DELIVERY OF THIS MESSAGE TO SUCH PERSON) YOU MAY NOT REVIEW, USE, DISCLOSE OR DISTRIBUTE THIS MESSAGE OR ANY FILES TRANSMITTED HEREWITH. IF YOU RECEIVE THIS MESSAGE IN ERROR, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND DELETE THIS MESSAGE AND ALL COPIES OF IT FROM YOUR SYSTEM.
Well, to be somewhat cynical about it, you don't. Either you have an SLA with a delay component, if that is the case the delay is either end-to-end if everything is in the same AS, or host-to-border if the other end is in some other AS. If this is the case you would already have an agreed upon measuretool (ex. SLA probes) and specified measuring points. If you dont have a specified delay clause in your SLA then it is best effort and irrelevant, in either case the hop by hop delay seen is academic at best.. So as long as your delay is lower then specified by the SLA the only result from trying to be "smart" and complaining about large hop to hop delay is that the engineer you talk to will call you names after he/she hangs up, or if it is over it is still very little/nothing you can do but wait and let the carrier engineers fix it. ------------------------------ Anders Lindbäck anders.lindback@dnz.se On 3 dec 2008, at 20.56, Braun, Mike wrote:
If the question is to measure hop by hop latency from source to destination, perhaps across routers you don't manage, how can this be done without using the ICMP time exceeded messages? End to end latency is easily done with Smokeping and the use of TCP (SYN, SYN ACK, ACK, RST) and them timestamp it. This gives you a very clear idea on application latency on any tcp port. Hop by hop is a different story and the only option I know of is with the reliance on the ICMP mechanism. One additional question on this; how do you measure hop by hop when the path dynamically changes, and then record the path change (time/date and the differences in latency on the new path)?
Mike
-----Original Message----- From: Anders Lindbäck [mailto:list-only@dnz.se] Sent: Wednesday, December 03, 2008 10:02 AM To: Pekka Savola Cc: nanog@nanog.org Subject: Re: Recommendation of Tools
Mtr is even less usefull then that, in its default mode it does a traceroute and then proceeds to ICMP Ping flood each IP in the list generated by the traceroute, the result is usually completly useless on WAN topologies due to asym-routing, ICMP node protections by carriers and punting etc..
And using UDP will not really provide better results due to the same thing, and IIRC Cisco from 12.0 has a standard setting of no more then 1 ICMP Unreach per 500ms..
------------------------------ Anders Lindbäck anders.lindback@dnz.se
On 3 dec 2008, at 12.00, Pekka Savola wrote:
On Tue, 2 Dec 2008, Antonio Querubin wrote:
On Wed, 3 Dec 2008, Pekka Savola wrote:
FWIW, Mtr measures latency/delay and loss based on ICMP messages heard back from the routers on path. As a result, in almost all cases, the real hop-by-hop latency of actual end-to-end data packets is better than it can report.
mtr has a recently added '-u' option to use UDP instead of ICMP echo requests.
But that doesn't change the gist of my message: it's still relying on ICMP ttl exceeded messages sent by the routers on the path to check the delays etc. As such it suffers from basically the same limitations as ICMP probing.
-- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
--
THIS E-MAIL MESSAGE AND ANY FILES TRANSMITTED HEREWITH, ARE INTENDED SOLELY FOR THE USE OF THE INDIVIDUAL(S) ADDRESSED AND MAY CONTAIN CONFIDENTIAL, PROPRIETARY OR PRIVILEGED INFORMATION. IF YOU ARE NOT THE ADDRESSEE INDICATED IN THIS MESSAGE (OR RESPONSIBLE FOR DELIVERY OF THIS MESSAGE TO SUCH PERSON) YOU MAY NOT REVIEW, USE, DISCLOSE OR DISTRIBUTE THIS MESSAGE OR ANY FILES TRANSMITTED HEREWITH. IF YOU RECEIVE THIS MESSAGE IN ERROR, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND DELETE THIS MESSAGE AND ALL COPIES OF IT FROM YOUR SYSTEM.
On Wed, Dec 3, 2008 at 11:01 AM, Anders Lindbäck <list-only@dnz.se> wrote:
And using UDP will not really provide better results due to the same thing, and IIRC Cisco from 12.0 has a standard setting of no more then 1 ICMP Unreach per 500ms..
then connect to a udp service with something like nsping
On Wed, 3 Dec 2008, Anders Lindbäck wrote:
Mtr is even less usefull then that, in its default mode it does a traceroute and then proceeds to ICMP Ping flood each IP in the list generated by the traceroute, the result is usually completly useless on WAN topologies due to asym-routing, ICMP node protections by carriers and punting etc..
No, it doesn't try pinging the routers in the middle, at least not anymore (I just re-checked with 0.71 and 0.75). I vaguely recall behaviour like that in the past, however, so it's possible that long time ago mtr did behave that way.
And using UDP will not really provide better results due to the same thing, and IIRC Cisco from 12.0 has a standard setting of no more then 1 ICMP Unreach per 500ms..
This is true and the point I was getting at, though I believe the bucket is much larger in any recent software release (also in 12.0 series). Actually, 5 years ago, you could see spot Cisco routers in "traceroute6" because they dropped the rate-limiter didn't respond to the middle packet and it resulted in a star. The rate-limiter has long since been fixed to be more lenient. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
On 4 dec 2008, at 14.05, Pekka Savola wrote:
On Wed, 3 Dec 2008, Anders Lindbäck wrote:
Mtr is even less usefull then that, in its default mode it does a traceroute and then proceeds to ICMP Ping flood each IP in the list generated by the traceroute, the result is usually completly useless on WAN topologies due to asym-routing, ICMP node protections by carriers and punting etc..
No, it doesn't try pinging the routers in the middle, at least not anymore (I just re-checked with 0.71 and 0.75). I vaguely recall behaviour like that in the past, however, so it's possible that long time ago mtr did behave that way.
According to the 0.75 sorcecode ICMP is still the default prot used, and the definition of MTR from bitwizards homepage disagress with you: "mtr combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool. As mtr starts, it investigates the network connection between the host mtr runs on and a user-specified destination host. After it determines the address of each network hop between the machines, it sends a sequence ICMP ECHO requests to each one to determine the quality of the link to each machine. As it does this, it prints running statistics about each machine. For a preview take a look at the screenshots." Even if you use UDP/TCP or whatever, the return packet will be ICMP and that will be ratelimited by any carrier worth there salt...
And using UDP will not really provide better results due to the same thing, and IIRC Cisco from 12.0 has a standard setting of no more then 1 ICMP Unreach per 500ms..
This is true and the point I was getting at, though I believe the bucket is much larger in any recent software release (also in 12.0 series). Actually, 5 years ago, you could see spot Cisco routers in "traceroute6" because they dropped the rate-limiter didn't respond to the middle packet and it resulted in a star. The rate- limiter has long since been fixed to be more lenient.
According to the 12.4T refrence it is still set to 1 packet / 500ms as default, however changes where made to how you can controll this in 12.4(2)T.
-- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
------------------------------ Anders Lindbäck anders.lindback@dnz.se
Hi...
According to the 0.75 sorcecode ICMP is still the default prot used, and the definition of MTR from bitwizards homepage disagress with you:
"mtr combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool. As mtr starts, it investigates the network connection between the host mtr runs on and a user-specified destination host. After it determines the address of each network hop between the machines, it sends a sequence ICMP ECHO requests to each one to determine the quality of the link to each machine. As it does this, it prints running statistics about each machine. For a preview take a look at the screenshots."
Even if you use UDP/TCP or whatever, the return packet will be ICMP and that will be ratelimited by any carrier worth there salt...
...recent attempts to get mtr working through a cisco fwsm got me sniffing, and yes indeed, icmp is the protocol in play with mtr (both outbound and inbound). Rgs, craig
perfSONAR is another, more long term solution for performance monitoring (if that's what you happen to be looking for). http://www.internet2.edu/performance/pS-PS/ -- Brad Fleming Network Engineer Kansas Research and Education Network Office: 785-856-9800 x.222 Moblie: 785-865-7231 NOC: 866-984-3662 On Dec 2, 2008, at 9:47 PM, Lee, Steven (NSG Malaysia) wrote:
Hi all, do you have any recommended tools that can measure latency/ delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
Regards, Steven Lee
Lee, Steven (NSG Malaysia) schrieb:
Hi all, do you have any recommended tools that can measure latency/delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
Try Smokeping for long-term latency stats: http://oss.oetiker.ch/smokeping/ Fredy
Lee, Steven (NSG Malaysia) wrote:
Hi all, do you have any recommended tools that can measure latency/delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
Regards, Steven Lee
Not necessarily hop-by-hop, and measures RTT rather than one-way delay, but if you can packet-sniff at two points in your network our SPP (Synthetic Packet Pairs) tool for passive round trip time estimation might be of interest - http://caia.swin.edu.au/tools/spp/documentation.html Key benefits - you don't need tight clock synchronisation between measurement points, and you can measure RTT using existing traffic flowing across your network (no additional traffic needs to be injected). Key downside - its a research project, only tested under FreeBSD, and may be entirely unsuitable :) cheers, gja
Lee, Steven (NSG Malaysia) wrote:
Hi all, do you have any recommended tools that can measure latency/delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
One other freeware/open-source tool you might want to look at is pchar which attempts to characterize available bandwidth on each link. Antonio Querubin whois: AQ7-ARIN
Am 04.12.2008 um 01:56 schrieb Antonio Querubin:
Lee, Steven (NSG Malaysia) wrote:
Hi all, do you have any recommended tools that can measure latency/ delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
One other freeware/open-source tool you might want to look at is pchar which attempts to characterize available bandwidth on each link.
intersting never heard of ;) -- pchar sends probe packets into the network of varying sizes and analyzes ICMP messages produced by intermediate routers, or by the target host. By measuring the response time for packets of different sizes, pchar can estimate the bandwidth and fixed round-trip delay along the path. pchar varies the TTL of the outgoing packets to get responses from different intermediate routers. It can use UDP or ICMP packets as probes; either or both might be useful in different situations. http://www.kitchenlab.org/www/bmah/Software/pchar/README -- thanks !!! -- Marc Manthey 50672 Köln - Germany Hildeboldplatz 1a Tel.:0049-221-3558032 Mobil:0049-1577-3329231 mail: marc@let.de PGP/GnuPG: 0x1ac02f3296b12b4d jabber :marc@kgraff.net IRC: #opencu freenode.net twitter: http://twitter.com/macbroadcast web: http://www.let.de Opinions expressed may not even be mine by the time you read them, and certainly don't reflect those of any other entity (legal or otherwise). Please note that according to the German law on data retention, information on every electronic information exchange with me is retained for a period of six months.
Lee, Steven (NSG Malaysia) wrote:
Hi all, do you have any recommended tools that can measure latency/delay hop by hop basis? Preferable the tools can measure the running (live) traffic.
Tools like smokeping, mtr, traceroute and all that will give you highly skewed results, whose accuracy will usually depend entirely on how busy the CPU is which responds to host IP packets (whether icmp, udp or even tcp). As most large routers push everything through hardware, if you measure your hop latency using inline router response times, your results will be trash. If you want to do this properly, you will need to install dedicated measurement hosts hanging off each router and measure response times to them instead. RIPE TTM boxes are pretty good for this: http://www.ripe.net/ttm/ Nick
On Wed, Dec 3, 2008 at 3:54 PM, Nick Hilliard <nick@foobar.org> wrote:
If you want to do this properly, you will need to install dedicated measurement hosts hanging off each router and measure response times to them instead. RIPE TTM boxes are pretty good for this: http://www.ripe.net/ttm/
tcptrace.org is also passive
participants (14)
-
Anders Lindbäck -
Andre Gironda -
Andrew Mulholland -
Antonio Querubin -
Brad Fleming -
Braun, Mike -
Craig Holland -
Fredy Kuenzler -
grenville armitage -
Lee, Steven (NSG Malaysia) -
Marc Manthey -
Nick Hilliard -
Pekka Savola -
Tehno Mage