Re: Is latency equivalent to RTT?
Assuming that one has measuring devices in every PoP, do you think it is harder to measure a full matrix of one way latency compared to measuring a full matrix of RTT?
Your problem with 1-way is finding a way to do it thats reliable, for rtt ping is about as simple as you can get and works a treat but for 1-way you have to be a bit more creative....
I assume that it's fairly common for people to have Solaris or Linux boxes in every PoP to do measurements. In that case, the difficulty isn't in measuring one-way latency, it's in synchronizing the time on all the servers. And with fairly cheap GPS and CDMA clocks that is a lot easier/cheaper than it once was. I really think that there are a lot of people out there who do not understand that RTT is not the same thing as two times the one-way latency. In other words if you measure (A-to-B + B-to-A) / 2 then you have failed to learn anything about one-way latency on either path A-to-B or B-to-A. Yet that's precisely what people are doing when they measure RTT and then assume that RTT/2 is equal to the one-way latency A-to-B. --Michael Dillon
Michael.Dillon@radianz.com wrote:
I assume that it's fairly common for people to have Solaris or Linux boxes
in every PoP to do measurements. In that case, the difficulty isn't in measuring one-way latency, it's in synchronizing the time on all the servers. And with fairly cheap GPS and CDMA clocks that is a lot easier/cheaper than it once was.
But what GPS clock can you install in a datacenter? AFAIK, they all require roof (or at least window) access in order to install the antenna. (At least, all the GPS based ntp servers I've looked at do). Is that not true of CDMA servers? How have others solved this issue? (Short of owning their datacenters.)
On Wed, 14 May 2003, Steve Francis wrote:
Michael.Dillon@radianz.com wrote:
I assume that it's fairly common for people to have Solaris or Linux boxes
in every PoP to do measurements. In that case, the difficulty isn't in measuring one-way latency, it's in synchronizing the time on all the servers. And with fairly cheap GPS and CDMA clocks that is a lot easier/cheaper than it once was.
But what GPS clock can you install in a datacenter? AFAIK, they all require roof (or at least window) access in order to install the antenna. (At least, all the GPS based ntp servers I've looked at do).
Half the sky is sufficient, so if you have a window that can open, you are usually fine. Just stick the receiver outside the window.
Is that not true of CDMA servers?
No, they work inside buildings if there is not too much interference. An advantage of these units is that they work whereever a CDMA phone works, so it really easy to check. We have 2 CDMA units installed in data-centers and they just work.
How have others solved this issue? (Short of owning their datacenters.)
For GPS: There are units on the market that can work with 250m of regular CAT5 cable between GPS receiver and PC, if the building has structured wiring, get permission to access the roof, then use the existing wiring to reach the computer room. Getting permission to access the roof is the tricky part. Henk ------------------------------------------------------------------------------ Henk Uijterwaal Email: henk.uijterwaal@ripe.net RIPE Network Coordination Centre WWW: http://www.ripe.net/home/henk P.O.Box 10096 Singel 258 Phone: +31.20.5354414 1001 EB Amsterdam 1016 AB Amsterdam Fax: +31.20.5354445 The Netherlands The Netherlands Mobile: +31.6.55861746 ------------------------------------------------------------------------------ That problem that we weren't having yesterday, is it better? (Big ISP NOC)
On Wed, 14 May 2003, Steve Francis wrote:
Michael.Dillon@radianz.com wrote:
I assume that it's fairly common for people to have Solaris or Linux boxes
in every PoP to do measurements. In that case, the difficulty isn't in measuring one-way latency, it's in synchronizing the time on all the servers. And with fairly cheap GPS and CDMA clocks that is a lot easier/cheaper than it once was.
But what GPS clock can you install in a datacenter? AFAIK, they all require roof (or at least window) access in order to install the antenna. (At least, all the GPS based ntp servers I've looked at do). Is that not true of CDMA servers?
cdma recievers require a visible cdma signal so if your cdma phone works in the datacenter the cdma reciver will as well...
How have others solved this issue? (Short of owning their datacenters.)
Ask/pay/rent for roof access for the antenna, it's not large but it does work better with an unobstructed video of the sky. There's also the question of how many stratum=1 time sources a given datacenter needs. if there's someone else with one it should be fairly unintrusive to share. Also if you just need a high level of syncronization between the time on all your hosts you can just deploy one standalone ntp server, sync it against public time sources and get everything synced against that. its probably a 95% solution to most people's timeing needs. -- -------------------------------------------------------------------------- Joel Jaeggli Academic User Services joelja@darkwing.uoregon.edu -- PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E -- In Dr. Johnson's famous dictionary patriotism is defined as the last resort of the scoundrel. With all due respect to an enlightened but inferior lexicographer I beg to submit that it is the first. -- Ambrose Bierce, "The Devil's Dictionary"
On Wed, 14 May 2003, Joel Jaeggli wrote:
Ask/pay/rent for roof access for the antenna, it's not large but it does work better with an unobstructed video of the sky. There's also the question of how many stratum=1 time sources a given datacenter needs. if there's someone else with one it should be fairly unintrusive to share.
When I still worked at Equinix, we had NTP chimers in all the datacenters which customers could either hardwire (IRIG-B, etc) or peer with NTP. However, it seemed that people who cared about time always wanted to have their own NTP chimers which they owned and controlled. The concern about relying on third-parties, and even the theorotical NTP attacks.
On Wed, 14 May 2003, Sean Donelan wrote: When I still worked at Equinix, we had NTP chimers in all the datacenters which customers could either hardwire (IRIG-B, etc) or peer with NTP. Slick! However, it seemed that people who cared about time always wanted to have their own NTP chimers which they owned and controlled. The concern about relying on third-parties, and even the theorotical NTP attacks. A well-chosen set of sources obviates the need for this as well. NTP is pretty good at spotting and ignoring martian sources. matt ghali --mghali@snark.net------------------------------------------<darwin>< Flowers on the razor wire/I know you're here/We are few/And far between/I was thinking about her skin/Love is a many splintered thing/Don't be afraid now/Just walk on in. #include <disclaim.h>
in every PoP to do measurements. In that case, the difficulty isn't in measuring one-way latency, it's in synchronizing the time on all the servers. And with fairly cheap GPS and CDMA clocks that is a lot easier/cheaper than it once was.
a robust mesh of strat-2 chimers gives one more resilence and more accuracy than syncing off a single source.
But what GPS clock can you install in a datacenter? AFAIK, they all require roof (or at least window) access in order to install the antenna. (At least, all the GPS based ntp servers I've looked at do). Is that not true of CDMA servers?
some GPS, some PPS, and an atomic source here and there give great diversity and only a few need roof access.
How have others solved this issue? (Short of owning their datacenters.)
Use NTP, run most systems as strat-2 --bill
On Wed, May 14, 2003 at 08:02:28AM -0700, Steve Francis quacked:
But what GPS clock can you install in a datacenter? AFAIK, they all require roof (or at least window) access in order to install the antenna. (At least, all the GPS based ntp servers I've looked at do). Is that not true of CDMA servers? How have others solved this issue? (Short of owning their datacenters.)
http://www.endruntechnologies.com/ I've got about 20 of their Praecis Ct units installed in various datacenters around the world, and they work in ... most of them. There are a couple that're buried deeply enough inside buildings that they can't receive enough of a signal, but that's about only 5% or 10% of the time. Most of the time they just work.
Michael.Dillon@radianz.com wrote:
I assume that it's fairly common for people to have Solaris or Linux boxes
in every PoP to do measurements. In that case, the difficulty isn't in measuring one-way latency, it's in synchronizing the time on all the servers. And with fairly cheap GPS and CDMA clocks that is a lot easier/cheaper than it once was.
(Warning, somewhat involved rant below from someone who's spent too much time trying to measure one-way latencies): It's easier, but it's still not trivial. There are some systems level issues that still creep up when you're doing one-way measurements that you might not often think about. For instance, FreeBSD (and now Linux, I think) have a recvmsg option to get a card-level timestamp of when a packet was _received_, called SO_TIMESTAMP, but there's no standard way of finding out when packets were actually transmitted. The measurement folk at CAIDA have a kernel patch for FreeBSD that puts a kernel-level timestamp in outgoing packets to get rid of that source of uncertainty: http://www.caida.org/tools/measurement/skitter/skping/index.xml Using both of those gets you away from calling gettimeofday() to time your packets. Kernels are usually pretty good about trying to make sure the user visible clock doesn't go backwards, but when you're checking it against an external source, you can notice some really weird happenings. On FreeBSD, setting kern.timecounter.method=1 will help. Anyway, while getting the basic equipment up and running is really easy, there's more to getting accurate one-way latencies than just running ping once and a while. Be nice if we had a high-resolution equivalent of the ICMP_TSTAMP request type, so we could just use a ping variant. -Dave -- work: dga@lcs.mit.edu me: dga@pobox.com MIT Laboratory for Computer Science http://www.angio.net/ I do not accept unsolicited commercial email. Do not spam me.
On Wed, May 14, 2003 at 12:00:02PM -0400, David G. Andersen quacked:
I've got about 20 of their Praecis Ct units installed in various datacenters around the world, and they work in ... most of them.
Eek. I should clarify this: The CDMA units work in the US, some spots in canada, and korea. Unfortunately, GSM doesn't provide the same time synch capabilities, so Europe is out. -Dave -- work: dga@lcs.mit.edu me: dga@pobox.com MIT Laboratory for Computer Science http://www.angio.net/ I do not accept unsolicited commercial email. Do not spam me.
## On 2003-05-14 12:05 -0400 David G. Andersen typed: DGA> Eek. I should clarify this: The CDMA units work in the US, DGA> some spots in canada, and korea. Unfortunately, GSM doesn't DGA> provide the same time synch capabilities, so Europe is out. DGA> Just for completness Israel has a CDMA cell-operator (Yes it's not in NA but neither is (South?)Korea) -- Rafi
I really think that there are a lot of people out there who do not understand that RTT is not the same thing as two times the one-way latency. In other words if you measure (A-to-B + B-to-A) / 2 then you have failed to learn anything about one-way latency on either path A-to-B or B-to-A. Yet that's precisely what people are doing when they measure RTT and then assume that RTT/2 is equal to the one-way latency A-to-B.
What you've said is true, but there are times when RTT measurements can be useful. If you are making measurements on your own backbone and know that paths are either symmetric or have a well-know latency ratio, then RTT might not be a bad measurement. Moreover, though this isn't really what the original question regarding "latency" measures was about, there are times when you don't care much about the absolute value of the RTT, but rather you're looking for latency spikes (i.e. RTT(t) - RTT(t-1) > 20ms ) as a crude warning of either a topology change (i.e. my traffic that normally goes direct from Boston to NYC is now going via Chicago) or of some sort of link/router issues. When this happens, it might not tell you which path or device is having a problem, but it is an easy monitor to set up to tell you that something has changed... Eric :)
participants (10)
-
bmanning＠karoshi.com -
David G. Andersen -
Eric Gauthier -
Henk Uijterwaal (RIPE-NCC) -
Joel Jaeggli -
just me -
Michael.Dillon＠radianz.com -
Rafi Sadowsky -
Sean Donelan -
Steve Francis