RE: The Qos PipeDream [Was: RE: Two Tiered Internet]
Hi Benson, Okay -- forget about banks, forget about other comparative analogies -- let's talk about the Internet. I think Bill Manning hit on it a couple of days ago; Bill said something about the Internet being about best effort and QoS should be (various) levels of 'better-than-best effort' -- and anything less that best effort is _not_ the Internet. I completely agree with this, and I would also add that anything less than best effort is not a QoS frob, it is penalization, no matter what you want to call, and is a Bad Thing (tm). I really don't want to get into a debate on service-level semantics (e.g. WRED, etc.) but I think most reasonable people can understand what I'm trying to illustrate. This thread has gone one far enough as it stands. :-) I think that the knobs are already 'out there' for service providers, etc. to create real 'services', but to create arbitrary services just to protect one's walled garden, and/or to generate revenue (while also penalizing some customers) is something that the market will have to sort out. It always does. Vote with your dollar$. Cheers, - ferg ps. Having looked at QoS issues from the inside-out, outside-in, and various other persepctives, I do know a thing or two about it. :-) -- "Schliesser, Benson" <bensons@savvis.net> wrote: Randy- I don't think your bank analogy is very strong, but never mind that. I agree with what you're saying in principle, that if a user/customer buys bit delivery at a fixed rate then we should deliver it. But as ISPs we don't sell this. As a network operator, I do sell various kinds of point-to-point connections with fixed/guaranteed rates. But when I sell "Internet", or L3VPN, etc., I'm selling end-to-end packet-switched full-mesh connectivity. In this service, not all endpoints are equal and traffic patterns are not fixed. I.e., the service is flexible. "QoS" is about giving the customer control over what/how traffic gets treated/dropped. It's not false advertising. That said, if QoS controls are used to enforce the provider's preferences and not the customers' then I might agree with the false advertising label. If the result is to have anti-competitive effects then I might have some harsher labels for it, too. Cheers, -Benson [snip] -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/
On Thu, 15 Dec 2005, Fergie wrote:
I think Bill Manning hit on it a couple of days ago; Bill said something about the Internet being about best effort and QoS should be (various) levels of 'better-than-best effort' -- and anything less that best effort is _not_ the Internet.
AT&T, Global Crossing, Level3, MCI, Savvis, Sprint, etc have sold QOS services for years. Level3 says 20% of the traffic over its backbone is "better than Best-Effort." Ok, maybe they aren't "the Internet." Internet2 gave up on premium QOS and deployed "less-than Best Effort" scavenger class. Ok, may they aren't "the Internet" either.
I think that the knobs are already 'out there' for service providers, etc. to create real 'services', but to create arbitrary services just to protect one's walled garden, and/or to generate revenue (while also penalizing some customers) is something that the market will have to sort out. It always does.
Vote with your dollar$.
Ah, good to see that you agree with Bill Smith from BellSouth. William Smith, chief technology officer at BellSouth, argues that competitive forces, rather than regulation, are all that's needed to prevent the totalitarian online environment that the web camp fears. "We have no intention whatsoever of saying 'You can't go here, you can't go there, you can't go somewhere else'," Smith said. "We have a very competitive situation with cable. If we start trying to restrict where our customers can go on the internet, we would see our DSL customers defect to cable in droves." But, he added, "If I go to the airport, I can buy a coach standby ticket or I can buy a first class ticket from Delta. I've made a choice as to which experience I want." But also realize all companies are acting in their own self-interest, even the companies that have hire lobbyists claiming to be "saving the Internet." The enemy of your enemy isn't always your friend. I agree QOS as defined by marketeers isn't very useful. But that is a strawman argument. Of course, I understand you think its just politics. On the other hand, those same QOS tools are very useful to the network engineer for managing all sorts of network problems such as DOS attacks and disaster recovery as well as more efficiently using all the available network paths. I have no idea how all this will turn out or if there are some dark smoke-filled rooms somewhere I don't know about where the henchmen are plotting. But I would really hate to see the network engineer's hands tied by a law preventing them from managing the network because of some people spreading a lot of FUD. The news articles are filled with lots of speculation about what "could" happen, but very few facts.
On Thu, 15 Dec 2005 19:15:49 -0500 (EST) Sean Donelan <sean@donelan.com> wrote:
AT&T, Global Crossing, Level3, MCI, Savvis, Sprint, etc have sold QOS services for years. Level3 says 20% of the traffic over its
What do they mean by QoS? Is it IntServ, DiffServ, PVCs, the law of averages or something else? I've had to deploy it on a campus network and in doing so it seems like I've tread into territory where few if any big networks are to be found. Nortel apparently removed DiffServ capability for their "ISP customers" from one of their VoIP product offerings specifically because the customers didn't want it. My impression is that DiffServ is not used by those types of networks you mentioned, but I'd be interested to hear that I'm mistaken.
backbone is "better than Best-Effort." Ok, maybe they aren't "the Internet." Internet2 gave up on premium QOS and deployed "less-than Best Effort" scavenger class. Ok, may they aren't "the Internet" either.
Scavenger is not currently enabled on Abielene. In fact, no QoS mechanisms are.
On the other hand, those same QOS tools are very useful to the network engineer for managing all sorts of network problems such as DOS attacks and disaster recovery as well as more efficiently using all the available network paths.
In my experience that is easier said than done. However, you remind me of what I think is what most who say they want QoS are really after. DoS protection. By focusing on DoS mitigation instead of trying to provide service differentiation, things begin to make more sense and actually become much more practical and deployable. John
On Thu, 15 Dec 2005, John Kristoff wrote:
On Thu, 15 Dec 2005 19:15:49 -0500 (EST) Sean Donelan <sean@donelan.com> wrote:
AT&T, Global Crossing, Level3, MCI, Savvis, Sprint, etc have sold QOS services for years. Level3 says 20% of the traffic over its
What do they mean by QoS? Is it IntServ, DiffServ, PVCs, the law of
I think also mostly this applies to private network things as well... which mostly ends up being: "backups get 20% of the pipe and oracle-forms gets 70%" (or some variation on that mix... what with 8 queues or whatever on the private network you can just go to town :) ) Speaking to MCI's offering on the public network it's (not sold much) just qos on the end link to the customer... It's supposed to help VOIP or other jitter prone things behave 'better'. I'm not sure that we do much in the way of qos towards the customer aside from respecting the bits on the packets that arrive (no remarking as I recall). So, what does this get you aside from 'feeling better' ?
averages or something else? I've had to deploy it on a campus network and in doing so it seems like I've tread into territory where few if any big networks are to be found. Nortel apparently removed DiffServ
most large networks (as was said a few times I think) don't really need it in their cores. I think I've seen a nice presentation regarding the queuing delay induced on 'large pipe' networks, basically showing that qos is pointless if your links are +ds3 and not 100% full. Someone might have a pointer handy for that?
capability for their "ISP customers" from one of their VoIP product offerings specifically because the customers didn't want it. My impression is that DiffServ is not used by those types of networks you mentioned, but I'd be interested to hear that I'm mistaken.
diffserv is the devil... and I think the voip product(s) in question aren't meant to be used in places where bandwidth is the constraint :) when you back that rack-sized (not kidding) PVG15000 up to your multi-oc-12 connection area you aren't really worried about bandwidth constraints. You may, however, want to heed the documentation provided which says to never, ever, ever connect the equipment to the public network... or not.
On the other hand, those same QOS tools are very useful to the network engineer for managing all sorts of network problems such as DOS attacks and disaster recovery as well as more efficiently using all the available network paths.
WRED comes to mind for this... sure. stop the sawtooth, make it smooth baby!
In my experience that is easier said than done. However, you remind me of what I think is what most who say they want QoS are really after. DoS protection. By focusing on DoS mitigation instead of trying to provide service differentiation, things begin to make more sense and actually become much more practical and deployable.
how does qos help with a dos attack? I've struggled with this several times internally, unless you remark everyone (in which case you'll be remarking good and bad and not getting any benefit) I'm not sure it does help... I'd be happy to be shown the error of my ways/thoughts though. Oh, and don't say: "Well we qos icmp down to stop the icmp flood damage, silly!" of course you do, and your attacker says: "Gee icmp isn't working, what about UDP? What about TCP? What about I make my bots make full tcp/80 connections?" Oh.. doh! no qos helps that eh? :( I could be wrong though.
On Fri, 16 Dec 2005 03:29:29 +0000 (GMT) "Christopher L. Morrow" <christopher.morrow@mci.com> wrote:
In my experience that is easier said than done. However, you remind me of what I think is what most who say they want QoS are really after. DoS protection. By focusing on DoS mitigation instead of trying to provide service differentiation, things begin to make more sense and actually become much more practical and deployable.
how does qos help with a dos attack?
My point is that it's not QoS, it's DoS mitigation. Whatever that means to you, that is the solution I think most people may ultimately be looking for when they say they want QoS. John
On Thu, 15 Dec 2005, John Kristoff wrote:
On Fri, 16 Dec 2005 03:29:29 +0000 (GMT) "Christopher L. Morrow" <christopher.morrow@mci.com> wrote:
In my experience that is easier said than done. However, you remind me of what I think is what most who say they want QoS are really after. DoS protection. By focusing on DoS mitigation instead of trying to provide service differentiation, things begin to make more sense and actually become much more practical and deployable.
how does qos help with a dos attack?
My point is that it's not QoS, it's DoS mitigation. Whatever that means to you, that is the solution I think most people may ultimately be looking for when they say they want QoS.
ah-ha! and here I thought they wanted buzzword compliance :) From what sales/customers say it seems like they have a perception that 'qos will let me use MORE of my too-small pipe' (or not spend as fast on more pipe) more than anything else.
ah-ha! and here I thought they wanted buzzword compliance :) From what sales/customers say it seems like they have a perception that 'qos will let me use MORE of my too-small pipe' (or not spend as fast on more pipe) more than anything else.
and i wonder who is selling that need? randy
On Fri, 16 Dec 2005, Randy Bush wrote:
ah-ha! and here I thought they wanted buzzword compliance :) From what sales/customers say it seems like they have a perception that 'qos will let me use MORE of my too-small pipe' (or not spend as fast on more pipe) more than anything else.
and i wonder who is selling that need?
the wierd thing is you'd think the telco would just say: "Well gosh, sorry we can't help you squeeze 10lbs of poo into your 5lb bag, wanna by a shiney new 10lb bag?" or maybe you meant equipment vendors? :)
ah-ha! and here I thought they wanted buzzword compliance :) From what sales/customers say it seems like they have a perception that 'qos will let me use MORE of my too-small pipe' (or not spend as fast on more pipe) more than anything else. and i wonder who is selling that need? the wierd thing is you'd think the telco would just say: "Well gosh, sorry we can't help you squeeze 10lbs of poo into your 5lb bag, wanna by a shiney new 10lb bag?" or maybe you meant equipment vendors? :)
bingo! buy more, and more complex, hardware and you can charge more. what they forget to mention is that income will get blown in opex and capex (with the vendors getting the latter). randy
On Fri, 16 Dec 2005, Randy Bush wrote:
ah-ha! and here I thought they wanted buzzword compliance :) From what sales/customers say it seems like they have a perception that 'qos will let me use MORE of my too-small pipe' (or not spend as fast on more pipe) more than anything else. and i wonder who is selling that need? the wierd thing is you'd think the telco would just say: "Well gosh, sorry we can't help you squeeze 10lbs of poo into your 5lb bag, wanna by a shiney new 10lb bag?" or maybe you meant equipment vendors? :)
bingo! buy more, and more complex, hardware and you can charge more. what they forget to mention is that income will get blown in opex and capex (with the vendors getting the latter).
charge more you say?? I need to talk to our marketting dept!!! :) The world of marketting and sales is so incestuously intertwined among consumers and consumee's ... it's an amazing thing.
On Fri, 16 Dec 2005, Christopher L. Morrow wrote:
ah-ha! and here I thought they wanted buzzword compliance :) From what sales/customers say it seems like they have a perception that 'qos will let me use MORE of my too-small pipe' (or not spend as fast on more pipe) more than anything else.
When you're running voip over a T1/E1, you really want to LLQ the VOIP packets because VOIP doesn't like delay (not so much a problem) nor jitter (big problem), nor packetloss (not so much a problem if it's less than a 0.1 percent or so). So combining voip and data traffic on a link that sometimes (more often now when windows machine have a decent TCP window) go full, even just in a fraction of a second, means you either go QoS or do what Skype does, crank up the jitter buffer when there is high-jitter, which means latency for the call goes up. So prioritizing packets in the access and core is good, for access because it's usually low-bandwidth and going to higher bw to remove congestion might mean factor 10 higher bw and a serious cost, in the core it's good to handle multiple faults, if the things that never should happen, you're not dropping your customers VOIP packets when the pipe is full that 0.1% of the time. But, if you take the above model and start to always run your pipes full and use core packet prioritization as an everyday thing to support your lack of core bw, then you're in much bigger doo-doo. -- Mikael Abrahamsson email: swmike@swm.pp.se
Thus spake "Mikael Abrahamsson" <swmike@swm.pp.se>
On Fri, 16 Dec 2005, Christopher L. Morrow wrote:
ah-ha! and here I thought they wanted buzzword compliance :) From what sales/customers say it seems like they have a perception that 'qos will let me use MORE of my too-small pipe' (or not spend as fast on more pipe) more than anything else.
When you're running voip over a T1/E1, you really want to LLQ the VOIP packets because VOIP doesn't like delay (not so much a problem) nor jitter (big problem), nor packetloss (not so much a problem if it's less than a 0.1 percent or so).
There's two problems, actually. The first is serialization delay, and afflicts any link under about 3Mb/s regardless of utilization. Access speeds are finally climbing past this, but for links where they haven't you need something like MLPPP for fragmentation and interleaving. The second is queueing delay, and that tends to only matter when average utilization passes 58% (someone with a stat background explained why, but my math isn't good enough to explain it). LLQ and WRED solve this well enough for end systems to cope with the result.
So combining voip and data traffic on a link that sometimes (more often now when windows machine have a decent TCP window) go full, even just in a fraction of a second, means you either go QoS or do what Skype does, crank up the jitter buffer when there is high-jitter, which means latency for the call goes up.
Adaptive jitter buffers are old technology; Skype is hardly the first company to use them. Most phones and softphones have them; it's the gateways at the other end that are usually stuck with static ones. S Stephen Sprunk "God does not play dice." --Albert Einstein CCIE #3723 "God is an inveterate gambler, and He throws the K5SSS dice at every possible opportunity." --Stephen Hawking
On Fri, 16 Dec 2005, Stephen Sprunk wrote:
Adaptive jitter buffers are old technology; Skype is hardly the first company to use them. Most phones and softphones have them; it's the gateways at the other end that are usually stuck with static ones.
Personally I find the delay of the mobile phones (200ms or so) annoying enough and with just a static 40ms buffer you end up with 80ms of RTT in the jitterbuffer alone, adding some access technologies like SHDSL or alike it's easily over 100ms, compared to less than 20-30ms for a regular call within a region. I am not a QoS buff, but it packet prioritization does have it's place, even if it's only WFQ. -- Mikael Abrahamsson email: swmike@swm.pp.se
most large networks (as was said a few times I think) don't really need it in their cores. I think I've seen a nice presentation regarding the queuing delay induced on 'large pipe' networks, basically showing that qos is pointless if your links are +ds3 and not 100% full. Someone might have a pointer handy for that?
Here in London, we have noticed that the double-length bendy buses have a harder time moving through city streets than motor scooters do. I suspect that the studies you are referring to show that the key factor is the ratio between the size of pipe and the size of the flows moving through that pipe.
diffserv is the devil... and I think the voip product(s) in question aren't meant to be used in places where bandwidth is the constraint :)
A single VoIP call is a rather slim volume of packets compared to many other uses of the Internet. If a network doesn't have systemic jitter caused by layer 2 congestion, then one would expect VoIP to work fine on a modern network. Indeed, that is what Bill Woodcock reported a year or so ago in regard to INOC-DBA. --Michael Dillon
One thing to note here is that while VoIP flows are low volume on a bits-per-second basis, they push substantially more packets per kilobit than other traffic types - as much as 50pps per 82Kbps flow. And I have seen cases of older line cards approaching their pps limits when handling large numbers of VoIP flows even though there's plenty of throughput headroom. That's not something LLQ or priority queueing are going to be able to help you mitigate at all. -C On Dec 16, 2005, at 4:29 AM, Michael.Dillon@btradianz.com wrote:
A single VoIP call is a rather slim volume of packets compared to many other uses of the Internet. If a network doesn't have systemic jitter caused by layer 2 congestion, then one would expect VoIP to work fine on a modern network. Indeed, that is what Bill Woodcock reported a year or so ago in regard to INOC-DBA.
--Michael Dillon
Chris Woodfield wrote:
One thing to note here is that while VoIP flows are low volume on a bits-per-second basis, they push substantially more packets per kilobit than other traffic types - as much as 50pps per 82Kbps flow. And I have seen cases of older line cards approaching their pps limits when handling large numbers of VoIP flows even though there's plenty of throughput headroom. That's not something LLQ or priority queueing are going to be able to help you mitigate at all.
-C
In that vein, and not quite on this topic, it would be real nice if voip applications made an effort to stop abusing networks with unneccessarily large pps. Something about intelligent edges? The payload length of voip applications often has a lot to do with rtt. Adapting payload length to the actuall average rtt could have a positive effect on pps throughput.
Joe Maimon wrote:
Chris Woodfield wrote:
One thing to note here is that while VoIP flows are low volume on a bits-per-second basis, they push substantially more packets per kilobit than other traffic types - as much as 50pps per 82Kbps flow. And I have seen cases of older line cards approaching their pps limits when handling large numbers of VoIP flows even though there's plenty of throughput headroom. That's not something LLQ or priority queueing are going to be able to help you mitigate at all.
-C
In that vein, and not quite on this topic, it would be real nice if voip applications made an effort to stop abusing networks with unneccessarily large pps.
VoIP by design will have high PPS per connection as opposed to data flows. At 20 ms sample rates you have 50 pps regardless of the CODEC or algorithm. Increasing the time per sample to 40 ms would cut this in half but the added latency would result in degraded quality. In addition, longer sample times would suffer much more degradation if there is packet loss.
Something about intelligent edges? The payload length of voip applications often has a lot to do with rtt. Adapting payload length to the actuall average rtt could have a positive effect on pps throughput.
I'm not sure why you say the payload length has much to do with RTT. Serialization delay on slow edge links could increase RTT, but this would worsen substantially with longer samples (assuming the same CODEC and compression). Payload length is a factor of the sample length and compression algorithm. More efficient compression will result in smaller payloads but overhead becomes a higher percentage of the overall flow. Only lengthier samples will reduce PPS, and the added latency in a two-way conversation will substantially reduce call quality. -- Jay Hennigan - CCIE #7880 - Network Administration - jay@west.net NetLojix Communications, Inc. - http://www.netlojix.com/ WestNet: Connecting you to the planet. 805 884-6323
Jay Hennigan wrote:
VoIP by design will have high PPS per connection as opposed to data flows. At 20 ms sample rates you have 50 pps regardless of the CODEC or algorithm. Increasing the time per sample to 40 ms would cut this in half but the added latency would result in degraded quality. In addition, longer sample times would suffer much more degradation if there is packet loss.
My point is that sampling length should take into effect the rtt. A rtt of 200ms tolerates far shorter sampling slices than does 20ms.
On Sun, 18 Dec 2005, Joe Maimon wrote:
Something about intelligent edges? The payload length of voip applications often has a lot to do with rtt. Adapting payload length to the actuall average rtt could have a positive effect on pps throughput.
What is your suggestion? High latency connections can have higher VOIP latency by increasing packet size, or "low" IP-latency connections can handle higher VOIP latency by increasing packet size? -- Mikael Abrahamsson email: swmike@swm.pp.se
Mikael Abrahamsson wrote:
On Sun, 18 Dec 2005, Joe Maimon wrote:
Something about intelligent edges? The payload length of voip applications often has a lot to do with rtt. Adapting payload length to the actuall average rtt could have a positive effect on pps throughput.
What is your suggestion? High latency connections can have higher VOIP latency by increasing packet size, or "low" IP-latency connections can handle higher VOIP latency by increasing packet size?
The latter.
On 18/12/05, Chris Woodfield <rekoil@semihuman.com> wrote:
One thing to note here is that while VoIP flows are low volume on a bits-per-second basis, they push substantially more packets per kilobit than other traffic types - as much as 50pps per 82Kbps flow. And I have seen cases of older line cards approaching their pps limits when handling large numbers of VoIP flows even though there's plenty of throughput headroom. That's not something LLQ or priority queueing are going to be able to help you mitigate at all.
Only older line cards ? Currently NPE-G1's are causing me more headaches in that regard. At least up until last friday. /Tony
* Sean Donelan:
AT&T, Global Crossing, Level3, MCI, Savvis, Sprint, etc have sold QOS services for years. Level3 says 20% of the traffic over its backbone is "better than Best-Effort."
Well, are you sure these traffic classes are actually enforced at the router level? Maybe it's just a difference in the SLA, and the packets are still treated the same across the network.
Internet2 gave up on premium QOS and deployed "less-than Best Effort" scavenger class.
I doubt that utilization on Abilene is high enough for QoS to make any difference. 8-)
participants (13)
-
Chris Woodfield -
Christopher L. Morrow -
Fergie -
Florian Weimer -
Jay Hennigan -
Joe Maimon -
John Kristoff -
Michael.Dillon＠btradianz.com -
Mikael Abrahamsson -
Randy Bush -
Sean Donelan -
Stephen Sprunk -
tony sarendal