95th Percentile again (was RE: C&W Peering Problem?)
On Sat, Jun 02, 2001 at 05:19:16PM -0500, Albert Meyer wrote:
I almost got caught by this one a few months ago. I was fixing to sign a contract with Exodus for a 100bT circuit when I noticed some funny-looking language and asked some probing questions, and then realized that I had to double their quoted rates before comparing them to everyone else. This moved them from the front of the pack back to UU-land.
UUNet is another story. They not only charge significantly more than everyone else, but they calculate 95th percentile on the higher of incoming and outgoing rather than the average. When I asked my salesperson why she couldn't give me a competitive rate, she said "Because we're UUNet." She seemed pretty taken aback when I explained
Exodus is the worst on billing bit for bit. The way I read the Exodus 95th percentile document (though I still havn't gotten it confirmed by a person who actually knew what they were talking about), they bill for the MAXIMIUM 95th percentile, on both inbound AND outbound. UUNet bills for the MAXIMIUM 95th percentile on inbound OR outbound, whichever is higher, as does AboveNet, and probably the majority of networks trying to be like UUNet. But a significant portion of other networks will bill for the AVERAGE under the 95th percentile. I think the MAXIMIUM is unclear to a lot of people, especially the sales people if you try to get a straight answer out of them. When most people refer to the UUNet "95th percentile", this is what they mean. You take traffic samples, line them up in order, lop off the top 5%, and whatever the number is for the sample right under that is what gets multiplied by the cost per mbit. This means that if you push 1Mbps for 25 days and 10Mbps for 5 days you will pay 10 * $$$ per mbit. Average means you will pay 2.5 * $$$ per mbit (((1 * 25) + (10 * 5)) / 30), obviously a major difference. A LOT of sales people are misleading or utterly clueless about this, and a lot of providers actually WILL bill you for the AVERAGE under the 95th percentile (though if you think about it the 95th percentile makes little sense if you average it, it was designed to extract the maximium amount of money while not making people utterly afraid to burst higher). Moral of the story, check for those words "AND" vs "OR", and "MAXIMIUM" vs "AVERAGE", ask for their boss and check it again, and then get it in writing. :P -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Saturday, June 2, 2001, at 06:49 PM, Richard A. Steenbergen wrote:
UUNet bills for the MAXIMIUM 95th percentile on inbound OR outbound, whichever is higher, as does AboveNet, and probably the majority of networks trying to be like UUNet. But a significant portion of other networks will bill for the AVERAGE under the 95th percentile.
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal? Thanks, Tim
Date: Sat, 2 Jun 2001 17:28:52 -0400 From: Timothy Brown <tcb@ga.prestige.net>
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal?
I only care to mention the obvious... this is essentially the same type of billing as average-use total traffic billing. Total traffic in + out, just not divided by number of days in a month. :-) I can't recall names, but I believe that several colo shops (space + bandwidth, not carrier-neutral, a la Exodus) do this. IMHO, 95th percentile has its drawbacks. Sure, one can charge more for "peaky" customers than with average-use billing, but that can backfire in extreme cases: Recall when the Starr Report was released... 5% of a month is 1.5 days, so the heavy traffic during that time was simply "above the cutoff". Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: (316) 794-8922 --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
I believe, as well, that 95th %tile billing is quite dumb, and there are better measurements (gigs, average (which, remember is not 50th %tile)), and there are no measurements at all ($x for y mb/s, whether you use it or not). Then again, VHS beat out BetaMax. -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- On Sat, 2 Jun 2001, E.B. Dreger wrote:
Date: Sat, 2 Jun 2001 17:28:52 -0400 From: Timothy Brown <tcb@ga.prestige.net>
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal?
I only care to mention the obvious... this is essentially the same type of billing as average-use total traffic billing. Total traffic in + out, just not divided by number of days in a month. :-)
I can't recall names, but I believe that several colo shops (space + bandwidth, not carrier-neutral, a la Exodus) do this.
IMHO, 95th percentile has its drawbacks. Sure, one can charge more for "peaky" customers than with average-use billing, but that can backfire in extreme cases: Recall when the Starr Report was released... 5% of a month is 1.5 days, so the heavy traffic during that time was simply "above the cutoff".
Eddy
---------------------------------------------------------------------------
Brotsman & Dreger, Inc. EverQuick Internet Division
Phone: (316) 794-8922
---------------------------------------------------------------------------
Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature.
These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
On Sat, 2 Jun 2001, E.B. Dreger wrote:
Date: Sat, 2 Jun 2001 17:28:52 -0400 From: Timothy Brown <tcb@ga.prestige.net>
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal?
I only care to mention the obvious... this is essentially the same type of billing as average-use total traffic billing. Total traffic in + out, just not divided by number of days in a month. :-)
I can't recall names, but I believe that several colo shops (space + bandwidth, not carrier-neutral, a la Exodus) do this.
Of course any system which bills for actual usage is pretty much statistically fair, regardless of whether its measured in the average of a rate or total amount sent/received. In my experience, people who bill in actual GB transfered tend to inflate it substantially to abuse those who can't do math, but there's nothing wrong with it as a system. I think people are more used to comparing price in $$$ per Mbit/sec though. $1 per gigabyte is equivalent to $316/Mbit fairly averaged.
IMHO, 95th percentile has its drawbacks. Sure, one can charge more for "peaky" customers than with average-use billing, but that can backfire in extreme cases: Recall when the Starr Report was released... 5% of a month is 1.5 days, so the heavy traffic during that time was simply "above the cutoff".
I'm pretty sure they make out like bandits everytime there is a major spike like that. Maybe the absolute peak was shorter then 1.5 days, but 2 days later I'm sure there were still people hitting it enough to lock in a very good peak for that month. Unless the customer is specifically trying to game the system by bursting only for 4.9% worth and using inbound traffic to match, they pretty much always win. But I don't think that's unfair, if 95th percentile is the rules they wanna play by to make money off the unsuspecting then they should play by it for the plotting as well. :P -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
$1 per gigabyte is equivalent to $316/Mbit fairly averaged.
Yes, but: Let's assume that someone sells at $1/gig, then is billed $316/mb/s/mon by thier provider. Let's further assume that the customer who is buying at $1/gig is averaging 1 mb/s, but has perfect sine-wave bandwidth usage, ie, 0 kb/s at midnight, 1 mb/s at 6a, 2 mb/s and noon, 1 mb/s at 6p, and 0 mb/s again at midnight. (Agreeing that a perfect sine wave of usage is mostly unlikely, but it's a reasonable assumption that said customer won't be at the average all month). Problem: Provider is billed for 2 mb/s.
On Sun, 3 Jun 2001, Alex Rubenstein wrote:
$1 per gigabyte is equivalent to $316/Mbit fairly averaged.
Yes, but:
Let's assume that someone sells at $1/gig, then is billed $316/mb/s/mon by thier provider. Let's further assume that the customer who is buying at $1/gig is averaging 1 mb/s, but has perfect sine-wave bandwidth usage, ie, 0 kb/s at midnight, 1 mb/s at 6a, 2 mb/s and noon, 1 mb/s at 6p, and 0 mb/s again at midnight. (Agreeing that a perfect sine wave of usage is mostly unlikely, but it's a reasonable assumption that said customer won't be at the average all month). Problem: Provider is billed for 2 mb/s.
If you can't get enough extra customers based on your better pricing, don't lower your price (or sell it for $2/gig). -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Sat, Jun 02, 2001 at 05:28:52PM -0400, Timothy Brown wrote:
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal?
This may be obvious, but billing by volume (bytes transferred) places far greater availability requirements on the measurement system than rate-based charging schemes. If I am charging by the byte, I have to count every packet. If my measurement system breaks, I lose money until it is fixed. If I am charging by the 95%tile of five-minute average throughput measurements obtained during a calendar month, I can make do with much more coarse-grained sampling. Measurement system breaks, I'm quite possibly going to bill the same amount as if it hadn't broken. Do Digital Island contracts specify any interpolation they are permitted to do in the event that their traffic data acquires black spots? Or is their measurement platform good enough to be able to count every packet reliably without loss? As to other examples, volume charging is still quite common in New Zealand; people have been counting bytes and charging by the gigabyte there since the first 9k6 circuit connecting the University of Waikato to NASA went live in April 1989. See http://www2.auckland.ac.nz/net/Accounting/nze.html "New Zealand Experiences with Network Traffic Charging" by Nevil Brownlee if you're interested in the history. Joe
On Sat, 2 Jun 2001, Joe Abley wrote:
On Sat, Jun 02, 2001 at 05:28:52PM -0400, Timothy Brown wrote:
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal?
This may be obvious, but billing by volume (bytes transferred) places far greater availability requirements on the measurement system than rate-based charging schemes.
If I am charging by the byte, I have to count every packet. If my measurement system breaks, I lose money until it is fixed.
If I am charging by the 95%tile of five-minute average throughput measurements obtained during a calendar month, I can make do with much more coarse-grained sampling. Measurement system breaks, I'm quite possibly going to bill the same amount as if it hadn't broken.
No, you are confused. A rate based billing system polls a byte counter on a switch or router at set intervals (ex: every 5 mins), subtracts the previously recorded value, and divides by the number of seconds in that interval. If the polling system cannot reach the device it is monitoring, samples can be missed, this is a very old problem of rate-based monitoring. Every rate-based system of which I am aware orders these "samples" to calculate 95th percentile, so a missed sample is equivilent to a 0 sample. A rate can be interpolated for the missing time, but it is pretty much guaranteed not to be accurate, and I'd suspect a case could be made against a provider who "makes up numbers" because of a failure in their billing system. A volume based billing system on the other hand, could theoretically poll only once a billing period. In reality it would probably poll more often, both to keep the customer apprised of their currently used amount, and to prevent the possibility of counter rollovers, but it would never "miss" a billing sample. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Sat, Jun 02, 2001 at 10:15:35PM -0400, Richard A. Steenbergen wrote:
On Sat, 2 Jun 2001, Joe Abley wrote:
On Sat, Jun 02, 2001 at 05:28:52PM -0400, Timothy Brown wrote:
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal?
This may be obvious, but billing by volume (bytes transferred) places far greater availability requirements on the measurement system than rate-based charging schemes.
If I am charging by the byte, I have to count every packet. If my measurement system breaks, I lose money until it is fixed.
If I am charging by the 95%tile of five-minute average throughput measurements obtained during a calendar month, I can make do with much more coarse-grained sampling. Measurement system breaks, I'm quite possibly going to bill the same amount as if it hadn't broken.
No, you are confused.
No, just viewing the world from a strange perspective :)
A rate based billing system polls a byte counter on a switch or router at set intervals (ex: every 5 mins), subtracts the previously recorded value, and divides by the number of seconds in that interval.
Yes. I referred to the result of that calculation as the "five-minute average throughput measurement", but I was being more general about the mechanics of measurement -- in some cases there are no counters to poll (see below).
If the polling system cannot reach the device it is monitoring, samples can be missed, this is a very old problem of rate-based monitoring. Every rate-based system of which I am aware orders these "samples" to calculate 95th percentile, so a missed sample is equivilent to a 0 sample.
No. If you are missing a "five-minute average throughput measurement" for some reason, you just have fewer samples to sort at the end of the month. Chances are you still have a reasonable approximation of the 95%ile sample value, if you don't miss too many.
A rate can be interpolated for the missing time,
I agree, that would be yucky.
[...]
A volume based billing system on the other hand, could theoretically poll only once a billing period. In reality it would probably poll more often, both to keep the customer apprised of their currently used amount, and to prevent the possibility of counter rollovers, but it would never "miss" a billing sample.
If you have bytes-in/bytes-out counters to poll, then you're totally right. [you also have to deal with counters being reset to zero due to mysteriously exploding router issues]. There are cases where there are no such counters, however, such as customers who obtain transit through a shared ethernet/FDDI/ATM interface, and where equivalent counters are not available at layer-2 (e.g. someone else runs the switches, switches suck, etc). The last time I worried about this we were using an ATM network to aggregate customer PVCs, and it was not possible to obtain per-PVC stats from the routers or the switches, for various disgusting reasons. We were carrying sufficiently little international transit traffic (this was NZ) that we were able to make measurements using NeTraMeT meters to sniff-n-count all international traffic through span ports on ethernet switches. In such an environment, billing 95%ile reliably is easier than billing volume accurately. Joe
On Sat, 2 Jun 2001, Joe Abley wrote:
No. If you are missing a "five-minute average throughput measurement" for some reason, you just have fewer samples to sort at the end of the month. Chances are you still have a reasonable approximation of the 95%ile sample value, if you don't miss too many.
[...]
If you have bytes-in/bytes-out counters to poll, then you're totally right. [you also have to deal with counters being reset to zero due to mysteriously exploding router issues].
[...]
There are cases where there are no such counters, however, such as customers who obtain transit through a shared ethernet/FDDI/ATM interface, and where equivalent counters are not available at layer-2 (e.g. someone else runs the switches, switches suck, etc).
There are only two ways you can poll the rate, either you poll a "rate" value maintained on the device, or you poll a difference in bytes divide by the length of time between samples to calculate a rate. Either way, if the device does not support polling of the "interface" in question you are pretty screwed. No matter how you stack it, if you miss a rate sample there is no way to go back and get the data again. You either discard it and lose the ability to bill the customer for it (which demands high availability polling systems), or you make up a number and hope the customer doesn't notice. Volume polling does not suffer from this problem. Volume polling does have more difficulty detecting corruption of the counters (due to a mysteriously exploding router, etc), for example adding a GB that wasn't actually transfered while a corrupted rate sample would be discarded in the 95th percentile. There are plenty of ways to detect this kind of thing though, and you could always just discard the top X% of volume samples just in case. On a side note, there is something neat to be said for the potential of "push billing" on a Juniper, by running a local program which collects billing information and never risks being unable to reach the device, then pushes the data out when it is convenient to do so. This could also be used to get more accurate samples and reduce the load of polling. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Sat, Jun 02, 2001 at 11:17:48PM -0400, Richard A. Steenbergen wrote:
There are only two ways you can poll the rate, either you poll a "rate" value maintained on the device, or you poll a difference in bytes divide by the length of time between samples to calculate a rate. Either way, if the device does not support polling of the "interface" in question you are pretty screwed.
Not necessarily; you just have to find other ways of measuring, as we did, to good effect.
No matter how you stack it, if you miss a rate sample there is no way to go back and get the data again. You either discard it and lose the ability to bill the customer for it (which demands high availability polling systems), or you make up a number and hope the customer doesn't notice.
No -- there is no need to do that. You don't need a sample for every single five-minute interval during the month to produce a meaningful 95%ile measurement for the month; you just need a representative sample population. You increase the chances of your sample population being representative if you consider lots of samples, but dropping one or two does not mean you lose revenue.
Volume polling does not suffer from this problem.
It does, if you don't have per-customer interface counters. You need to count every packet using some other method, and if you can't count packets, you can't bill for them. Joe
On Sat, Jun 02, 2001 at 11:30:24PM -0400, Joe Abley wrote:
It does, if you don't have per-customer interface counters. You need to count every packet using some other method, and if you can't count packets, you can't bill for them.
i gave up on per-customer interface accounting, didn't scale for me. for a while, i had a BSD box in the middle of my network, and i used ipfw rules (which worked both as counters for accounting, and as ingress/egress filters). we've since moved to cisco, and, well, now i have cache flow stats which are parsed into customer subnets. unfortuneately, i've practically had to install seperate interfaces for the cache flow data, as it is a steady huge flow of data, especially for sub-30 minute periods. -- [ Jim Mercer jim@reptiles.org +1 416 410-5633 ] [ Now with more and longer words for your reading enjoyment. ]
On Sat, Jun 02, 2001 at 11:58:33PM -0400, Joe Abley wrote:
On Sat, Jun 02, 2001 at 11:37:56PM -0400, Jim Mercer wrote:
we've since moved to cisco, and, well, now i have cache flow stats which are parsed into customer subnets.
How do you bill? Per byte, flat-rate, some measurement of rate, or other?
i have the ability to do per-byte or 95th percentile. most customers are 95th percentile. -- [ Jim Mercer jim@reptiles.org +1 416 410-5633 ] [ Now with more and longer words for your reading enjoyment. ]
i gave up on per-customer interface accounting, didn't scale for me.
Thats a very bold statement. A what point (what metric?) did you feel that this method didn't scale? NAC is no super-duper tier-1 (I had to throw that in), but we do monitor
1400 interfaces every 5 minutes, 100 or so at more than 105 mb/s (that magic number for 32 bit counter rollover in 5 minutes, and yes, we use 64 bit counters), shove them all in a nice SQL table, and we've not seen any reason for non-scalibilty, at least for a while (at least 5000 more interfaces before will have to rewrite the collection engine).
we've since moved to cisco, and, well, now i have cache flow stats which are parsed into customer subnets.
Eeek. Relying on flow-stats? Yikes.
On Sun, Jun 03, 2001 at 12:40:44AM -0400, Alex Rubenstein wrote:
i gave up on per-customer interface accounting, didn't scale for me.
Thats a very bold statement. A what point (what metric?) did you feel that this method didn't scale?
i'm not super-duper, but i'm tier2 and the bulk of my business is wholesale (the basic service is a connection and transit, nothing else). most of my customers are ethernet connected, and some customers share an interface. i got into this before it was cheap to do 802.11q switching, so my billing system needed to deal with multiple customers on a single ethernet.
NAC is no super-duper tier-1 (I had to throw that in), but we do monitor
1400 interfaces every 5 minutes, 100 or so at more than 105 mb/s
i don't have near that many interfaces. however, the rollover issues were starting to become apparent. fortuneately with the BSD and cache flow stuff, i get 64bit counters.
we've since moved to cisco, and, well, now i have cache flow stats which are parsed into customer subnets.
Eeek. Relying on flow-stats? Yikes.
its working for me. -- [ Jim Mercer jim@reptiles.org +1 416 410-5633 ] [ Now with more and longer words for your reading enjoyment. ]
i'm not super-duper, but i'm tier2 and the bulk of my business is wholesale (the basic service is a connection and transit, nothing else).
most of my customers are ethernet connected, and some customers share an interface. i got into this before it was cheap to do 802.11q switching, so my billing system needed to deal with multiple customers on a single ethernet.
Thats irrelevant; look at the counters on the customers switch-port. Doesn't matter what VLAN (or none) they are on.
however, the rollover issues were starting to become apparent. fortuneately with the BSD and cache flow stuff, i get 64bit counters.
As does any modern-day IOS (except for some reason, 64 bit counters seem broken on Cat 3500 XLs; anyone else seen this?)
its working for me.
Is it? Have you verified that, in actuality, it is accurate? Having done a small-bit of verification on flowstats versus counting bytes via SNMP, i have caught some interesting anamolies.
On Sun, Jun 03, 2001 at 01:34:08PM -0400, Alex Rubenstein wrote:
i'm not super-duper, but i'm tier2 and the bulk of my business is wholesale (the basic service is a connection and transit, nothing else).
most of my customers are ethernet connected, and some customers share an interface. i got into this before it was cheap to do 802.11q switching, so my billing system needed to deal with multiple customers on a single ethernet.
Thats irrelevant; look at the counters on the customers switch-port. Doesn't matter what VLAN (or none) they are on.
ah, well, again, i was using multiport bsd implementations, not switches. as it stands, i'm fairly content using the cache flow from the cisco's. one day, i might actually install a full 802.11q compliant switching framework, but, well, not today. my comments were intended for those who may not have the economic or political management support for a full 802.11q switched framework, and that it is possible to do reasonable billing without going there. your mileage may vary.
its working for me.
Is it? Have you verified that, in actuality, it is accurate?
accurate? unless you have certified CPA's inspecting each packet of traffic, on what basis are you determining accuracy? you have to trust something.
Having done a small-bit of verification on flowstats versus counting bytes via SNMP, i have caught some interesting anamolies.
i have seen a number of interesting anomalies on both the flowstats and SNMP. flowstats (if you have sufficient storage) at least allows you to replay the events leading up to the anomaly. SNMP, as good as it is, is basically use it or lose it. either you trust the data, or you don't, as there is not a lot of detail there. aside from the fact that i'm yet to find a way to use SNMP/cisco to allow me to do accurate (without totally swamping the CPU) details of per-subnet bandwidth usage without per subnet interfaces. -- [ Jim Mercer jim@reptiles.org +1 416 410-5633 ] [ Now with more and longer words for your reading enjoyment. ]
On Sat, 2 Jun 2001, Joe Abley wrote:
No matter how you stack it, if you miss a rate sample there is no way to go back and get the data again. You either discard it and lose the ability to bill the customer for it (which demands high availability polling systems), or you make up a number and hope the customer doesn't notice.
No -- there is no need to do that. You don't need a sample for every single five-minute interval during the month to produce a meaningful 95%ile measurement for the month; you just need a representative sample population. You increase the chances of your sample population being representative if you consider lots of samples, but dropping one or two does not mean you lose revenue.
Actually you gain revenue if you drop samples below the 95th percentile mark, since you are forcing the cutoff point higher by reducing the number of samples. I think your argument is in favor of 95th percentile vs an accurate average, not rate vs amount samples. If for some reason you lose a sample with an average system, your revenue goes down, whereas if you lose a sample in 95th percentile you're more likely not to make it go down much. But this is completely circumvented by polling the amount instead of polling the rate. Measurements in amount are always better then measurements by rate. If you have some horribly ghetto hack that makes you count the packets yourself and you have the possibility of missing samples, it may not be completely better then 95th percentile, but this is a seperate issue.
Volume polling does not suffer from this problem.
It does, if you don't have per-customer interface counters. You need to count every packet using some other method, and if you can't count packets, you can't bill for them.
I'd say the real problem is with the vendor. Fortunantly most people have counters. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Sun, Jun 03, 2001 at 12:04:03AM -0400, Richard A. Steenbergen wrote:
On Sat, 2 Jun 2001, Joe Abley wrote:
No matter how you stack it, if you miss a rate sample there is no way to go back and get the data again. You either discard it and lose the ability to bill the customer for it (which demands high availability polling systems), or you make up a number and hope the customer doesn't notice.
No -- there is no need to do that. You don't need a sample for every single five-minute interval during the month to produce a meaningful 95%ile measurement for the month; you just need a representative sample population. You increase the chances of your sample population being representative if you consider lots of samples, but dropping one or two does not mean you lose revenue.
Actually you gain revenue if you drop samples below the 95th percentile mark, since you are forcing the cutoff point higher by reducing the number of samples.
Right. So, dropping samples != dropping revenue.
I think your argument is in favor of 95th percentile vs an accurate average, not rate vs amount samples. If for some reason you lose a sample with an average system, your revenue goes down, whereas if you lose a sample in 95th percentile you're more likely not to make it go down much.
Not really. For any averaging function you care to apply to the sample population, there will be some samples that tend to increase the result, and some that tend to decrease the result. Whether or not the billable value goes up or down depends on the sample that was dropped, on the remaining samples, and on the averaging function being used. I don't see how you can say in general that losing a sample "with an average system" makes revenue go down. You can certainly speculate about particular "averaging" functions being more likely to increase or decrease given random loss from a particular sample distribution, but that wasn't what we were talking about (we were talking about rate vs. volume).
But this is completely circumvented by polling the amount instead of polling the rate. Measurements in amount are always better then measurements by rate.
Always?
If you have some horribly ghetto hack that makes you count the packets yourself and you have the possibility of missing samples, it may not be completely better then 95th percentile, but this is a seperate issue.
Except in this case, maybe :)
Volume polling does not suffer from this problem.
It does, if you don't have per-customer interface counters. You need to count every packet using some other method, and if you can't count packets, you can't bill for them.
I'd say the real problem is with the vendor. Fortunantly most people have counters.
Suppose you are selling transit to several customers across a switch operated by someone else (an exchange operator, for example), such that the traffic for several customers is carried by a single interface on your router. Suppose direct interconnects are not practical, and suppose you have no access to any counters that may be available on the switch. The options are: (1) do not sell to these customers, or (2) find some way to sell to these customers by counting packets yourself. Option (1) presents a far more consistent opportunity to decrease potential revenue than does option (2). I do not believe this is a particularly far-fetched scenario: hence I think this is not simply a vendor problem. Joe
On Sun, 3 Jun 2001, Joe Abley wrote:
I think your argument is in favor of 95th percentile vs an accurate average, not rate vs amount samples. If for some reason you lose a sample with an average system, your revenue goes down, whereas if you lose a sample in 95th percentile you're more likely not to make it go down much.
Not really. For any averaging function you care to apply to the sample population, there will be some samples that tend to increase the result, and some that tend to decrease the result. Whether or not the billable value goes up or down depends on the sample that was dropped, on the remaining samples, and on the averaging function being used.
No, you're working under the assumption that the divisor goes up only with increased samples, while the system I outlined continues to go up with the progression of time. No reason that can't be changed though, and that isn't important to the argument... :P
I'd say the real problem is with the vendor. Fortunantly most people have counters.
Suppose you are selling transit to several customers across a switch operated by someone else (an exchange operator, for example), such that the traffic for several customers is carried by a single interface on your router. Suppose direct interconnects are not practical, and suppose you have no access to any counters that may be available on the switch.
The options are: (1) do not sell to these customers, or (2) find some way to sell to these customers by counting packets yourself. Option (1) presents a far more consistent opportunity to decrease potential revenue than does option (2).
You can do it with VLANs, I believe Equinix does this on their exchange switches. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
[ On Saturday, June 2, 2001 at 23:17:48 (-0400), Richard A. Steenbergen wrote: ]
Subject: Re: 95th Percentile again (was RE: C&W Peering Problem?)
No matter how you stack it, if you miss a rate sample there is no way to go back and get the data again. You either discard it and lose the ability to bill the customer for it (which demands high availability polling systems), or you make up a number and hope the customer doesn't notice. Volume polling does not suffer from this problem.
What the heck are you talking about? Only a totally amateur design would fail to account for the possibility of a dropped sample (or any other of several critical issues faced by anyone using counters to determine the average or Nth percentile rates). In fact the accounting for bulk throughput per period is done in almost exactly the same as any rate-based accounting too (only the counter sample time might differ, but of course you can't stretch it too far for the former case lest you risk an undetectable wrap-around event). -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
On Sun, 3 Jun 2001, Greg A. Woods wrote:
[ On Saturday, June 2, 2001 at 23:17:48 (-0400), Richard A. Steenbergen wrote: ]
Subject: Re: 95th Percentile again (was RE: C&W Peering Problem?)
No matter how you stack it, if you miss a rate sample there is no way to go back and get the data again. You either discard it and lose the ability to bill the customer for it (which demands high availability polling systems), or you make up a number and hope the customer doesn't notice. Volume polling does not suffer from this problem.
What the heck are you talking about? Only a totally amateur design would fail to account for the possibility of a dropped sample (or any other of several critical issues faced by anyone using counters to determine the average or Nth percentile rates).
In fact the accounting for bulk throughput per period is done in almost exactly the same as any rate-based accounting too (only the counter sample time might differ, but of course you can't stretch it too far for the former case lest you risk an undetectable wrap-around event).
Actually I was refering to the more common methods of rate based measurement, MRTG. The names of the providers who use this will be omitted. :P -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
[ On Sunday, June 3, 2001 at 01:04:00 (-0400), Richard A. Steenbergen wrote: ]
Subject: Re: 95th Percentile again!
Actually I was refering to the more common methods of rate based measurement, MRTG.
Well, MRTG works the way I said, but it also fails to account for several very critical issues necessary for any auditable accounting system. Using it for billing is, how shall I say, not smart! If I'm not mistaken it's author even warns against this kind of use. Certainly cricket users are warned regularly not to use it for billing purposes, and it's even less likely to make an error than MRTG (particularly recent versions that do NOT use the "derive" method for counter variables -- "derive" mimics MRTG's errant methods, but is more "robust" in the face of errant SNMP agents, at least for capacity planning purposes).
The names of the providers who use this will be omitted. :P
Usually the errors that can happen to MRTG will be in the customer's favour, but not always. Any providers using MRTG to calculate rates for billing purposes should probably call their lawyers on Monday AM and make sure their bases are covered should any of their customers gain even half a clue. Any customers who are being billed by an ISP using MRTG should quickly install their own more accurate and properly designed accounting system and monitor their end of the pipe to make sure that any errors in MRTGs measurements remain in their favour. -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
Any customers who are being billed by an ISP using MRTG should quickly install their own more accurate and properly designed accounting system and monitor their end of the pipe to make sure that any errors in MRTGs measurements remain in their favour.
Pretty much every billing scheme is based upon statistical sampling in some form. It's not exactly fair to ignore sampling errors in your favor and then cry foul should the odds go against you. On the other hand, providers that use statistical sampling should disclose that to their customers so that they understand that they're being billed using systems that aren't necessarily 100% reproducible. DS
[ On Saturday, June 2, 2001 at 22:23:50 (-0700), David Schwartz wrote: ]
Subject: RE: 95th Percentile again!
Pretty much every billing scheme is based upon statistical sampling in some form.
Huh? No proper scheme of usage-based accounting, be it a bulk- throughput measurment, or a 95th percentile measurement, is in any way based on "statistical sampling"! Both schemes involve counting each and every byte passed thorugh the pipe, and indeed of keeping an accurate timestamp for each sample too (if you're interested in being able to audit your results). So long as there's no loss/noise on the pipe then both schemes mathematically must produce the same results on both ends of the pipe. I.e. both the total byte counts per billing period must match, as must the level of the 95th percentiles of rates calculated from these samples. Although there are some schemes that seem to allow you to divide your billing period into segments and "drop" most of the samples which calculate to rates under the Nth percentile after each segment, even they do not equate to a "statistical sampling". All of the data is considered in detail and none is actually thrown away or ignored until after the necessary calculations and checks have been made with it -- it's just that the resulting data set isn't possible to audit after the fact.
It's not exactly fair to ignore sampling errors in your favor and then cry foul should the odds go against you.
Indeed. Fortunately it's not necessary to regularly put up with such sampling errors (at least not so long as your router/switch/whatever has a properly implemented SNMP agent or other reliable means to access its interface byte counters).
On the other hand, providers that use statistical sampling should disclose that to their customers so that they understand that they're being billed using systems that aren't necessarily 100% reproducible.
The phrase "statistical sampling" would suggest that you're thinking of some scheme where periodic samples are taken of the counters and then these values are used on the spot to calculate throughput and then those throughput numbers archived over time and used periodically to estimate the average throughput over time. I suppose this is in effect what you might end up with if you used the "consolidated" part of RRDtool data, such as from the monthly graph generated by Cricket (i.e. if don't keep all samples for at least your full billing period, if not two periods). MRTG results are probably similarly unauditable from an accounting point of view. However as we already know it's not very wise to use even a properly and carefully configured Cricket, let alone MRTG, for billing purposes. -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
[ On Saturday, June 2, 2001 at 22:23:50 (-0700), David Schwartz wrote: ]
Subject: RE: 95th Percentile again!
Pretty much every billing scheme is based upon statistical sampling in some form.
Huh? No proper scheme of usage-based accounting, be it a bulk- throughput measurment, or a 95th percentile measurement, is in any way based on "statistical sampling"!
Both schemes involve counting each and every byte passed thorugh the pipe, and indeed of keeping an accurate timestamp for each sample too (if you're interested in being able to audit your results). So long as there's no loss/noise on the pipe then both schemes mathematically must produce the same results on both ends of the pipe. I.e. both the total byte counts per billing period must match, as must the level of the 95th percentiles of rates calculated from these samples.
I don't agree that this is so for 95th percentile. Exactly which five minute interval a packet is counted in will affect the results. There is no way to totally agree on which such interval a packet belongs in. Similarly, where the five-minute intervals begin and end is arbitrary and affects the final numbers. Now it's perfectly reasonable for both ends to agree that the provider will do the sampling and the provider's results, unless in actual error, shall be the basis for the billing. Nevertheless, the agreement is to use a billing scheme based upon statistical sampling.
It's not exactly fair to ignore sampling errors in your favor and then cry foul should the odds go against you.
Indeed. Fortunately it's not necessary to regularly put up with such sampling errors (at least not so long as your router/switch/whatever has a properly implemented SNMP agent or other reliable means to access its interface byte counters).
The interface byte counters won't tell you where the packets went. So any such billing scheme would be based ultimately upon statistical sampling. The provider would determine that typically some of your packets are local and cost very little and some are remote and may cost much more. Rather than counting each packet and figuring out its cost, the provider relies upon prior statistical sampling to come up with some 'average' cost which he bills you on the basis of. Sometimes what happens in this case is the customer or the provider realize that this particular traffic pattern does not match the statistical sample on which the billing was based. Richard Steenbergen told me a story about a company that colocated all their servers at POPs of the same provider and paid twice for traffic between their machines. Needless to say, they had to negotiate new pricing. Why? Because their traffic pattern made the statistical sampling upon which their billing was based inappropriate. If a billing scheme were not based upon statistical sampling, it would require the provider to somehow accurately determine how much each packet cost him to get to you or handoff from you and bill you based upon that on something like a cost plus basis.
However as we already know it's not very wise to use even a properly and carefully configured Cricket, let alone MRTG, for billing purposes.
I agree, but all of the alternatives are ultimately based upon statistical sampling. NetFlow, for example, loses a certain percentage of the packets because it's UDP based. The provider compensates for this by raising his rates. If he expects 3% of his accounting records to be lost, he raises his rates to 103% hoping that he'll get a fair statistical sample. If this assumption is violated, for example if packets are more likely to drop at peak times and a particular customer passes most of their traffic at peak times, then the statistical assumptions upon which the billing is based will be violated, and the ISP will get taken advantage of. If he counts bytes out an Ethernet port, he'll be billing you for some broadcast traffic that costs him nothing. He'll be billing you for some local traffic that costs him nothing. He'll be billing you for some short-range traffic that costs him very little. But he uses statistical sampling to come up with some 'per byte' cost. If, for example, most of a particular customer's traffic is from another customer in the same POP, again the statistical assumptions upon which the billing is based will be violated, and the customer will likely have to negotiate some other billing mechanism. Every billing scheme I have ever seen has been based upon statistical sampling. The closest to an exception I've seen is Level3's distance-based scheme. DS
On Sat, Jun 02, 2001 at 11:59:17PM -0700, David Schwartz wrote:
I don't agree that this is so for 95th percentile. Exactly which five minute interval a packet is counted in will affect the results. There is no way to totally agree on which such interval a packet belongs in. Similarly, where the five-minute intervals begin and end is arbitrary and affects the final numbers.
It doesn't! You might get a different result as two samplers do not start at the same time. With 5-minute-sampling this max. differs for <5 minutes. You are always counting a five-minute-average and *NOT* the current bandwidth consumption. If you were doing so you are absolutely right. But that's a complete diffent story. So adding up all the five-minute-samples should yield the same amount as if you would have counted bytes. --Arnold
On Sat, Jun 02, 2001 at 11:59:17PM -0700, David Schwartz wrote:
Sometimes what happens in this case is the customer or the provider realize that this particular traffic pattern does not match the statistical sample on which the billing was based. Richard Steenbergen told me a story about a company that colocated all their servers at POPs of the same provider and paid twice for traffic between their machines. Needless to say, they had to negotiate new pricing. Why? Because their traffic pattern made the statistical sampling upon which their billing was based inappropriate.
With IP you can't say who has to pay for traffic. Sender or recievier. Therefore you bill both. With what we called multi-POP customers you surely have to take care as those customers don't want to pay for traffic twice. But it's not so difficult to implement this into your billing system.
If a billing scheme were not based upon statistical sampling, it would
Once again: five-minute-bandwidth-average nor counting bytes is "statistical sampling". Don't mix up definitions. There is a well founded theory of statistical sampling. But that does not apply to what we are talking about. But of course each measurement is error proned. And "quality providers" tell their customers how accurate their accountings are. -- Arnold
[ On Saturday, June 2, 2001 at 23:59:17 (-0700), David Schwartz wrote: ]
Subject: RE: 95th Percentile again!
I don't agree that this is so for 95th percentile. Exactly which five minute interval a packet is counted in will affect the results. There is no way to totally agree on which such interval a packet belongs in. Similarly, where the five-minute intervals begin and end is arbitrary and affects the final numbers.
Perhaps you should sit down with a table of numbers and compare the results by hand. I think you'll find that you are gravely mistaken. (I can provide you with some raw numbers that are guaranteed to have been sampled out-of-sync at the ends of the same pipe if you'd like.) The only time there can ever be a descrepancy is at the "edge". I.e. if during the last sample time in the billing period the ISP sees a huge count of bytes, but the customer (because his last full sample was five minutes less one second before the end of the period) sees zero bytes, *AND* iff this one large sample throws the Nth percentile calculation for the entire billing period up over the next billing increment, then the lack of syncronisation will cause a "problem" (for the customer in this case :-). However the chances of this kind of error happening in real life are so tiny as to be almost impossible (at least if the billing period is orders of magnitude larger than the sample period, which of course is what we're supposing here). I count over three orders of magnitude difference for a 30-day billing period and a 5-min sample period. For the customer it's easy to avoid too -- just unplug your network (scheduled down time) during the 10-minute period between billing cyle roll-overs. :-)
The interface byte counters won't tell you where the packets went.
Clearly if the ISP is at one end of the pipe and the customer's at the other then the out/in (and in/out at the other end) counters are an extremely accurate count of where the packets went! Obviously such a scheme "limits" in some ways the viable alternatives for connecting customers, and it certainly forces you to do your data collection at specific points.
So any such billing scheme would be based ultimately upon statistical sampling.
Please try and talk sense man! Regardless of what you're buying or selling there's absolutely NOTHING "statistical" about byte counting! It's pure accounting, plain and simple. It's 100% auditable and 100% verifiable too!
The provider would determine that typically some of your packets are local and cost very little and some are remote and may cost much more. Rather than counting each packet and figuring out its cost, the provider relies upon prior statistical sampling to come up with some 'average' cost which he bills you on the basis of.
The only way to do that is to count flows instead of bytes and the only way I know of doing that is indeed based only on statistical sampling. Any customer who'd be willing to suffer under such a scheme is either not very clueful or getting one heck of a deal on their pricing....
Sometimes what happens in this case is the customer or the provider realize that this particular traffic pattern does not match the statistical sample on which the billing was based. Richard Steenbergen told me a story about a company that colocated all their servers at POPs of the same provider and paid twice for traffic between their machines. Needless to say, they had to negotiate new pricing. Why? Because their traffic pattern made the statistical sampling upon which their billing was based inappropriate.
You're talking apples and oranges -- please stop mis-directing the topic in an apparent attempt to "call the kettle black".
If a billing scheme were not based upon statistical sampling, it would require the provider to somehow accurately determine how much each packet cost him to get to you or handoff from you and bill you based upon that on something like a cost plus basis.
Iff. but that's not what we're talking about here.
I agree, but all of the alternatives are ultimately based upon statistical sampling. NetFlow, for example, loses a certain percentage of the packets because it's UDP based. The provider compensates for this by raising his rates. If he expects 3% of his accounting records to be lost, he raises his rates to 103% hoping that he'll get a fair statistical sample. If this assumption is violated, for example if packets are more likely to drop at peak times and a particular customer passes most of their traffic at peak times, then the statistical assumptions upon which the billing is based will be violated, and the ISP will get taken advantage of.
Duh. But this isn't what we're talking about.
If he counts bytes out an Ethernet port, he'll be billing you for some broadcast traffic that costs him nothing. He'll be billing you for some local traffic that costs him nothing. He'll be billing you for some short-range traffic that costs him very little. But he uses statistical sampling to come up with some 'per byte' cost. If, for example, most of a particular customer's traffic is from another customer in the same POP, again the statistical assumptions upon which the billing is based will be violated, and the customer will likely have to negotiate some other billing mechanism.
I don't see the problem. It's a very simple matter to adjust the pricing to fit. You can do some "statistical sampling" to set the price, just like anyone might do in any form of cost estimation, but what's on the invoice in the end is a pure accounting of the actual traffic. You can do the same for packet loss too. It's only the price/unit that's based on statistical sampling and cost estimates. Why is this so difficult for some people to understand?
Every billing scheme I have ever seen has been based upon statistical sampling. The closest to an exception I've seen is Level3's distance-based scheme.
You've obviously never looked beyond the silly schemes you're apparently stuck on talking about. I know of many billing systems that are based on pure bulk-throughput accounting and several that are based on true Nth percentile usage. None of them, not a single one, are based on statistical samples of anything -- *ALL* are pure 100% byte-counting and all of them count each and every byte. -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
"samples" to calculate 95th percentile, so a missed sample is equivilent to a 0 sample. A rate can be interpolated for the missing time, but it is pretty much guaranteed not to be accurate, and I'd suspect a case could be made against a provider who "makes up numbers" because of a failure in their billing system.
Or, just take the next sample and divide it by 10 minutes, rather than 5, and count it as two samples in the 95th calculation.
If I am not mistaken, the true "benefit" to 95% billing is that it allows the provider to charge for bits they never delivered. The average will skew on a burst of traffic (>5% of the average) and you pay for it as if you had averaged that level the entire time. It seems like quite an irrational settlement model. Why not simply bill for every bit that crosses your network? There certainly is a per-bit cost. I cannot, off the top of my head, think of another telecommunications industry that relies on a system of averages for settlement. It speaks pretty clearly of how immature the Internet industry really is. Or maybe not. Perhaps the electrical suppliers here in California should bill in the 95th percentile, and cite the Internet as a rational example. Regards, James
On Sun, 3 Jun 2001, James Thomason wrote:
If I am not mistaken, the true "benefit" to 95% billing is that it allows the provider to charge for bits they never delivered. The average will skew on a burst of traffic (>5% of the average) and you pay for it as if you had averaged that level the entire time.
It seems like quite an irrational settlement model. Why not simply bill for every bit that crosses your network? There certainly is a per-bit cost.
You have to provision for peaks, even though a lot of the time a lot of links are used to perhaps 20-30%, but they must be able to handle peak loads during office hours etc.
I cannot, off the top of my head, think of another telecommunications industry that relies on a system of averages for settlement. It speaks pretty clearly of how immature the Internet industry really is.
Over here you pay for available bandwidth. You want E1? You pay for E1. You want E3? You pay for full E3 (or whatever it might be ratelimited to). There is no check to see how much of it you actually use. Is that a better model?
Or maybe not. Perhaps the electrical suppliers here in California should bill in the 95th percentile, and cite the Internet as a rational example.
When I was in the bay area I was told that the power companies charged at whatever maximum rate you could use. You want a 1000 amp circuit into your data center? (or whatever). You pay for 1000 amps, whether you use 1 or 1000 Amps. Is that better? For whom? I don't know anyway. -- Mikael Abrahamsson email: swmike@swm.pp.se
[ On Sunday, June 3, 2001 at 11:39:21 (-0700), James Thomason wrote: ]
Subject: 95th Percentile = Lame
If I am not mistaken, the true "benefit" to 95% billing is that it allows the provider to charge for bits they never delivered.
From the customer's point of view it not only dereases the cost of the "local loop", but it also gives them some of the benefits of more advanced delivery systems such as frame relay or ATM. I.e. even if all you ever move on average is 128kbps of traffic you've still got the
You are mistaken! :-) The true benefit of Nth percentile billing is that it allows the provider to sell low-cost, very high-speed, ports and only charge the customer for the true costs (plus profit) that customer incurs. For example it allows the ISP to sell a 100baseTX connection to a customer who only really needs about 128kbps or less, but who happens to be in the next rack/cage/room/etc. from your network. potential of bursting a small amount of data through at full wire speed. Even more importantly if your ISP is also paying on the Nth percentile rate then you're pretty much stuck with either paying through the nose for available bandwidth (in which case you'd better either have very deep pockets, or really need what you buy), or also paying on the Nth percentile. (of course all your data travels at wire speed -- it's just the gaps between the packets that make it appear otherwise! ;-) It is really too bad that ATM didn't beat 100mbit ethernet to the desktop....
The average will skew on a burst of traffic (>5% of the average) and you pay for it as if you had averaged that level the entire time.
Yes, of course, and that's the whole point. The user's got this big fat pipe and if they make the mistake of actually using it in any "significant" way then they increase the provider's costs and thus have to pay "through the nose" for their mistake! ;-) If you don't want to over-pay then don't over-burst! ;-) Like I said, it's really too bad that ATM didn't get to the desktop first. If a customer really wants to be careful then they can install an edge device that does the appropriate kinds of QoS or rate limiting and manage their bursts themselves.
I cannot, off the top of my head, think of another telecommunications industry that relies on a system of averages for settlement. It speaks pretty clearly of how immature the Internet industry really is.
There's more than one (other?) telecom industry? :-) But there's really nothing immature about doing rate-based billing! The telcos have been doing data-comm in this same manner for for maybe decades. I've even heard tell of those who allow their customers to adjust the CIR (and automatically get billed for the new rate, of course). The only thing "immature" about it is the underlying link layers we have to deal with in most cases. If everything were FR or ATM then there'd be no problem, right? ;-)
Or maybe not. Perhaps the electrical suppliers here in California should bill in the 95th percentile, and cite the Internet as a rational example.
If I'm not mistaken the concept of rate-based billing originated with electricity suppliers (who, it seems from stories I've heard, couldn't figure out how to do it properly either!). -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
On Sun, 3 Jun 2001, Greg A. Woods wrote:
You are mistaken! :-)
The true benefit of Nth percentile billing is that it allows the provider to sell low-cost, very high-speed, ports and only charge the customer for the true costs (plus profit) that customer incurs.
For example it allows the ISP to sell a 100baseTX connection to a customer who only really needs about 128kbps or less, but who happens to be in the next rack/cage/room/etc. from your network.
From the customer's point of view it not only dereases the cost of the "local loop", but it also gives them some of the benefits of more advanced delivery systems such as frame relay or ATM. I.e. even if all you ever move on average is 128kbps of traffic you've still got the potential of bursting a small amount of data through at full wire speed.
Even more importantly if your ISP is also paying on the Nth percentile rate then you're pretty much stuck with either paying through the nose for available bandwidth (in which case you'd better either have very deep pockets, or really need what you buy), or also paying on the Nth percentile.
I still fail to see how "peak bits" or "bursted bits" are more expensive than "regular bits". A 100Mbit FE port costs whatever it costs, and does not fluctuate with usage. This is true of almost all of your links within the network - excluding those where you have negotiated usage-based billing. An OC3, point to point, costs as much as it costs irrelevant of its usage. Therefore, every bit that crosses this circuit has a cost. Why not simply pass this cost on to the customer bit for bit?
(of course all your data travels at wire speed -- it's just the gaps between the packets that make it appear otherwise! ;-) Yes, of course, and that's the whole point. The user's got this big fat pipe and if they make the mistake of actually using it in any "significant" way then they increase the provider's costs and thus have to pay "through the nose" for their mistake! ;-)
If you don't want to over-pay then don't over-burst! ;-)
Again, where is this substantiated? How does this drive up costs for providers (anyone?) ?
There's more than one (other?) telecom industry? :-)
But there's really nothing immature about doing rate-based billing!
The telcos have been doing data-comm in this same manner for for maybe decades. I've even heard tell of those who allow their customers to adjust the CIR (and automatically get billed for the new rate, of course).
The only thing "immature" about it is the underlying link layers we have to deal with in most cases. If everything were FR or ATM then there'd be no problem, right? ;-)
To reiterate - I did not mean to imply that existing telecommunications settlement models are mature, by any stretch of the imagination. :) They are however, a bit more mature than existing Internet settlment models. Again, its not as if when traffic levels rise, more monkeys are required to shovel coal into the engine. The equipment and circuits all have fixed delivery levels, and (roughly) fixed costs. What is immature is the non-approximate, non-quantified cost of bits on a per provider and per-customer basis, coupled with the ASSUMED equal-cost exchange of bits between carriers. Which I should add is completely ludicrous. It costs ATT a quite different amount to haul bits around than GBLX - why not settle those costs like other adults?
If I'm not mistaken the concept of rate-based billing originated with electricity suppliers (who, it seems from stories I've heard, couldn't figure out how to do it properly either!).
Did it!? No wonder they are f**ked up. :))) Regards, James
-- Greg A. Woods
+1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
I still fail to see how "peak bits" or "bursted bits" are more expensive than "regular bits". A 100Mbit FE port costs whatever it costs, and does not fluctuate with usage. This is true of almost all of your links within the network - excluding those where you have negotiated usage-based billing. An OC3, point to point, costs as much as it costs irrelevant of its usage. Therefore, every bit that crosses this circuit has a cost.
Why not simply pass this cost on to the customer bit for bit?
It is NOT that the each bit has the same cost - it is the cost of maintaining enough EXTRA bandwidth so that the downstreams do not bounce up against the ceiling. That amount is basically covered by using the 95 rule.
If I am not mistaken, the true "benefit" to 95% billing is that it allows the provider to charge for bits they never delivered. The average will skew on a burst of traffic (>5% of the average) and you pay for it as if you had averaged that level the entire time.
I'm not sure what you mean by "you pay for it as if you had averaged that level the entire time". Couldn't someone equally say that I can burst at a full DS3 24 hours a day 7 days a week and I pay for it as if I only sustained that bandwidth 5% of the time? The reality is that a customer who sustains a full DS3 24 hours a day 7 days a week costs about as much to service as a customer who sustains a full DS3 only a smaller portion of the time. Plus, when there is excess bandwidth available, it makes sense to let the customer have it.
It seems like quite an irrational settlement model. Why not simply bill for every bit that crosses your network? There certainly is a per-bit cost.
Because bits moved at peak time cost you more than bits moved off-peak. You have to design and build a network to tolerate your maximum sustained bandwidth, not your average bandwidth. Plus, you want to reward customers who can and do move bulk transfers to off-peak times. Which would you rather have, a customer who sustains 1Mbps 24 hours a day seven days a week or a customer who sustains 100Mbps every Monday from 2PM to 3PM? Do you think the cost per-bit is the same?
Or maybe not. Perhaps the electrical suppliers here in California should bill in the 95th percentile, and cite the Internet as a rational example.
It's a shame that the current electricity metering and billing system has no way to reward those people who shift some of their load off-peak. If it did, the on-peak rate could be raised while leaving the off-peak rate the same. This would help ease the crisis significantly while having much less impact on poorer people who can't afford to pay 40% more for their electricity. Imagine if electrical companies could bill based upon actual cost (minute to minute). Imagine if people could set their meter to turn off different circuits if the rate exceeded different amounts. Do you realize how much of the problem this would solve? Yes, Internet billing is still in its infancy. Billing based upon peak available bandwidth is obviously not right as it punishes people for leaving room for growth and needlessly slows down their transfers when bandwidth is available. Billing based just upon bits moved is obviously not right as it fails to reward load leveling and makes it too hard to leverage existing customers to get future ones. I can say from experience that 95th percentile billing seems to happen to produce the right number. DS
The reality is that a customer who sustains a full DS3 24 hours a day 7 days a week costs about as much to service as a customer who sustains a full DS3 only a smaller portion of the time. Plus, when there is excess bandwidth available, it makes sense to let the customer have it. Of course yes. Because if you uase line 5% of your time, ISP must maintain enougph bandwidth 100% of the time. And I dues not know examples when this 5% of the high load was out of peak hours.
So, they are saying _we maintain backbone to allow you work 95% of the time, and please pay for it in full_. UUnet provid[ed] (really) berst ISP service I ever saw from American (not European, they are much better in quality and responsibility) company, and it is just because of their smart billing schemas.
It seems like quite an irrational settlement model. Why not simply bill for every bit that crosses your network? There certainly is a per-bit cost.
Because bits moved at peak time cost you more than bits moved off-peak. You have to design and build a network to tolerate your maximum sustained bandwidth, not your average bandwidth. Plus, you want to reward customers who can and do move bulk transfers to off-peak times.
Which would you rather have, a customer who sustains 1Mbps 24 hours a day seven days a week or a customer who sustains 100Mbps every Monday from 2PM to 3PM? Do you think the cost per-bit is the same?
Or maybe not. Perhaps the electrical suppliers here in California should bill in the 95th percentile, and cite the Internet as a rational example.
It's a shame that the current electricity metering and billing system has no way to reward those people who shift some of their load off-peak. If it did, the on-peak rate could be raised while leaving the off-peak rate the same. This would help ease the crisis significantly while having much less impact on poorer people who can't afford to pay 40% more for their electricity.
Imagine if electrical companies could bill based upon actual cost (minute to minute). Imagine if people could set their meter to turn off different circuits if the rate exceeded different amounts. Do you realize how much of the problem this would solve?
Yes, Internet billing is still in its infancy. Billing based upon peak available bandwidth is obviously not right as it punishes people for leaving room for growth and needlessly slows down their transfers when bandwidth is available. Billing based just upon bits moved is obviously not right as it fails to reward load leveling and makes it too hard to leverage existing customers to get future ones.
I can say from experience that 95th percentile billing seems to happen to produce the right number.
DS
On Sun, 3 Jun 2001, David Schwartz wrote:
I'm not sure what you mean by "you pay for it as if you had averaged that level the entire time". Couldn't someone equally say that I can burst at a full DS3 24 hours a day 7 days a week and I pay for it as if I only sustained that bandwidth 5% of the time?
You could, in fact, make this statement. This is the problem - you pay for traffic you did not use 95% of the time, just like someone who DID use that amount 95% of the time. The argument people seem to be making is that the cost of provisioning for this bursty traffic is the justification for average-based billing. Yet 95th percentile billing assumes that in a worst-case scenario (for the carrier), 5 percent of bits passed are worthless.
The reality is that a customer who sustains a full DS3 24 hours a day 7 days a week costs about as much to service as a customer who sustains a full DS3 only a smaller portion of the time. Plus, when there is excess bandwidth available, it makes sense to let the customer have it.
Why is this the case? Further, where is the hard data that shows this scenario to be the case? Why does it make sense to let the customer have "excess bandwidth"? Does it not ALWAYS cost money to pass a bit across your network?
Because bits moved at peak time cost you more than bits moved off-peak. You have to design and build a network to tolerate your maximum sustained bandwidth, not your average bandwidth. Plus, you want to reward customers who can and do move bulk transfers to off-peak times.
What is the basis for this assumption? Facts for the service provider: 1. Hardware costs are fixed. 2. Leased line costs are fixed. 3. "Bandwidth" (Peering or Transit) may be variable. Why do "peak bits" cost more than "regular bits" ?
Which would you rather have, a customer who sustains 1Mbps 24 hours a day seven days a week or a customer who sustains 100Mbps every Monday from 2PM to 3PM? Do you think the cost per-bit is the same?
Why do I care? Should not the cost of providing network access (at peak usage) be my -basis- of cost that is passed on to the customer? I guess if I bill on 95th percentile I would rather have the bursty customer since he pays as if he used 100MBps the whole time.
It's a shame that the current electricity metering and billing system has no way to reward those people who shift some of their load off-peak. If it did, the on-peak rate could be raised while leaving the off-peak rate the same. This would help ease the crisis significantly while having much less impact on poorer people who can't afford to pay 40% more for their electricity.
I disagree with you here. However, I do not want to descend into a power discussion on-list. :) I do not think there is any real evidence to substantiate the assumption that "peak power" costs more than "off peak power". Instead, power companies would like you to move your electricity consumption to more convenient times well within acceptable profit margins.
to minute). Imagine if people could set their meter to turn off different circuits if the rate exceeded different amounts. Do you realize how much of the problem this would solve?
Now, this WOULD be cool.
Yes, Internet billing is still in its infancy. Billing based upon peak available bandwidth is obviously not right as it punishes people for leaving room for growth and needlessly slows down their transfers when bandwidth is available. Billing based just upon bits moved is obviously not right as it fails to reward load leveling and makes it too hard to leverage existing customers to get future ones.
I can say from experience that 95th percentile billing seems to happen to produce the right number.
DS
[ On Sunday, June 3, 2001 at 19:12:50 (-0700), James Thomason wrote: ]
Subject: RE: 95th Percentile = Lame
The argument people seem to be making is that the cost of provisioning for this bursty traffic is the justification for average-based billing. Yet 95th percentile billing assumes that in a worst-case scenario (for the carrier), 5 percent of bits passed are worthless.
That's not it at all. First off anyone speaking in general terms would never use an explicit number when talking about rate-based billing. That's something between the customer and the ISP and specific to many technical details as well as market forces. I try to always the phrase "Nth percentile", for example. Some studies have even shown that the 75th percentile is a closer approximation to CIR in some frame relay configurations (see, for instance, the paper referred to last time this thread "occured" -- I no longer have the URL, but it was "Kawaihiko and the Third-Quartile Day" by Nevil Brownlee and Russell Fulton). Secondly the basic assumption here is that the average ISP will have more than one customer and that the usage profiles of two customers will never be exactly the same from minute to minute.
What is the basis for this assumption?
Facts for the service provider:
1. Hardware costs are fixed. 2. Leased line costs are fixed. 3. "Bandwidth" (Peering or Transit) may be variable.
Why do "peak bits" cost more than "regular bits" ?
Why do you have to ask a question to which answer is so obvious?!?!?!? ;-) It should be obvious that peak bits cost more than non-peak bits because of basic supply and demand economics. More people want more bits at peak times and there are only so many to go around. Making more of them available is more costly (more expensive hardware, more expensive "last mile" line costs, and indeed maybe even more premium bandwidth and access charges). If all of those things together were not the case then there would be no such thing as a peak period in traffic volume!
Why do I care? Should not the cost of providing network access (at peak usage) be my -basis- of cost that is passed on to the customer?
Well, are you actually engineering your network to at least make a fair attempt at handling the peak traffic volumes, or not? If so then you're right and you probably don't care (though your customers certainly will because your prices may not be competitive because of this!).
I disagree with you here. However, I do not want to descend into a power discussion on-list. :) I do not think there is any real evidence to substantiate the assumption that "peak power" costs more than "off peak power". Instead, power companies would like you to move your electricity consumption to more convenient times well within acceptable profit margins.
I'll fall for it because I think I have a very good example that may provide a bit of an analogy. Perhaps you should take a look at the history of Trans-Alta Utilities, formerly owned by the City of Calgary, Alberta, and how it used its surplus power (mostly hyrdo generated, IIRC) on non-peak periods. Once upon a time Calgary was one of the most brightly lit night-time spots of any significant size on the surface of the globe (and may still be, though it seemed a lot darker at night there the last time I visited). Back in the early 1980's I seem to remember hearing things like "we have more street lights per capita than any other city in the world" and so on.... You could see the glow of the city from 60-90 miles away (and that's not just because it sits on the prairies or has a huge dome of pollution sitting over it! :-). [Recently there's been a plan in place to reduce the wattage of street lamps in Calgary because escalating energy costs have doubled and the city's electric bill for residential street lamps in Jan 2001 over the Jan 2000 bill!] In Calgary's case they had to build plants capable of delivering electricity at peak usage periods (with room to spare, of course, for both anomalous demand as well as to cover unscheduled down-time). Since their peak usage was almost always during daylight hours, they had many kilowatts of surplus power available all night long because there was ample "free" water behind the dams to spin their generators. Their costs were entirely in building capacity and in devivery -- the energy itself is/was more or less "free", at least at that large a scale. So, if peak-power doesn't "cost more" than off-peak power then how come the City of Calgary was able to burn so much non-peak power without taxing its residents as much as would have been necessary if they didn't own their own power plants and were instead forced to pay flat rate charges all night long? Certianly there are economies of scale, but they don't account entirely for the reasons why electricity suppliers would like consumers to "balance" the load more evenly across the day. It really does cost more to build more/bigger generators when the peak usage grows, and those costs must be passed on. If consumers could balance their demands to create a flat-line usage rate then existing capacity would stretch much much further thus amortising capital costs used to build that capacity over a much longer period of time. Whether or not these savings would be passed on to the consumer depends no doubt on whether your power company is government owned or not. Even when the energy source isn't "free" there are additional costs to having to engineer supply systems to feed whatever the input fuel is, and in some cases the increased demand for fuel will increase its price too! Economies of scale can only go so far. Is one super-sized nuclear plant "cheaper" than 10,000 evenly distributed slowpoke reactors? What about in the long term when the slowpoke can "burn" all the "spent" fuel from the old-fashioned reactors? What about from a safety perspective? Indeed many large electricity consumers in various parts of the world do in fact have to pay different rates at different times of day, and they're forced to do so because their suppliers face increased costs if their peak usage can't be kept under control. Almost anyone in North America at least can tell you there are periods of peak usage where sometimes hard to get a packet through edge-wise, and other times when you can spew extra packets all over the place and hardly notice any delays or conjestion. Clearly this would tend to indicate that the Internet (here at least) has been reasonbly well engineered to cover the peak usage and thus is drastically over- engineered from the point of view of anyone who doesn't need to use it during peak periods. -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
[ snip ]
It's a shame that the current electricity metering and billing system has no way to reward those people who shift some of their load off-peak. If it did, the on-peak rate could be raised while leaving the off-peak rate the same. This would help ease the crisis significantly while having much less impact on poorer people who can't afford to pay 40% more for their electricity.
[ snip ]
to minute). Imagine if people could set their meter to turn off different circuits if the rate exceeded different amounts. Do you realize how much of the problem this would solve?
in nassau/suffolk in new york (aka long island), the local power company is setting up a program whereby your central air conditioning system gets connected, via the internet (!!), and you get a small $$ bonus in return for allowing them to shut off your a/c unit during peak usage times. the box and hookup are free, as well as software to allow you to control your own unit over the 'net. i'm under the impression that uptake of this project is not overwhelming. -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York
Then we could have script kiddies telling us "We own your A/C"... I could just see the power they could wield. Once the system's hacked they could turn them off and on at a regular frequency (or other things) and whack the crap outta the electricity companies equipment. Imagine 100 000 A/Cs coherently going on and off every minute or so... scott On Mon, 4 Jun 2001, Henry Yen wrote:
[ snip ]
It's a shame that the current electricity metering and billing system has no way to reward those people who shift some of their load off-peak. If it did, the on-peak rate could be raised while leaving the off-peak rate the same. This would help ease the crisis significantly while having much less impact on poorer people who can't afford to pay 40% more for their electricity.
[ snip ]
to minute). Imagine if people could set their meter to turn off different circuits if the rate exceeded different amounts. Do you realize how much of the problem this would solve?
in nassau/suffolk in new york (aka long island), the local power company is setting up a program whereby your central air conditioning system gets connected, via the internet (!!), and you get a small $$ bonus in return for allowing them to shut off your a/c unit during peak usage times. the box and hookup are free, as well as software to allow you to control your own unit over the 'net.
i'm under the impression that uptake of this project is not overwhelming.
-- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York
Date: Mon, 4 Jun 2001 12:23:45 -0400 From: Henry Yen <henry@AegisInfoSys.com> To: nanog@merit.edu Subject: OT: electrical [was: 95th Percentile = Lame]
[ snip ]
in nassau/suffolk in new york (aka long island), the local power company is setting up a program whereby your central air conditioning system gets connected, via the internet (!!), and you get a small $$ bonus in return for allowing them to shut off your a/c unit during peak usage times. the box and hookup are free, as well as software to allow you to control your own unit over the 'net.
Once upon a time there was a similar program in Wichita, except they shut your AC down via wireless. Called the power company to see about moving the box when switching from siding to stone a few years ago, and they said that they quit doing the remote shutdown. Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: (316) 794-8922 --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
I worked for Relcom ISP, and we just introduced this 95% billinfg as a big advantage. The reason (and idea) of 95% rule is very simple. If you (customer) overload your link, you cause a lot of extra (unpayed) traffic inside your ISP; without this rule you will not want to upgrade and so waste bandwidth payed by the otheer customers. May be, 95% is not the best way to measure overload (packet loss is better) but at least this work, and so UUnet was smard introducing this rule. Are there alternatives to this? Of course, they are - provide worst (oversubscribed) performance, or bill by every byte (we did it, too), or get out of the business. Of course, other telecommunications does niot use such rule - they just charge you for the full bandwidth no matter do you use it or not. It's the difference. ----- Original Message ----- From: "James Thomason" <james@divide.org> To: <nanog@merit.edu> Sent: Sunday, June 03, 2001 11:39 AM Subject: 95th Percentile = Lame
If I am not mistaken, the true "benefit" to 95% billing is that it allows the provider to charge for bits they never delivered. The average will skew on a burst of traffic (>5% of the average) and you pay for it as if you had averaged that level the entire time.
It seems like quite an irrational settlement model. Why not simply bill for every bit that crosses your network? There certainly is a per-bit cost.
I cannot, off the top of my head, think of another telecommunications industry that relies on a system of averages for settlement. It speaks pretty clearly of how immature the Internet industry really is.
Or maybe not. Perhaps the electrical suppliers here in California should bill in the 95th percentile, and cite the Internet as a rational example.
Regards, James
On Sun, 3 Jun 2001, James Thomason wrote:
If I am not mistaken, the true "benefit" to 95% billing is that it allows the provider to charge for bits they never delivered. The average will skew on a burst of traffic (>5% of the average) and you pay for it as if you had averaged that level the entire time.
James: I think the others have made the point regarding the above comment. However, let me explain my view of this a bit differently. As pointed out, the providers have to build to provide a certain quality of service during peak hours. It's the behavior of their network at those peak hours that determines what they need to build/buy and therefore their costs, which then determines what they need to charge their customers. Personally, I think the 95% billing is a rational simplification of a more complex problem. If you accept that peak loading occurs for something around 3-4 hours/day (sounds about right) and that 95% billing ignores the top 1.2 hours of usage each day, it's kind of like drawing a line horizontally roughly midway up the usage curve during the few hours of peak usage. In a way, it's what others have refered to as "average peak" usage. If you don't assume that a customer's usage is a smooth curve, but rather has some random variability during the few hours around peak loading time, and you assume that you have enough customers with similar somewhat random variations that those variations average out over the bulk of all customers to be a somewhat smooth loading, then you can only come to one conclusion. You can rationally evaluate, provision and bill for, that "average peak" usage of your network and have some level of confidence that you're being fair not only to yourself, but also your customer. As I see it, the only way to be even more fair, would be to truely average a customer's peak usage using some kind of "rolling average" calculation, then take the peak of that rolling average. That would possibly be more fair to customers who have wierd peak usage curves which could skew the 95% value to the point where it is NOT a reasonable estimation of the customer's average peak usage. Chuck Scott
I think the others have made the point regarding the above comment. However, let me explain my view of this a bit differently. As pointed out, the providers have to build to provide a certain quality of service during peak hours. It's the behavior of their network at those peak hours that determines what they need to build/buy and therefore their costs, which then determines what they need to charge their customers.
Very true. Providers have built their network to ensure a certain level of quality (that they cannot quantify due to packet based delivery), and the cost of this infrastructure reflects what they must charge customers to effect delivery of service. What I fail to understand is why is this cost not simply passed on equally to ALL customers? Averages of any kind have the effect of removing the structure of the underlying data. Averaging either: 1. Destroys real bits. 2. Creates make-believe bits. Why not simply quantify the cost of a transimitted bit, either generally, or based on point of egress, and pass that cost on equally to all customers? Do bits have value? The responses I have seen so far indicate: Bits have value sometimes depending on ..... a. if they are included in the average. b. if they increase or decrease the theoretical cost to the provider. But do not all bits have value all of the time? Are there free bits? Perhaps it was initially very difficult for a provider to quantify this cost - perhaps it still is. I am sure that assuming equal costs among providers that exchange traffic "freely" only contributes to this problem.
Personally, I think the 95% billing is a rational simplification of a more complex problem. If you accept that peak loading occurs for something around 3-4 hours/day (sounds about right) and that 95% billing ignores the top 1.2 hours of usage each day, it's kind of like drawing a line horizontally roughly midway up the usage curve during the few hours of peak usage. In a way, it's what others have refered to as "average peak" usage. If you don't assume that a customer's usage is a smooth curve, but rather has some random variability during the few hours around peak loading time, and you assume that you have enough customers with similar somewhat random variations that those variations average out over the bulk of all customers to be a somewhat smooth loading, then you can only come to one conclusion. You can rationally evaluate, provision and bill for, that "average peak" usage of your network and have some level of confidence that you're being fair not only to yourself, but also your customer. As I see it, the only way to be even more fair, would be to truely average a customer's peak usage using some kind of "rolling average" calculation, then take the peak of that rolling average. That would possibly be more fair to customers who have wierd peak usage curves which could skew the 95% value to the point where it is NOT a reasonable estimation of the customer's average peak usage.
What seems to be truely fair to me, is to have each and every customer pay a fair and reasonable price for the delivery of the actual bits they transmit and receive. I think this applies to carriers too.
Chuck Scott
Why not simply quantify the cost of a transimitted bit, either generally, or based on point of egress, and pass that cost on equally to all customers?
Why doesn't UPS just figure out the average cost of sending a package and charge that for every package? The answer is obvious, it won't encourage customers to make best use of existing capabilities and will encourage those people whose packages cost above average to use UPS and those whose packages cost below average to use other carriers. Billing based upon this type of statistical sampling is awful.
Do bits have value?
Value and cost are not the same thing. Providers don't bill based upon value.
The responses I have seen so far indicate:
Bits have value sometimes depending on ..... a. if they are included in the average. b. if they increase or decrease the theoretical cost to the provider.
But do not all bits have value all of the time? Are there free bits?
Yes, there are free bits. My network traffic at off-peak time could triple and it wouldn't cost me an extra penny. Triple my traffic on-peak and either my performance would fall to unacceptable levels, my bandwidth costs would go up, I'd have to provision new circuits and upgrade hardware, or all of these things.
Perhaps it was initially very difficult for a provider to quantify this cost - perhaps it still is. I am sure that assuming equal costs among providers that exchange traffic "freely" only contributes to this problem.
Providers could try to use a very simple way to bill, like total bits, but accept that it won't correlate well with the actual cost to provide the service, or providers could do a very good job of figuring out exactly how much each bit costs them, but the billing method will be complex and will be based upon factors beyond your control such as whether your peak times happen to align with other customers peak times. Neither of these extremes seems to work too well. So compromises start to make sense.
What seems to be truely fair to me, is to have each and every customer pay a fair and reasonable price for the delivery of the actual bits they transmit and receive. I think this applies to carriers too.
In other words, those who move cheaper bits should pay to subsidize those who move more expensive bits? Airmail and ground should be the same price so those who pick the least expensive possible delivery they can live with subsidize those who want everything sent the fastest possible way? Rational, efficient use of existing resources should be encouraged. Those who smooth out their load or move bulk transfers to off-peak times should be rewarded. If you want to tap into the lucractive VPN market, it helps if you can discount traffic between your own customers -- that way if you get the New York branch of FOO, you have leverage to get the San Francisco branch. Ideally, the price should match as closely as possible the actual cost to provide the service. You can make exceptions to keep the billing scheme comprehensible. As a practical matter, it is simply amazing how well 95% billing matches the actual costs associated with providing a circuit. I have found no better method that doesn't start to become incomprehensible. We've been talking about having 'half-price' times where your traffic is halved before going into the 95 percentile algorithm. We'd have a server that would tell you which times were half-price and we'd guarantee at least two such hours a day. Customers who do automated bulk data transfers could code to query this server and find the best times to move their data. But this tends to fall under the 'too complicated' or 'too much work for too little effect' category. DS
On Sun, 3 Jun 2001, David Schwartz wrote:
Why doesn't UPS just figure out the average cost of sending a package and charge that for every package? The answer is obvious, it won't encourage customers to make best use of existing capabilities and will encourage those people whose packages cost above average to use UPS and those whose packages cost below average to use other carriers. Billing based upon this type of statistical sampling is awful.
Lol! Exactly! Billing on statistical sampling IS awful, isn't it!? Why not instead, figure out what your EXACT costs are for a particular transaction, and bill based on that? FYI: UPS does look at a series of statistical samples to determine its package shipping costs from/to particular sources/destinations.
Do bits have value?
Value and cost are not the same thing. Providers don't bill based upon value.
Not saying they are, but this was slightly out of context. Providers SHOULD bill on value AND cost. They can't today.
Yes, there are free bits. My network traffic at off-peak time could triple and it wouldn't cost me an extra penny. Triple my traffic on-peak and either my performance would fall to unacceptable levels, my bandwidth costs would go up, I'd have to provision new circuits and upgrade hardware, or all of these things.
I see exactly what you mean here, and I agree. But, in fact those off-peak bits DO cost you money, just no more than what you have already budgeted.
Providers could try to use a very simple way to bill, like total bits, but accept that it won't correlate well with the actual cost to provide the service, or providers could do a very good job of figuring out exactly how much each bit costs them, but the billing method will be complex and will be based upon factors beyond your control such as whether your peak times happen to align with other customers peak times.
I think that technical solutions could be implemented to reduce the impact of "provisioning for peaks". The cost may even be the same. I agree that you, nor the customer, would have a great degree of control over the market. You WOULD however, be able to set cost/quality threshholds in an ideal world.
In other words, those who move cheaper bits should pay to subsidize those who move more expensive bits? Airmail and ground should be the same price so those who pick the least expensive possible delivery they can live with subsidize those who want everything sent the fastest possible way?
Not at all. I think you are trying to speak of quality characteristics in what is a commodity environment. The bandwidth I receive in my cage at Exodus does not differ from that of the cage next to me. I think that cost should be "fair" for the level of service I receive.
Rational, efficient use of existing resources should be encouraged. Those who smooth out their load or move bulk transfers to off-peak times should be rewarded. If you want to tap into the lucractive VPN market, it helps if you can discount traffic between your own customers -- that way if you get the New York branch of FOO, you have leverage to get the San Francisco branch.
Ideally, the price should match as closely as possible the actual cost to provide the service. You can make exceptions to keep the billing scheme comprehensible. As a practical matter, it is simply amazing how well 95% billing matches the actual costs associated with providing a circuit. I have found no better method that doesn't start to become incomprehensible.
If you have any data in this area you can share with me I would be most appreciative (everyone!).
We've been talking about having 'half-price' times where your traffic is halved before going into the 95 percentile algorithm. We'd have a server that would tell you which times were half-price and we'd guarantee at least two such hours a day. Customers who do automated bulk data transfers could code to query this server and find the best times to move their data. But this tends to fall under the 'too complicated' or 'too much work for too little effect' category.
I do not actually think this is incredibly complicated. The long distance companies do it. But you know exactly what your per-minute rate is. Easy to quantify. :)
DS
Related to the problem of billing for traffic, I have a problem trying to account for certain traffic. We're using a Cisco 72xx with 12.1 and I have some customers for whom I've setup traffic shaping, based on access lists, at the border to control their aggrigate bandwidth to the rest of the network without limiting their intranet traffic. We need to be able to report the shaping activity to them so they know how it's affecting their traffic. Unfortunately, there doesn't seem to be a way to monitor traffic shaping stats via snmp. Noticing an update note about snmp and class based traffic shaping with access lists, I tried to set it up that way but for some reason can't use the "shape" command as in the example below ("shape ..." isn't an available command at that point). Router(config)# policy-map dts-interface-class-action Router(config-pmap)# class class1 Router(config-pmap-c)# shape average 16000000 Anyone have a clue on either issue or if it's possible to use snmp to monitor traffic that passes certain access lists? Sorry if this is getting a bit off for this list. Chuck
This may be obvious, but billing by volume (bytes transferred) places far greater availability requirements on the measurement system than rate-based charging schemes.
Not particularly. If you have the average (not median) of usage for the month, even losing a sample here or there, you'd be just as accurate as a 95th %tile which may have missed the same measurements.
If I am charging by the byte, I have to count every packet. If my measurement system breaks, I lose money until it is fixed.
See above, 'average'.
At 6/3/01 11:58 AM, Joe Abley wrote:
This may be obvious, but billing by volume (bytes transferred) places far greater availability requirements on the measurement system than rate-based charging schemes.
No its not obvious. The SNMP byte counters are odometers - as long as you get two clean samples per counter wrap you can accurately count bytes. The trick is to ensure that you get a minimum of two clean samples of the odometer reading per counter wrap - for high speed interfaces that typically implies reading the MIB2 64 bit interface counters, or triggering an SNMP poll at relatively tight time intervals. (My previous comments a month or so back about the inaccuracies inherant in 95% systems still apply - given a particular (extreme case) traffic load pattern it is possible for two measurement systems that are not phase locked, using precisely the same sampling technique and computation to deliver outcome values for the 95% point where one is up to twice the value of the other. )
Geoff,
(My previous comments a month or so back about the inaccuracies inherant in 95% systems still apply - given a particular (extreme case) traffic load pattern it is possible for two measurement systems that are not phase locked, using precisely the same sampling technique and computation to deliver outcome values for the 95% point where one is up to twice the value of the other. )
my heart I can't see how the 95% relates to twice the value. More general how would it look like for n, whereby 0 < n < 1? --Arnold
No its not obvious. The SNMP byte counters are odometers - as long as you get two clean samples per counter wrap you can accurately count bytes. The trick is to ensure that you get a minimum of two clean samples of the odometer reading per counter wrap - for high speed interfaces that typically implies reading the MIB2 64 bit interface counters, or triggering an SNMP poll at relatively tight time intervals.
FYI: 2^64 is 1.844 * 10^19 at 2.5 gb/s (OC48 line speed) (2,500,000,000 bits/sec), you transfer 312,500,000 bytes/sec, or 298 megabytes/sec. 2^64 / 312,500,000 = 6.189 * 10^16 seconds per rollover. or, 1.719 * 10^13 hours or, 1,962,741,057 years. My point: at least for the near future, 64 bit counters won't roll.
(My previous comments a month or so back about the inaccuracies inherant in 95% systems still apply - given a particular (extreme case) traffic load pattern it is possible for two measurement systems that are not phase locked, using precisely the same sampling technique and computation to deliver outcome values for the 95% point where one is up to twice the value of the other. )
Of course; but, if using 64 bit counters, they should be damn near close.
[ On Monday, June 4, 2001 at 00:21:31 (+1000), Geoff Huston wrote: ]
Subject: Re: 95th Percentile again (was RE: C&W Peering Problem?)
No its not obvious. The SNMP byte counters are odometers - as long as you get two clean samples per counter wrap you can accurately count bytes. The trick is to ensure that you get a minimum of two clean samples of the odometer reading per counter wrap - for high speed interfaces that typically implies reading the MIB2 64 bit interface counters, or triggering an SNMP poll at relatively tight time intervals.
The worst problem with using SNMP counters is not the wrap-around (properly implemented that happens "rarely" even on high-speed links since the `standard' does `mandate' use of 64-bit counters for truly high-speed links), but rather accidental resets caused by improper agent implementations, or reboots (or both). You have to detect not only counter roll-over, but also resets, and you can only do the latter if the agent's uptime value is also reset when the counters are reset. Otherwise you have to do what MRTG and recent versions of cricket do and simply ignore all roll-over and reset events (and thus take the loss on the counter deltas for those intervals). Which is why taking measurements of even MIB-2 64-bit counters very frequently (eg. even as often as every five minutes) is "wise" to do even if you're simply billing on bulk throughput per period. It's not very hard to scale a collection engine that can run in parallel (on parallel hardware if necessary) to do this, and indeed the data volume should not be an issue even at a one-minute collection interval! Another problem I've seen is with SNMP agents that can't scale to handle a full compliment of ports on their host routers/switches. This is an important consideration to keep in mind when choosing a hardware vendor. I think this is still an area that needs covering by an independent test lab too....
(My previous comments a month or so back about the inaccuracies inherant in 95% systems still apply - given a particular (extreme case) traffic load pattern it is possible for two measurement systems that are not phase locked, using precisely the same sampling technique and computation to deliver outcome values for the 95% point where one is up to twice the value of the other. )
Well, IIRC, your example was one of true extremes in the "coarse" variety, and one in which any ISP (or customer, if it's the other way around) who's paying attention will spot and nix immediately (because they're well aware of the wicked ways of the world and will clearly have anticipated them in their contracts). I.e. you can't play games with the system because you can't be a customer if you do! ;-) (unless maybe all your customers play the same game and you mandate that they play "in sync" with each other thus guaranteeing your own utilisation is flat.... :-) -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <woods@robohack.ca> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>
On Sat, Jun 02, 2001 at 09:58:49PM -0400, Joe Abley wrote:
On Sat, Jun 02, 2001 at 05:28:52PM -0400, Timothy Brown wrote:
As an interesting aside to this discussion, Digital Island bills for total traffic transmitted per month (in GB increments). Does anyone using them have any comments on this approach besides the obvious? Does anyone else do a similar deal?
This may be obvious, but billing by volume (bytes transferred) places far greater availability requirements on the measurement system than rate-based charging schemes.
If I am charging by the byte, I have to count every packet. If my measurement system breaks, I lose money until it is fixed.
If I am charging by the 95%tile of five-minute average throughput measurements obtained during a calendar month, I can make do with much more coarse-grained sampling. Measurement system breaks, I'm quite possibly going to bill the same amount as if it hadn't broken.
[summarising some off-list conversations] For those that see this logic as reversed because they are polling interface counters (and hence calculating 5-minute average rates involves more polling than recording simple bytes past), consider the "measurement system" above to consist not only of your SNMP poller, but also of the firmware on your router or switch that counts packets. Having router/switch vendors provide part of the measurement system (by implementing interface counters on their products) is something that some people are saying works well for them. It's also something that some people say causes them pain and is hard to do well: + devices reboot when receiving apparently legal SNMP GETs + interface counters become corrupt or reset non-deterministically + interface counters reset on reboot + interface OIDs change on OIR + SNMP GETs cause high CPU load which affects performance or stability or both + polling code that works with vendor A doesn't work with vendor B, C, E or F + apparently-resolved SNMP issues reappear as if by magic when firmware is upgraded Other people choose not to use interface counters, but count packets in other ways (Jim's ipf counters and NetFlow collectors, the NeTraMeT/RTFM meters we used to use in New Zealand). This is not an argument in favour of any particular method of measurement. My point is that, in both cases, the stability of the measurement system as a whole needs to be weighed against the availability requirements of the billing model. Of the traffic-based charging regimes I have heard of, it seems to me that simple pay-by-the-byte places one of the highest availability requirements on the measurement infrastructure, and hence is probably one of the hardest to do well. Joe
participants (18)
-
Alex Rubenstein -
Alexei Roudnev -
Arnold Nipper -
Charles Scott -
David Klindt -
David Schwartz -
E.B. Dreger -
Geoff Huston -
Henry Yen -
James Thomason -
Jim Mercer -
Joe Abley -
Mikael Abrahamsson -
Nipper, Arnold
-
Richard A. Steenbergen -
scott w -
Timothy Brown -
woods＠weird.com