On Jan 21, 2008, at 6:50 AM, Roderick Beck wrote:
I didn't find that very convincing. Overspending on infrastructure does not force prices up or raise long term costs.
Excess capacity leads to prices falling in telecom and every other market.
Basic economics.
Maybe we should go a little more basic still... In the absence of competition (and esp. in the presence of risk of empowering competitive entrants), supply has no general/necessary effect on prices at all. So excess capacity of a product that is completely monopolized (or priced by cartel fiat, ala OPEC or SC) is largely irrelevant. On the other hand, in the specific case of Internet access (retail or wholesale level), prices have a huge impact on demand -- that's called "price elasticity". Given the absence of any mechanism to reveal "natural" prices or "actual" demand, how does one know whether/how much "excess capacity" there is anyway?
I'm sure the Americans are experiencing excess capacity right now in housing ... :)
American construction / real estate investors and speculators are certainly experiencing excess capacity in the most extreme way, but that's because the fully loaded price of housing has gone up a great deal (and/or, the actual price point is finally being discovered/ revealed) -- at least for the credit-challenged market segment that was the primary engine of demand behind the recent boom. For them especially, lending terms have deteriorated tremendously. Now general interest rates are falling again, but credit requirements are tightening so much that, for many in that group, credit will not be available "at any price". If that seems counterintuitive, think Japan between c. 1993-2005: huge "excess capacity" in real estate, inflation-adjusted interest rates at zero or below, and yet (or perhaps "as a consequence") no one buys, because the banks refuse to lend... Come to think of it, our sector has been struggling with its own roughly similar terms-of-exchange crisis since about 2004-2005... arguably driven by very similar prior circumstances as well... worth investigating a bit further perhaps... TV
Sent wirelessly via BlackBerry from T-Mobile.
-----Original Message----- From: Tom Vest <tvest@eyeconomics.com>
Date: Mon, 21 Jan 2008 01:10:16 To:Geoff Huston <gih@apnic.net> Cc:Matthew Moyle-Croft <mmc@internode.com.au>, Randy Bush <randy@psg.com>, Andy Davidson <andy@nosignal.org>, Andrew Odlyzko <odlyzko@dtc.umn.edu>, nanog@merit.edu Subject: Re: Lessons from the AU model
On Jan 20, 2008, at 11:14 PM, Geoff Huston wrote:
Matthew Moyle-Croft wrote:
Southern Cross cost some US $1B to construct about a decade ago
RFS was Nov 2001. They full paid the debt from a US$1.3B cost of construction in Oct 2005. (see http://www.southerncrosscables.com/public/News/newsdetail.cfm? StoryID=14) So, they're making some VERY decent money out of the duopoly with AJC.
Yes, that exercise managed to weather the slump in prices a couple of years back when supply far exceeded demand, and then exploit their excellent technical position when demand picked up and translate that position into good revenue streams that appear to be well above initial construction and ongoing operational costs.
I don't believe AJC has had a similar story, but others may know more here.
Hence why Telstra's building their OWN cable to Hawaii. It's cheaper to build than buy!
My comment is that its generally more complicated than that, and from a sufficiently distanced view overspending on infrastructure forces up prices as much as underspending. The only real revenue stream to fund this infrastructure comes from this pool of 24M folk living at the bum end of the Pacific Ocean. Paying for a large number of underutilized cable projects does have a higher total recurrent cost than would be the case of there were efficient sharing of a smaller number of cable projects, and ultimately its consumers who fund this inefficiency in supply. So sometimes competition provides natural incentives for cost efficient investments, that ultimately benefit consumers, and sometimes competition gets it wrong and over-invests because the actors cannot resolve their individual requirements in ways that result in efficient sharing of common venture infrastructure investments, and in such cases the consumer ends up paying for the inefficiency in infrastructure investment. So sometimes it is cheaper to lease than construct, and sometimes its not.
For anyone who has the opportunity and inclination to truly monopolize an isolated market, the "best" price to sell critical inputs to a potential competitor is certainly (infinity) -- and if the input is fungible, then *everyone* is a potential competitor. Sure your market grows less/more slowly, and your near-term revenues are lower -- but you have an untroubled eternity of guaranteed revenue/profit to console you. Building new facilities, even terrifically expensive and completely redundant ones, will cost less than (infinity) -- so of course this makes perfect (if utterly crazy) sense.
(and yes, since oceans are not the only features that impose large multipliers on facilities construction costs, this is still a fairly common situation in the world)
However, it's not just "consumers" who are paying the price; it's also every other producer in the economy. So if the bottleneck exploiter is not generating as much or more jobs, innovations, private investments, taxes, and other returns to the general economy, compared to the amount that were foregone (never created) by every other productive endeavor in the market, then it's the economy as a whole, present and future, that's paying the price.
Here endth the Nanog lesson in economics from me ( :-) )
My only point in entering this thread was to make the observations that the lessons from the AU model may not be very generic - small isolated communities often have a unique set of constraints for investments in communications systems and that often results in different industry structures, different relationships between the actors and often results in different pricing structures in the consumer market. I'm not sure that I'd be confident in generalizing this particular history into anything more generic that would apply to other communities in other parts of the world.
(My anti-fatalist rant)
This all seems pretty straightforward -- except of course that you, we, and presumably the A-Cs and other fairly observant people can make similar observations, and perhaps even arrive at similar/shared conclusions. Given that, and the potentially extreme/enduring costs (i.e., for everyone but the bottleneck exploiter) borne by everyone who happens to get stuck in the wrong kind of market, shouldn't we look for other, more practical lesson(s) as well?
TV
On 21/01/2008, at 10:49 PM, Tom Vest wrote:
In the absence of competition (and esp. in the presence of risk of empowering competitive entrants), supply has no general/necessary effect on prices at all. So excess capacity of a product that is completely monopolized (or priced by cartel fiat, ala OPEC or SC) is largely irrelevant.
It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable) So infrastructure spending can (and does) affect the price. We get that every day in .au (Transmission on the monopoly route between Melbourne and Hobart costs 3 times more than transmission between Sydney and LA; and other potential cable operators have always known that the monopoly has an excess of supply hidden away somewhere which they can roll out at bargain basement prices if a competitor ever arrives in the market)
[ housing ] Come to think of it, our sector has been struggling with its own roughly similar terms-of-exchange crisis since about 2004-2005... arguably driven by very similar prior circumstances as well... worth investigating a bit further perhaps...
I think the dominant factor that the American internet sector has been grappling with goes back further than that. It has its origins in the dot-com boom, when lots of people who didn't have any real money rolled out enormous infrastructure buildouts. When they inevitably went broke their infrastructure was bought at cents in the dollar, enabling the current generation of Internet companies to behave as if the infrastructure they're using was a lot cheaper than it really is. So hardly anyone has been selling below cost, but almost everyone has been selling below replacement cost. So everyone can extract profits for years, making out like bandits as they grow in to the excess capacity that was installed between 1999 and 2001, and they won't have a day of reckoning until they run out of capacity and find that they haven't been earning enough from their networks to service the debt they're going to need to take out to perform the next round of infrastructure upgrades. Example: You cannot seriously expect me to believe that the price of transatlantic connectivity actually reflects the cost of laying cables across the Atlantic. It defies common sense that a Gig-E tail from NYC to London is priced within an order of magnitude of a Gig-E tail from NYC to Boston. Metered charging systems are, to me, evidence of a realization that the business model underlying much of the Internet's last five years is unsustainable. You guys might think they're a novel and unwelcome arrival at the moment, but give it a few years and we'll see what happens :-) - mark -- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
On Jan 21, 2008, at 6:10 PM, Mark Newton wrote:
On 21/01/2008, at 10:49 PM, Tom Vest wrote:
In the absence of competition (and esp. in the presence of risk of empowering competitive entrants), supply has no general/necessary effect on prices at all. So excess capacity of a product that is completely monopolized (or priced by cartel fiat, ala OPEC or SC) is largely irrelevant.
It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable)
So infrastructure spending can (and does) affect the price.
Hi Mark, So you're saying that if a cable owner/monopolist simply lit another fiber pair, that would cause them to reduce prices? This mistakes a contingent symptom or tactic or mechanism -- i.e., the intentionally misleading public explanation ("we're sold out") -- with the real cause/strategy/motives behind "artificially" high prices...
We get that every day in .au (Transmission on the monopoly route between Melbourne and Hobart costs 3 times more than transmission between Sydney and LA; and other potential cable operators have always known that the monopoly has an excess of supply hidden away somewhere which they can roll out at bargain basement prices if a competitor ever arrives in the market)
[ housing ] Come to think of it, our sector has been struggling with its own roughly similar terms-of-exchange crisis since about 2004-2005... arguably driven by very similar prior circumstances as well... worth investigating a bit further perhaps...
I think the dominant factor that the American internet sector has been grappling with goes back further than that. It has its origins in the dot-com boom, when lots of people who didn't have any real money rolled out enormous infrastructure buildouts. When they inevitably went broke their infrastructure was bought at cents in the dollar, enabling the current generation of Internet companies to behave as if the infrastructure they're using was a lot cheaper than it really is.
So hardly anyone has been selling below cost, but almost everyone has been selling below replacement cost.
This makes perfect sense for resources that are not subject to "multiplexing effects" -- esp. unanticipated (undiscounted) multiplexing effects. Put it this way: how would you define (much less calculate) "replacement cost" for an asset whose financing was predicated on a useful of capacity of (x), but which, with fractional additional investment relative to the original outlay, can be leveraged to deliver (x)^4-n capacity -- with n yet to be determined? Must every increment of the now vastly larger resource be priced as it would have been assuming the "original" max cap? How much must the "replacement cost" replace? The original (x) capacity? The as-yet indeterminate (x)^n capacity? The originally anticipated/full scarcity-based/monopoly-backed profits?
So everyone can extract profits for years, making out like bandits as they grow in to the excess capacity that was installed between 1999 and 2001, and they won't have a day of reckoning until they run out of capacity and find that they haven't been earning enough from their networks to service the debt they're going to need to take out to perform the next round of infrastructure upgrades.
Example: You cannot seriously expect me to believe that the price of transatlantic connectivity actually reflects the cost of laying cables across the Atlantic. It defies common sense that a Gig-E tail from NYC to London is priced within an order of magnitude of a Gig-E tail from NYC to Boston.
Once you get acquainted with the power of that ^n, you'll believe ;-) Unfortunately, your location gives you few opportunities to familiarize yourself.
Metered charging systems are, to me, evidence of a realization that the business model underlying much of the Internet's last five years is unsustainable. You guys might think they're a novel and unwelcome arrival at the moment, but give it a few years and we'll see what happens :-)
If fine-grained metered pricing comes to the rest of the world, it'll be because people roll over for it (you guys weren't given a choice). If/when that happens, I'll be lobbying my local gov to turn over the water infrastructure to me so I can replace it with household Evian vending machines; and I'd recommend you all get in on the ground floor in the air market ASAP. Better be quick though, because the revolution will be just around the corner... TV
On 22/01/2008, at 10:21 AM, Tom Vest wrote:
On Jan 21, 2008, at 6:10 PM, Mark Newton wrote:
It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable)
So infrastructure spending can (and does) affect the price.
Hi Mark,
So you're saying that if a cable owner/monopolist simply lit another fiber pair, that would cause them to reduce prices?
No, I'm saying that if another infrastructure provider launched itself onto the marketplace, that'd immediately inspire the incumbent who is presently using scarcity as a justification for high prices to light another fiber pair. It's a bit more complicated than that, but arguing that supply has no effect on price is overly simplistic when there are monopolistic barriers preventing new sources of supply.
Put it this way: how would you define (much less calculate) "replacement cost" for an asset whose financing was predicated on a useful of capacity of (x), but which, with fractional additional investment relative to the original outlay, can be leveraged to deliver (x)^4-n capacity -- with n yet to be determined? Must every increment of the now vastly larger resource be priced as it would have been assuming the "original" max cap? How much must the "replacement cost" replace? The original (x) capacity? The as-yet indeterminate (x)^n capacity? The originally anticipated/full scarcity-based/monopoly-backed profits?
Whichever one of those comes out to the largest number. Eventually the asset will reach its capacity -- we can't keep upgrading things forever. Submarine cable systems also have a useful working life, so even if they haven't reached capacity they'll run out of legs eventually anyway (the ones installed during the dot-com boom are approaching their half-life) When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates. So if they don't have a billion or so dollars stored away somewhere, they're selling below replacement value.
Once you get acquainted with the power of that ^n, you'll believe ;-) Unfortunately, your location gives you few opportunities to familiarize yourself.
Well, no, we have footprint in Australia, the USA and Japan at the moment. We'd have built out to Europe by now if not for the fact that global IP transit is being sold cheaper than transatlantic transmission, so what's the point of building a POP across the pond? Now, given that transatlantic transmission is already artificially cheap due to the acquisition of distressed assets in 2001, what does that tell you about IP transit pricing?
Metered charging systems are, to me, evidence of a realization that the business model underlying much of the Internet's last five years is unsustainable. You guys might think they're a novel and unwelcome arrival at the moment, but give it a few years and we'll see what happens :-)
If fine-grained metered pricing comes to the rest of the world, it'll be because people roll over for it (you guys weren't given a choice).
No, it won't be because people roll over for it, it'll be because carriers and service providers just get on with it and do it. See my first post to this thread to see the progression which outlines why the introduction of metering by _one_ serious player in any given economy virtually forces every other player to switch to metering as well. Do you think Australian ISPs haven't tried to offer US-style flat-rate services? Of course they have. And they get destroyed in the marketplace. Here's the thing that metering gives you: it stratifies the marketplace. It gives you two classes of customer. One class is customers who know they can live painlessly within the boundaries of whatever quotas you're offering. They don't complain, they just pay their flat monthly bill every month and get on with their lives. The other class is customers who do so much P2P that the imposition of quotas is a painful and unwelcome experience. They whinge and bitch loudly about how awful their ISP is, and migrate en-masse towards whichever ISPs are providing "unlimited" services. The only people who truly care about "unlimited" are the ones who know they can't live within any limits. That means "unlimited" ISPs almost exclusively attract the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers. And the metered ISPs cater for normal folks who aren't like that. That's the dynamic some of you are missing which makes quotas inevitable. If one moderately large player adopts it, the rest of you are going to have to adopt it too.
If/when that happens, I'll be lobbying my local gov to turn over the water infrastructure to me so I can replace it with household Evian vending machines;
Where I come from household water is already metered, so I'm not sure what you're talking about :-)
and I'd recommend you all get in on the ground floor in the air market ASAP. Better be quick though, because the revolution will be just around the corner...
A sensible provider will set quotas large enough that 98% of their customers will never hit any limits. The only people they'll piss off will be the 2% of customers they don't want in the first place. Hardly fertile ground for a revolution. - mark -- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
On Jan 21, 2008, at 9:53 PM, Mark Newton wrote:
On 22/01/2008, at 10:21 AM, Tom Vest wrote:
On Jan 21, 2008, at 6:10 PM, Mark Newton wrote:
It goes a bit deeper than that when the monopoly can compound the problem my artificially constraining capacity by underspending on infrastructure (e.g., only lighting one pair on a multi-pair cable)
So infrastructure spending can (and does) affect the price.
Hi Mark,
So you're saying that if a cable owner/monopolist simply lit another fiber pair, that would cause them to reduce prices?
No, I'm saying that if another infrastructure provider launched itself onto the marketplace, that'd immediately inspire the incumbent who is presently using scarcity as a justification for high prices to light another fiber pair.
Actually, what I think you said, this time, is that new capacity investment, when it takes the form of competition, can affect prices. Which goes without saying. But the same thing can happen without new capex, e.g., if a new entrant can purchase (otherwise unutilized) wholesale capacity and multiplex/resell, either at a lower price point or with other value adds, e.g., customer service, etc. It's the competition, not the new investment, that drives the price train... unless of course you outlaw resale or make it technically impossible.
It's a bit more complicated than that, but arguing that supply has no effect on price is overly simplistic when there are monopolistic barriers preventing new sources of supply.
Put it this way: how would you define (much less calculate) "replacement cost" for an asset whose financing was predicated on a useful of capacity of (x), but which, with fractional additional investment relative to the original outlay, can be leveraged to deliver (x)^4-n capacity -- with n yet to be determined? Must every increment of the now vastly larger resource be priced as it would have been assuming the "original" max cap? How much must the "replacement cost" replace? The original (x) capacity? The as-yet indeterminate (x)^n capacity? The originally anticipated/full scarcity-based/monopoly-backed profits?
Whichever one of those comes out to the largest number.
But the problem is, you never know which one will come out to the larger number until the proverbial "long run". In some countries, those holding the capacity are betting that (zero to very few sales) + (very high prices) + (freedom from internal and external competition) will deliver the big number. Others are betting that many small slices of a much bigger pie will deliver the big number. Who's to say, today, which one will be right in the end? I bet I can guess which one you'd rather live in however.
Eventually the asset will reach its capacity -- we can't keep upgrading things forever. Submarine cable systems also have a useful working life, so even if they haven't reached capacity they'll run out of legs eventually anyway (the ones installed during the dot-com boom are approaching their half-life)
When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates.
I believe that someone will be able to "build" (i.e., finance) more, when/where more is required. Whether that's the most recent inheritors of the old/distressed assets or someone completely new remains to be seen.
So if they don't have a billion or so dollars stored away somewhere, they're selling below replacement value.
With very few exceptions there's no "they"; the old "they" is gone, the new "they" didn't take over until fairly recently, didn't bankroll the original construction, and isn't bearing the financing costs of that construction.
Once you get acquainted with the power of that ^n, you'll believe ;-) Unfortunately, your location gives you few opportunities to familiarize yourself.
Well, no, we have footprint in Australia, the USA and Japan at the moment.
We'd have built out to Europe by now if not for the fact that global IP transit is being sold cheaper than transatlantic transmission, so what's the point of building a POP across the pond? Now, given that transatlantic transmission is already artificially cheap due to the acquisition of distressed assets in 2001, what does that tell you about IP transit pricing?
It's cheap alright; how much should it be? As for transit < L2 pricing? That tells me that someone has a traffic imbalance problem, and you are the beneficiary; revel in your good fortune :-)
Metered charging systems are, to me, evidence of a realization that the business model underlying much of the Internet's last five years is unsustainable. You guys might think they're a novel and unwelcome arrival at the moment, but give it a few years and we'll see what happens :-)
If fine-grained metered pricing comes to the rest of the world, it'll be because people roll over for it (you guys weren't given a choice).
No, it won't be because people roll over for it, it'll be because carriers and service providers just get on with it and do it.
If one does it alone, in the presence of competition, they will be punished into oblivion. If all do it at the same time, in the presence of rule of law, that will likely prompt an immediate anti-trust intervention.
See my first post to this thread to see the progression which outlines why the introduction of metering by _one_ serious player in any given economy virtually forces every other player to switch to metering as well.
Funny, that's also exactly how flat-rate pricing went from unprecedented to unavoidable in big open markets like the US, Japan, UK, etc. So the dynamic you're describing doesn't explain much -- except maybe that for some regional markets, outcomes may be completely contingent on what "one serious player" decides to do.
Do you think Australian ISPs haven't tried to offer US-style flat-rate services? Of course they have. And they get destroyed in the marketplace.
Look, the system is extremely well, um, arranged in your home market. Maybe now you have competitive metro p2p, competitive access, multiple peering points and abundant T2 peering. Regardless, you live in an English-speaking country with a cosmopolitan user base, and an absolutely inescapable international bottleneck. It's no surprise that, to date, flat-rate has punished AU ISPs. Sounds like change is afoot however; I wonder how closer to flat-rate AU will come after one or two new cables are completed... ?
Here's the thing that metering gives you: it stratifies the marketplace. It gives you two classes of customer.
One class is customers who know they can live painlessly within the boundaries of whatever quotas you're offering. They don't complain, they just pay their flat monthly bill every month and get on with their lives.
The other class is customers who do so much P2P that the imposition of quotas is a painful and unwelcome experience. They whinge and bitch loudly about how awful their ISP is, and migrate en-masse towards whichever ISPs are providing "unlimited" services. The only people who truly care about "unlimited" are the ones who know they can't live within any limits.
That means "unlimited" ISPs almost exclusively attract the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers. And the metered ISPs cater for normal folks who aren't like that.
That's the dynamic some of you are missing which makes quotas inevitable. If one moderately large player adopts it, the rest of you are going to have to adopt it too.
Here's another dynamic you missed. The coexistence of flat-rate and metered access creates two tiers of regional markets: the flat rate ones that generate lots of content and service innovations, attract lots of talent and FDI, and over time come to occupy an increasingly central position in the evolving global L2/L3 topology -- and the ones that don't, whose native innovators and content providers prefer to ship out or offshore production to the former. Metered access exacts a price at the national level.
If/when that happens, I'll be lobbying my local gov to turn over the water infrastructure to me so I can replace it with household Evian vending machines;
Where I come from household water is already metered, so I'm not sure what you're talking about :-)
Agreed, but I almost never hear my water vendor claiming that my rates need to be increased so they can cross-subsidize a new, separate national plumbing platform.
and I'd recommend you all get in on the ground floor in the air market ASAP. Better be quick though, because the revolution will be just around the corner...
A sensible provider will set quotas large enough that 98% of their customers will never hit any limits.
The only people they'll piss off will be the 2% of customers they don't want in the first place.
Hardly fertile ground for a revolution.
That almost sounds unobjectionable, I'll grant you -- at least in currently unmetered, high demand markets; everywhere else the bandwidth demand for any/every customer segment is nothing more than an artifact of whatever metered pricing plan is currently in place. So the reasonableness of the bandwidth cap level is itself contingent on the reasonableness of the metered plan. But even assuming you manage to define a "reasonable" cap, how will you defend it against competitors, and how will you determine when & how to adjust it (presumably upwards) as the basket of "typical" user content and services gets beefier -- or will that simply tip more and more people into some premium user category? Some of us would rather fight than switch (to an enterprise account) ;-) Cheers, TV
- mark
-- Mark Newton Email: newton@internode.com.au ( Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Tom Vest wrote:
So if they don't have a billion or so dollars stored away somewhere, they're selling below replacement value.
With very few exceptions there's no "they"; the old "they" is gone, the new "they" didn't take over until fairly recently, didn't bankroll the original construction, and isn't bearing the financing costs of that construction.
Round this neck of the woods we call em "banks" and they've generally been around for a few centuries and some of them, current financial shifts notwithstanding, expect to be around for a while yet! It's these folk that have the task of pricing risk and the greater the risk the more expensive the capital, of course. The situation Mark was alluding to was a persistent theme of the submarine cable conferences a couple of years back, where folk were telling each other that the retail IP market got itself hooked on access to wholesale submarine transit assets at distressed prices and the proposition was being made that this business model of flogging off bankrupt assets for cents in the dollar simply didn't allow for further construction to be cost effective for investors. If further investment were going to happen then the retail market needed to flow back dollars not cents to investors. But life goes on, more fibre strands have been lit, more DWDM systems have been reconditioned to support more lambdas and some more cable systems have been proposed and some have been built. Predictions of the demise of the industry were, of course, exaggerated. But the major flow on from the current generation of DWDM submarine cable systems, as expressed in the impact or pricing at the retail level, to me looks like its been fully realized and unless we see a new means of cramming a couple of orders of magnitude more bits into a long distance submarine fibre pair without spending a couple of orders of magnitude more dollars then we are back to the constant unit price proposition where more bits requires more money. Which is a roundabout way of saying that I'm very sceptical of Tom's exponential optimism in this particular area of infrastructure investment:-) regards, Geoff
gih@apnic.net (Geoff Huston) writes:
Which is a roundabout way of saying that I'm very sceptical of Tom's exponential optimism in this particular area of infrastructure investment :-)
the internet often hasn't made good economic sense. consider the chewage and swallowage of resources by modems, which eventually reached the limit where every iota of voicegrade on a circuit was in use, causing telco's to lose profit because they had an overcommit model and whereas they could compress 98% of a voice call they could not compress any of a modem call. flat rate connectivity (insensitive to distance or kilosegments or etc) is only practical under massive overprovisioning. we need moore's law to keep on going forever, and we need similar speeds of advance in optics and in power utilization. companies who can't massively overprovision will just continue to be shouldered aside by companies who can. new capital will keep on entering the market, and win or lose based on some disruptive approach, and then the winners will get shouldered aside a generation or two later. it's 2008, and no part of this should sound like news or insight to anybody. what i like about non-flat models like the one described as working for NZ and AU is that it will keep truly worthless flows off the network. finally there's a reason not to mindlessly share everything with everybody everywhere. (which is the only part of the equation that free market capitalism can't otherwise solve.) -- Paul Vixie
On 22/01/2008, at 3:59 PM, Tom Vest wrote:
When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates.
I believe that someone will be able to "build" (i.e., finance) more, when/where more is required.
Faith-based network rollouts. Neat :-)
Sounds like change is afoot however; I wonder how closer to flat- rate AU will come after one or two new cables are completed... ?
My guess? There'll continue to be industry-wide acceptance of the notion that a limit you never reach and don't care about is indistinguishable from no limit at all, and quotas will continue to steadily rise, stimulated by the lowering of the costs of supply, so that more and more people will never reach them and never care about them. _Most_ people couldn't care less about whether their monthly limit is 100 Gbytes or 150 Gbytes. But there are a small number of people who will be unhappy with any number smaller than 6 Tbytes, which is roughly the amount you can download on a 24 Mbit/sec ADSL2+ access tail if you run it flat-out 24x7 for a whole month. If new cable systems lower the cost of getting across the Pacific by a factor of 2, then 24 Mbit/sec of transpacific for that customer will _still_ cost us about $3000 per month, so you won't see us flocking to service those sorts of customers. There is simply no way that any provider in this marketplace is going to offer to service any customers on those terms. To get Australian providers to sell "unlimited" access you'd need the price of crossing the Pacific to drop by two orders of magnitude. Despite the best efforts of some people to run their broadband access at line rate, residential broadband is very much a "CIR + burst" kind of service. All of our customers can burst to line rate (they're paying for it, so they should be able to get it). None of our customers can burst at line rate 24x7 for a month without paying for it. You can work out the CIR by dividing the number of bits in the quota by the number of seconds in a month.
Here's another dynamic you missed. The coexistence of flat-rate and metered access creates two tiers of regional markets: the flat rate ones that generate lots of content and service innovations, attract lots of talent and FDI, and over time come to occupy an increasingly central position in the evolving global L2/L3 topology -- and the ones that don't, whose native innovators and content providers prefer to ship out or offshore production to the former. Metered access exacts a price at the national level.
I don't doubt that at all, and the USA has certainly done very well out of it. But how sustainable is it in the long term? You guys are about to have some serious economic challenges that are going to affect us too, but nowhere near as much as they'll affect you. I don't doubt that a difference in Internet access pricing policies has benefited your economy, that much is completely obvious. But if you're going to get all macro and holistic on me all of a sudden, are you prepared to assert that the benefit it has brought to your economy outweighs the other consequences that your ways of doing business have been creating? Because from where I'm sitting, it looks like the native innovators are in India at the moment, and the global L2/L3 topology that's been built over the last five years has been centering itself on Singapore, Hong Kong and Tokyo, and most of the high-growth parts of the US economy have been based on unsustainable debt rather than actual, existing money, and despite your claims of high growth my standard of living in Australia is at least as good as yours. So don't be too hasty to tell me all about the benefits of doing business in the American way. Not yet, anyway. :-)
Where I come from household water is already metered, so I'm not sure what you're talking about :-)
Agreed, but I almost never hear my water vendor claiming that my rates need to be increased so they can cross-subsidize a new, separate national plumbing platform.
That's exactly what my water utility is claiming. Australia is in the worst drought in recorded history, and all the water utilities are saying their rates are going to rise so that they can afford to build desalination plants and feed them with electricity. I'm tellin' ya, man, the parallels to this discussion are eerie :-)
The only people they'll piss off will be the 2% of customers they don't want in the first place.
Hardly fertile ground for a revolution.
That almost sounds unobjectionable, I'll grant you -- at least in currently unmetered, high demand markets; everywhere else the bandwidth demand for any/every customer segment is nothing more than an artifact of whatever metered pricing plan is currently in place. So the reasonableness of the bandwidth cap level is itself contingent on the reasonableness of the metered plan.
The metered plan will be reasonable, as dictated by all the competitive pressures you keep banging on about. With multiple vendors of metered Internet access, the vendor offering the best deal will get the customers. "Best" varies according to the eye of the beholder. Some prefer high reliability, zero effective contention, on-net content sources, titanium-plated CPE, clueful support, and hot and cold running network engineers. Some prefer it cheap-and-cheerful, and will prefer the provider that costs the least amount per month regardless of the quality. Some shop at Target, some shop on Rodeo drive. Some buy Ladas, some buy BMWs. You get what you pay for.
But even assuming you manage to define a "reasonable" cap, how will you defend it against competitors, and how will you determine when & how to adjust it (presumably upwards) as the basket of "typical" user content and services gets beefier -- or will that simply tip more and more people into some premium user category?
You make it sound like this stuff is hard and unworkable. Quotas and prices are adjusted according to competitive pressure. In 2000, Telstra was offering 3 Gbyte per month caps and we were offering 4.5 Gbyte per month. Now the industry norm is more like 40 - 60 Gbytes. I'm sure that in another 2 years it'll be 150 - 200 Gbytes. Competitive pressure is every bit as powerful in a metered marketplace as it is in a non-metered one, and to pretend that the alternative to the current US status quo somehow involves the end of business as we know it is just crazy talk. In Canada, which has much lower transit costs than ,au, the benchmark quota is presently about 100 Gbytes. How many broadband customers in the USA would actually have a problem with that as a limit? Here's a question for you: Power is metered. Water is metered. Gas is metered. Heating oil is metered. Even cable-TV is packaged so that you pay more if you want to use more channels... ... what economic fundamentals exist to suggest that Internet access should be the _only_ domestic utility that's delivered to households unmetered? - mark -- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
On Tue, 22 Jan 2008, Mark Newton wrote:
Power is metered. Water is metered. Gas is metered. Heating oil is metered. Even cable-TV is packaged so that you pay more if you want to use more channels...
I know of places in my nick of the world where all those are flat-rate. When the usage difference is small enough, metering is not effective. Typical dorm here includes power, water, (gas is usually not used, but most of the places that have gas charge ~ USD15 per month per apartment for gas, flatrate), and heating. Basic cable included in rent. I also know of quite a few regular apartments that have this model. In my apartment I pay for power. Water, heating and basic cable is included in the monthly fee. Some claim that metering is 50% of cost in the telco industry, and I have no reason to doubt that. Staying out of metering saves money on all levels, less complex equipment, less supportcalls, less hassle with billing. I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Tue, 22 Jan 2008, Mikael Abrahamsson wrote:
I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that.
DOS's against home customers arn't *that* common, certainly those that last long enough to hit bandwidth quota don't happen very often. In the past when you paid for going over your quota people did get $5000 bills for their home accounts. The terms and conditions made the customer responsible for it fullstop. It was their job to monitor their usage. The "throttle on cap" method tends to fix the problem. The customer does a huge amount of traffic unexpectantly so they just get slower Internet. Repeat until customer learns to not leave p2p programs running all night or let junior hang around the wrong channels on IRC. Usually they will get an email when they are at 80% of their limit or something which helps more. -- Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
Mikael Abrahamsson wrote:
Some claim that metering is 50% of cost in the telco industry, and I have no reason to doubt that. Staying out of metering saves money on all levels, less complex equipment, less supportcalls, less hassle with billing.
I have to agree with this, although the figure is trending downwards. Certainly one situation I have seen with ~$200mm in broadband metering revenue was spending $80m-ish a year on the various platforms that managed metering/rating/billing and the ops cost that went with it. The flipside is that I can see products (BRAS/BNG and their associated control plane solutions) being developed, launched, and marketed right now that make this much easier to manage. If the telephants adopt this, then it will reduce their billing cost substantially - although it adjusts (disrupts?) their traditional messy OSS/BSS.
I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that.
In my experience (NZ & AU) inbound DDoS attacks are usually waived by the service provider. This may not apply with all ISPs, but when I drove an ISP we did try and protect customers from that form of bill-shock. aj.
Hi, Regarding Dos filtering, I guess that really depends on whether we are talking about completing the attack and filtering in upstream transits, or, filtering source / traffic classification within the AS keeping the destination alive throughout the attack and utilising WAN/Transit bandwidth in the process. For me personally, if the first scenario is picked up under the horizon of the 95% bill then its a non issue. If the customer wants to ride it out themselves with no filtering from us, then it is going to appear on their port and hence burst bill for the month. And if the customer wants us to upstream source/classify filter traffic to give them a sanitized feed while we still incur the traffic - well they either pay or get turned off. Personally I don't want a Blue Solutions and if the customer were to think I am going to filter 24gbps of DoS delivering them a few megs of cleaned up traffic and that is all they pay for.... Well, they will be getting a P45. Dos filtering is expensive and value add and people need to pay fair price for things in this world, life is not free / cheap. A lack of imagination or focused usage from the IPTv / download brigade - does not mean something has lower value just that their business model is flawed and they cant afford the tools for the job + then keep all the advertising revenue to themselves. Why should access providers be giving away Ferraris for less than the manufacture cost of Maxi's (a really naff old UK car - http://en.wikipedia.org/wiki/Austin_Maxi) There is a UK ISP offering a LLU DSL service (line rental + internet + telephone service) for less than the monthly cost of the copper line rental from BT - to try and build market share to "pay" for that expensive DSLAM rollout by "buying" market share - I do at times wish the FDs / marketing / sales people at some of these companies would go and sit some basic business studies / accountant / economics classes. My 2c. Ben -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Alastair Johnson Sent: 22 January 2008 09:55 To: nanog@merit.edu Subject: Re: Lessons from the AU model Mikael Abrahamsson wrote:
Some claim that metering is 50% of cost in the telco industry, and I have no reason to doubt that. Staying out of metering saves money on all levels, less complex equipment, less supportcalls, less hassle with billing.
I have to agree with this, although the figure is trending downwards. Certainly one situation I have seen with ~$200mm in broadband metering revenue was spending $80m-ish a year on the various platforms that managed metering/rating/billing and the ops cost that went with it. The flipside is that I can see products (BRAS/BNG and their associated control plane solutions) being developed, launched, and marketed right now that make this much easier to manage. If the telephants adopt this, then it will reduce their billing cost substantially - although it adjusts (disrupts?) their traditional messy OSS/BSS.
I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that.
In my experience (NZ & AU) inbound DDoS attacks are usually waived by the service provider. This may not apply with all ISPs, but when I drove an ISP we did try and protect customers from that form of bill-shock. aj.
Or even Blue Security. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ben Butler Sent: 22 January 2008 10:26 To: nanog@merit.edu Subject: RE: Lessons from the AU model Hi, Regarding Dos filtering, I guess that really depends on whether we are talking about completing the attack and filtering in upstream transits, or, filtering source / traffic classification within the AS keeping the destination alive throughout the attack and utilising WAN/Transit bandwidth in the process. For me personally, if the first scenario is picked up under the horizon of the 95% bill then its a non issue. If the customer wants to ride it out themselves with no filtering from us, then it is going to appear on their port and hence burst bill for the month. And if the customer wants us to upstream source/classify filter traffic to give them a sanitized feed while we still incur the traffic - well they either pay or get turned off. Personally I don't want a Blue Solutions and if the customer were to think I am going to filter 24gbps of DoS delivering them a few megs of cleaned up traffic and that is all they pay for.... Well, they will be getting a P45. Dos filtering is expensive and value add and people need to pay fair price for things in this world, life is not free / cheap. A lack of imagination or focused usage from the IPTv / download brigade - does not mean something has lower value just that their business model is flawed and they cant afford the tools for the job + then keep all the advertising revenue to themselves. Why should access providers be giving away Ferraris for less than the manufacture cost of Maxi's (a really naff old UK car - http://en.wikipedia.org/wiki/Austin_Maxi) There is a UK ISP offering a LLU DSL service (line rental + internet + telephone service) for less than the monthly cost of the copper line rental from BT - to try and build market share to "pay" for that expensive DSLAM rollout by "buying" market share - I do at times wish the FDs / marketing / sales people at some of these companies would go and sit some basic business studies / accountant / economics classes. My 2c. Ben -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Alastair Johnson Sent: 22 January 2008 09:55 To: nanog@merit.edu Subject: Re: Lessons from the AU model Mikael Abrahamsson wrote:
Some claim that metering is 50% of cost in the telco industry, and I have no reason to doubt that. Staying out of metering saves money on all levels, less complex equipment, less supportcalls, less hassle with billing.
I have to agree with this, although the figure is trending downwards. Certainly one situation I have seen with ~$200mm in broadband metering revenue was spending $80m-ish a year on the various platforms that managed metering/rating/billing and the ops cost that went with it. The flipside is that I can see products (BRAS/BNG and their associated control plane solutions) being developed, launched, and marketed right now that make this much easier to manage. If the telephants adopt this, then it will reduce their billing cost substantially - although it adjusts (disrupts?) their traditional messy OSS/BSS.
I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that.
In my experience (NZ & AU) inbound DDoS attacks are usually waived by the service provider. This may not apply with all ISPs, but when I drove an ISP we did try and protect customers from that form of bill-shock. aj.
On Tue, 22 Jan 2008, Mikael Abrahamsson wrote:
I know of places in my nick of the world where all those are flat-rate. When the usage difference is small enough, metering is not effective.
Ahh, the key phrase is "usage difference is small enough."
Typical dorm here includes power, water, (gas is usually not used, but most of the places that have gas charge ~ USD15 per month per apartment for gas, flatrate), and heating. Basic cable included in rent. I also know of quite a few regular apartments that have this model. In my apartment I pay for power. Water, heating and basic cable is included in the monthly fee.
But what if the usage differences were large. If there was one tenant that left the hot water running 24 hours, 7 days a week; so other tenants complained they didn't get enough hot water. One tenant plugged in maximum wattage heaters on every circuit and left them on high 24 hours a day; left the television volume turned up to the maximum 24 hours a day; and so forth. Should the response from the landlord to install bigger hot water heaters, more electrical circuits, rebuild all the walls with increased sound proofing, and pass the cost of all those improvements to all the tenants in the building? Or, would the landlord consider asking one tenant with unusually high demands to leave or raise that just that tenant's rates. If you were the neighbor of such a tenant in a building, would you be pleased that your monthly fees were being increased or that one tenant was using all the hot water and generating a lot of noise all day and all night? Or might you complain to the landlord about those problems.
On Tue, 22 Jan 2008, Sean Donelan wrote:
If there was one tenant that left the hot water running 24 hours, 7 days a week; so other tenants complained they didn't get enough hot water. One
That has never happened to me. We have good enough infrastructure that one tenant filling up their hot water bath doesn't deplete the infrastructure of "hot water production" in my building. I seriously doubt anyone would notice me running hot water 24/7, because the infrastructure is able to handle that. No, everybody can't do it, but if I need to for a couple of hours, it works.
tenant plugged in maximum wattage heaters on every circuit and left them on high 24 hours a day; left the television volume turned up to the maximum 24 hours a day; and so forth.
I know people who run servers in their dorms due to this. It might go away, power is easy to meter.
If you were the neighbor of such a tenant in a building, would you be pleased that your monthly fees were being increased or that one tenant was using all the hot water and generating a lot of noise all day and all night? Or might you complain to the landlord about those problems.
Your analogy is halting, but that's to be expected. I certainly wouldn't want to pay more for the landlord to install metering everywhere. There is much overhead in metering and billing on that. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Tue, 22 Jan 2008, Mikael Abrahamsson wrote:
Your analogy is halting, but that's to be expected. I certainly wouldn't want to pay more for the landlord to install metering everywhere. There is much overhead in metering and billing on that.
I suspect you have never been a landlord or needed to respond to tenants.
On 22/01/2008, at 7:30 PM, Mikael Abrahamsson wrote:
I am also hesitant regarding billing when a person is being DDOS:ed. How is that handled in .AU? I can see billing being done on outgoing traffic from the customer because they can control that, but what about incoming, the customer has only partial control over that.
It isn't hard. For a start, there are only one or two providers where customers ever get excess bills, so we aren't really talking about "billing when a person is being DDOS'ed", we're talking about "rate- limiting after a person has been DDOS'ed." On the rare occasion when it happens, the ISPs who aren't bastards are usually understanding enough to back out measurements which are DDOS related. Whether or not your ISP is a bastard is one of the factors you'd use to determine which ISP is offering a package that fits your needs in a competitive environment. In my observation, customers who are repeatedly DDOS'ed are usually doing something to provoke it. I don't think I've ever seen an instance where a DDOS'ed customer has had a repeat occurrance after that datapoint is illuminated for them. - mark -- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Mark Newton wrote:
Despite the best efforts of some people to run their broadband access at line rate, residential broadband is very much a "CIR + burst" kind of service. All of our customers can burst to line rate (they're paying for it, so they should be able to get it). None of our customers can burst at line rate 24x7 for a month without paying for it. You can work out the CIR by dividing the number of bits in the quota by the number of seconds in a month.
Indeed. If you look at New Zealand, a very similar economic model to Australia (except less population and a much bigger density problem), there are regulated wholesale products[1,2,3] that offer a 32Kbps CIR per subscriber, and linerate PIR. 32Kbps working out to approximately 10GB per month, you can guess what the most common subscriber data cap is - and surprisingly few actually exceed it, although it has definitely gone up. Incidentally, the incumbent in NZ launched a flat rate DSL package. It did not go well, and ultimately cost them several million dollars in subscriber refunds. Perhaps some of the guys posting on this thread (Mark? MMC?) would be able to provide an average subscriber bandwidth (in GB or Kbit/s) use of their subscriber base. Break it down by <10G, >40G type accounts? aj [1] http://www.comcom.govt.nz/IndustryRegulation/Telecommunications/Wholesale/Ov... for what the regulator is doing [2] http://www.telecom.co.nz/content/0,8748,205743-204225,00.html?nv=tpd [3] http://www.telecom.co.nz/content/0,8748,204215-204225,00.html?nv=sd
We've figured our customer base ranges between 8 to 12 kbps/customer. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Alastair Johnson Sent: Tuesday, January 22, 2008 4:09 AM To: nanog@merit.edu Subject: Re: Lessons from the AU model Mark Newton wrote:
Despite the best efforts of some people to run their broadband access at line rate, residential broadband is very much a "CIR + burst" kind of service. All of our customers can burst to line rate (they're paying for it, so they should be able to get it). None of our customers can burst at line rate 24x7 for a month without paying for it. You can work out the CIR by dividing the number of bits in the quota by the number of seconds in a month.
Indeed. If you look at New Zealand, a very similar economic model to Australia (except less population and a much bigger density problem), there are regulated wholesale products[1,2,3] that offer a 32Kbps CIR per subscriber, and linerate PIR. 32Kbps working out to approximately 10GB per month, you can guess what the most common subscriber data cap is - and surprisingly few actually exceed it, although it has definitely gone up. Incidentally, the incumbent in NZ launched a flat rate DSL package. It did not go well, and ultimately cost them several million dollars in subscriber refunds. Perhaps some of the guys posting on this thread (Mark? MMC?) would be able to provide an average subscriber bandwidth (in GB or Kbit/s) use of their subscriber base. Break it down by <10G, >40G type accounts? aj [1] http://www.comcom.govt.nz/IndustryRegulation/Telecommunications/Wholesale/Ov erview.aspx for what the regulator is doing [2] http://www.telecom.co.nz/content/0,8748,205743-204225,00.html?nv=tpd [3] http://www.telecom.co.nz/content/0,8748,204215-204225,00.html?nv=sd
On Jan 22, 2008, at 3:01 AM, Mark Newton wrote:
On 22/01/2008, at 3:59 PM, Tom Vest wrote:
When the cable is full or EOL'ed its owner should have earned enough to build a new one at current market rates.
I believe that someone will be able to "build" (i.e., finance) more, when/where more is required.
Faith-based network rollouts. Neat :-)
Just faith in the "rationality" of a system with lots of independent feedback loops. If your faith has conflicting tenets, e.g., it compels you to believe that banks will demand, and you will be able to credibly promise, bubble-era returns on gross facilities investments, then I think we can chalk the rest up to religious differences and call it a day ;-)
Sounds like change is afoot however; I wonder how closer to flat- rate AU will come after one or two new cables are completed... ?
My guess? There'll continue to be industry-wide acceptance of the notion that a limit you never reach and don't care about is indistinguishable from no limit at all, and quotas will continue to steadily rise, stimulated by the lowering of the costs of supply, so that more and more people will never reach them and never care about them.
Occasional rhetorical indulgences notwithstanding, I'm a pragmatist; an ever-rising upper limit that 99% of the population never ever notices is not much of a limit. However I've rarely (actually, before now, *never*) heard the AU/NZ situation described thusly.... I must be spending too much time with the wrong 2% I guess. And I've yet to hear how one will be credibly define or sustainably (and legally) maintain such escalating limits.
_Most_ people couldn't care less about whether their monthly limit is 100 Gbytes or 150 Gbytes. But there are a small number of people who will be unhappy with any number smaller than 6 Tbytes, which is roughly the amount you can download on a 24 Mbit/sec ADSL2+ access tail if you run it flat-out 24x7 for a whole month.
If new cable systems lower the cost of getting across the Pacific by a factor of 2, then 24 Mbit/sec of transpacific for that customer will _still_ cost us about $3000 per month, so you won't see us flocking to service those sorts of customers.
You don't know what the "cost" is today; you only know the price. The implications left as an exercise for the reader...
There is simply no way that any provider in this marketplace is going to offer to service any customers on those terms. To get Australian providers to sell "unlimited" access you'd need the price of crossing the Pacific to drop by two orders of magnitude.
...
Despite the best efforts of some people to run their broadband access at line rate, residential broadband is very much a "CIR + burst" kind of service. All of our customers can burst to line rate (they're paying for it, so they should be able to get it). None of our customers can burst at line rate 24x7 for a month without paying for it. You can work out the CIR by dividing the number of bits in the quota by the number of seconds in a month.
Here's another dynamic you missed. The coexistence of flat-rate and metered access creates two tiers of regional markets: the flat rate ones that generate lots of content and service innovations, attract lots of talent and FDI, and over time come to occupy an increasingly central position in the evolving global L2/L3 topology -- and the ones that don't, whose native innovators and content providers prefer to ship out or offshore production to the former. Metered access exacts a price at the national level.
I don't doubt that at all, and the USA has certainly done very well out of it.
But how sustainable is it in the long term? You guys are about to have some serious economic challenges that are going to affect us too, but nowhere near as much as they'll affect you. I don't doubt that a difference in Internet access pricing policies has benefited your economy, that much is completely obvious.
ok.
But if you're going to get all macro and holistic on me
(actually that's my job)
all of a sudden, are you prepared to assert that the benefit it has brought to your economy outweighs the other consequences that your ways of doing business have been creating?
I wouldn't attempt to justify anything (and folks that know me will attest that they often hear something like the opposite), but even if I did, even for purely rhetorical amusement, I would start by pointing out that flat-rate access doesn't have any relationship to anything else you might be obliquely alluding to here. Being macro doesn't imply or necessitate being a dope ;-)
Because from where I'm sitting, it looks like the native innovators are in India at the moment,
Hard to argue with that. Since "constructive" innovation is a gross numbers game (e.g., education * addressable population), demographics ultimately drives a lot of dynamics -- once other bottlenecks are relieved.
and the global L2/L3 topology that's been built over the last five years has been centering itself on Singapore, Hong Kong and Tokyo,
No question, and that recent trend shift has contributed a lot toward reducing asymmetries in inter- and intra-regional capacity between Asia and ROTW.
and most of the high-growth parts of the US economy have been based on unsustainable debt rather than actual, existing money, and despite your claims of high growth my standard of living in Australia is at least as good as yours. So don't be too hasty to tell me all about the benefits of doing business in the American way. Not yet, anyway. :-)
Sigh. Am I going to have to secure Canadian or Swiss citizenship in order to be able to have a straight conversation about flat-rate access? For the purposes of any future discussions of this subject, let's just stipulate that I'm talking about Japanese flat-rate, or Korean flat rate... will that help to keep us on point?
Where I come from household water is already metered, so I'm not sure what you're talking about :-)
Agreed, but I almost never hear my water vendor claiming that my rates need to be increased so they can cross-subsidize a new, separate national plumbing platform.
That's exactly what my water utility is claiming. Australia is in the worst drought in recorded history, and all the water utilities are saying their rates are going to rise so that they can afford to build desalination plants and feed them with electricity.
I'm tellin' ya, man, the parallels to this discussion are eerie :-)
Is the actual current/sustainable supply of water delivery a trade secret, so that rational analysis is impossible and debates about it are interminable? Will the cost and capacity of the installed base and new desalinization plants be forever subject to NDA? Will the requested rate increases reflect an unknown markup or be transparently tied to the fully loaded cost of (expanded) service delivery amortized over some realistic time horizon? Will that full load have to include a large financing premium based not on risk but simply on someone's ambition to get a piece of (all of) the water action in perpetuity? There are many parallels I agree. Not so many on the critical points however.
The only people they'll piss off will be the 2% of customers they don't want in the first place.
Hardly fertile ground for a revolution.
That almost sounds unobjectionable, I'll grant you -- at least in currently unmetered, high demand markets; everywhere else the bandwidth demand for any/every customer segment is nothing more than an artifact of whatever metered pricing plan is currently in place. So the reasonableness of the bandwidth cap level is itself contingent on the reasonableness of the metered plan.
The metered plan will be reasonable, as dictated by all the competitive pressures you keep banging on about.
With multiple vendors of metered Internet access, the vendor offering the best deal will get the customers.
"Best" varies according to the eye of the beholder. Some prefer high reliability, zero effective contention, on-net content sources, titanium-plated CPE, clueful support, and hot and cold running network engineers. Some prefer it cheap-and-cheerful, and will prefer the provider that costs the least amount per month regardless of the quality.
Some shop at Target, some shop on Rodeo drive. Some buy Ladas, some buy BMWs. You get what you pay for.
But how you pay for it matters. If your BMW comes with a meter that "noticeably" increments every km you traverse, and your monthly car payment varies solely based on usage, then maybe a lot more people will "have" a BMW, but a lot fewer are going to be on the road at any given time. If your whole economy is built around that psychology, then you're likely to have a lot fewer people doing things that assume that drivers are an "addressable" market. More important still, if the rate of innovation, not only in cars but everything else, is somehow tied to the sum of all good and bad experiences on the road, then you're all-BMW economy is going to perpetually lag behind neighboring markets that provide Ladas on a largely unmetered basis. All I'm saying is that as bottlenecks are eliminated and the cost of supporting Aussie's globally diverse traffic exchange proclivities falls, it'll probably get harder and harder to justify "noticeable" caps or metered pricing models, or any kind of pricing model that seem to defy those trends. That's likely to happen at every level of the market, whether or not you're selling BMWs or Trabants. At least that's what triggered the shift in Japan, in the UK, and yes in the US too.
But even assuming you manage to define a "reasonable" cap, how will you defend it against competitors, and how will you determine when & how to adjust it (presumably upwards) as the basket of "typical" user content and services gets beefier -- or will that simply tip more and more people into some premium user category?
You make it sound like this stuff is hard and unworkable.
Quotas and prices are adjusted according to competitive pressure. In 2000, Telstra was offering 3 Gbyte per month caps and we were offering 4.5 Gbyte per month. Now the industry norm is more like 40 - 60 Gbytes. I'm sure that in another 2 years it'll be 150 - 200 Gbytes. Competitive pressure is every bit as powerful in a metered marketplace as it is in a non-metered one, and to pretend that the alternative to the current US status quo somehow involves the end of business as we know it is just crazy talk.
Okay I concede that point; competition within markets with only metered service options can be just as or even more vigorous then competition within unmetered markets. But competition between metered and unmetered markets tends to reward the latter, and competition within mixed markets tends to reward the latter. Moreover, to whatever degree that metering reduces usage, and usage is related to innovation, the unmetered markets are likely to grow and evolve faster. Maybe flat-rate access is "objectively" unsustainable in some markets, regardless of what (potentially self-interested) proponents claim. Maybe it is sustainable, even in the "long run", despite what (potentially self-interested) critics claim. One thing is certain: to date, only a handful of companies (Internet, wireless voice, or POTS) have *ever* stepped up to any flat-rate service voluntarily. The vast majority of flat-rate providers in business today were just as skeptical, and just as implacably opposed to the idea, right up to the point that competition forced them (and permitted us) to discover that they were wrong. At least that's what happened in Japan, in the UK, and yes in the US and many other places too.
In Canada, which has much lower transit costs than ,au, the benchmark quota is presently about 100 Gbytes. How many broadband customers in the USA would actually have a problem with that as a limit?
Here's a question for you:
Power is metered. Water is metered. Gas is metered. Heating oil is metered.
The pricing model for each is justified by the scarcity of the underlying finite natural resources. The prices themselves, and any increases, are sustained by in fact that the underlying cost components are usually transparent to interested third parties.
Even cable-TV is packaged so that you pay more if you want to use more channels...
Yes, and if I want to buy two books at the bookstore, I have to pay for each one separately. Premium cable channel prices are sustained by access to otherwise inaccessible content, nothing more.
... what economic fundamentals exist to suggest that Internet access should be the _only_ domestic utility that's delivered to households unmetered?
My driveway is not metered. The street in front of me is not metered. If someone can convincingly, empirically demonstrate how incremental usage of my driveway and the adjacent road -- e.g., how many passengers I'm carrying, how many bags of purchased goods, maybe the value of the goods in those bags, etc. -- imposes sufficient real incremental burden on the system to justify usage-based pricing, then I'll be glad to do my part -- or maybe I'll start looking for a neighborhood with a different accounting regime. Until then, I'm going to look to those other places, and remember past champions of driveway metering, and keep my hands in my pockets... Peace, TV
- mark
-- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
On Tue, 22 Jan 2008, Tom Vest wrote:
Okay I concede that point; competition within markets with only metered service options can be just as or even more vigorous then competition within unmetered markets. But competition between metered and unmetered markets tends to reward the latter, and competition within mixed markets tends to reward the latter. Moreover, to whatever degree that metering reduces usage, and usage is related to innovation, the unmetered markets are likely to grow and evolve faster.
Maybe flat-rate access is "objectively" unsustainable in some markets, regardless of what (potentially self-interested) proponents claim. Maybe it is sustainable, even in the "long run", despite what (potentially self-interested) critics claim. One thing is certain: to date, only a handful of companies (Internet, wireless voice, or POTS) have *ever* stepped up to any flat-rate service voluntarily. The vast majority of flat-rate providers in business today were just as skeptical, and just as implacably opposed to the idea, right up to the point that competition forced them (and permitted us) to discover that they were wrong. At least that's what happened in Japan, in the UK, and yes in the US and many other places too.
If you beleive it is inevitable, why do you care what other people do in the mean time. If you are correct, they will eventually loose and go away. Let them try it and see what happens. Why pre-judge the results?
$quoted_author = "Tom Vest" ;
Occasional rhetorical indulgences notwithstanding, I'm a pragmatist; an ever-rising upper limit that 99% of the population never ever notices is not much of a limit.
Sure it is. By knowing that no-one sharing the backhaul to the DSLAM at my CO can afford to do line rate 24x7 makes me sleep better at night. By using an ISP with sane limits I've never noticed performance degradation, even during peak periods.
However I've rarely (actually, before now, *never*) heard the AU/NZ situation described thusly.... I must be spending too much time with the wrong 2% I guess.
The wrong 2% of what? :-) If you want to see the flat-rate churning horde in all their glory visit whirlpool.net.au, otherwise affectioninatly known as "whingepool" because of the bitching'n'whining every time an ISP goes under or attends ECONOMICS 101 and brings in sustainable limits.
And I've yet to hear how one will be credibly define or sustainably (and legally) maintain such escalating limits.
Simon's post pretty much sums up the kind of maths that justifies them. http://www.merit.edu/mail.archives/nanog/msg05636.html Legal? What's legality have to do with this discussion? cheers marty -- "Multiple coffee suppliers feeding into a Redundant Arrangement of Independent Dispensers, to further reduce the chance of uncaffeinated downtime and increase Mean Time To Drowsiness." --Steve VanDevender alt.sysadmin.recovery - <ergtnb$8vb$1@isis.novusordo.net>
On Tue, 22 Jan 2008, Tom Vest wrote:
But even assuming you manage to define a "reasonable" cap, how will you defend it against competitors, and how will you determine when & how to adjust it (presumably upwards) as the basket of "typical" user content and services gets beefier -- or will that simply tip more and more people into some premium user category?
Seriously Tom it's not *that* hard and like we've been saying plenty of ISPs in many countries manage to do it. The different companies just play around until your profit, costs and income all balance nicely In NZ and Aus as the costs of bandwidth have decreased (and the demand has increased) then the bandwidth quotas have tended to go up. Lets say that a customer costs around $25 per month in last mile charges, staff, billing, marketing, profit for a 6Mb/s DSL. In the US right now you spend $5 on marginal bandwith usage which at $10 per MB/s gets you 150GB across the month. In Australia where the bandwidth cost is closer to $200 per MB/s per month that $5 only gets you around 8GB. Pricing flat rate will either put you out of business or cost so much that you not get 90% of customers. So in Australia you'll do a $30 cheap account with a 5GB/month quota, a $40 account with a 15GB Quota and a $60 account with a 40GB/month quota. This keep you bandwidth cost about the same and allows you to capture low end customers for $30 as well as heavier users at $60. Cheap entry level option and heavier users can pay more to get more. In the US 150GB is more than 90-something percent of users do and 6MB/s line speeds ( mostly) keeps the heavier users from going high enough to screw the average above that. So you keep a simple flat pricing scheme because that is easier to market and makes you more money. Just like helpdesk calls are usually free even though some users use up hundreds of dollars worth of them per month. On the other hand, image a few years down the track and you are at a US provider with $5 per Mb/s per month transit costs, most of your customers have 100Mb/s connections and : the bottom 30% of your customers average 0.25 TB / month = $4 / customer The next 40% of your customers average 1 TB / month = $15 / customer the top 30% of them average 3 TB / month = $45 / customer So you average bandwidth cost is around $20 per customer. Options are: 1. Increase prices to $45 per month and keep flat rate 2. Introduce tiered accounts 3. Traffic shap until bandwidth costs drop enough. Now right now option 3 seems to be common which sort of indicates that bandwidth usage by home customers *is* a problem. In many cases the choke point is at the last mile but it still doesn't really change the numbers. Providers have a budget to spend per customer on bandwidth, when they start to exceed that then something has to give. Of course with a bit of luck the cost of providing bandwith will keep falling as fast or faster than the average customer demand. Personally I doubt that long term home demand will exceed 30Mb/s or 10TB per month ( around 1 HDTV channel) on average. -- Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
On Tue, 22 Jan 2008, Mark Newton wrote:
That means "unlimited" ISPs almost exclusively attract the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers. And the metered ISPs cater for normal folks who aren't like that.
Ah, you've discovered our secret plan. If the US stops being the flat-rate P2P seeder to the world, what is the rest of the world going to do? Or is it going to topple countries one by one into metering? Are the flat-rate Swedish ISPs are going to get stuck footing the bill for the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers for the world :-)
Dear Mark, ----- Original Message ----- From: "Mark Newton" <newton@internode.com.au>
Do you think Australian ISPs haven't tried to offer US-style flat-rate services? Of course they have. And they get destroyed in the marketplace.
Here's the thing that metering gives you: it stratifies the marketplace. It gives you two classes of customer.
One class is customers who know they can live painlessly within the boundaries of whatever quotas you're offering. They don't complain, they just pay their flat monthly bill every month and get on with their lives.
The other class is customers who do so much P2P that the imposition of quotas is a painful and unwelcome experience. They whinge and bitch loudly about how awful their ISP is, and migrate en-masse towards whichever ISPs are providing "unlimited" services. The only people who truly care about "unlimited" are the ones who know they can't live within any limits.
That means "unlimited" ISPs almost exclusively attract the most voracious, least profitable, noisiest, most difficult to support, loudest complaining customers. And the metered ISPs cater for normal folks who aren't like that.
I can't agree with your definition of users. In community that has been also in a bit isolated (czech republic), there has been monopoly for cables/lines to Western europe till end of the 2K. Then after deliberalisation (and later sale of state owned ILEC) many new players entered the market and prices felt down. So it seems like Australia is somewhere in our trace. In meantime, some companies tried to start metered services. Some failed, some survived. But no one from surviving based his metering on payments on gigabytes. Most of them are working to soft limits and aggregation. Just because there is a problem that user cannot understand what is he consuming. You can meter water or minutes consumed on telephone. But if you are tuning up your favorite newschannel over Internet, do you know how many MB you are transferring each minute? Other problem is unsolicited traffic. Most of "plain users" have their PCs not well secured. And worms and botnets can create same traffic as P2P. Same situation for application that are making money over your business (i.e. Joost or Skype). You can argue that people cannot understand eletricity consumption and they are paying for usage, but - someone has to fire a coal or broke uranium atoms to develop elettricity. Are ISPs developping Mbytes or are they paying to Yahoo!, Google or other ICP? For me, raising this discussion from ISP community will lead to discussion on net neutrality from other (unwanted) side. May AT&T pay to Yahoo! or Google for data that they are selling to their customers? Regards Michal P.S: To not be completelly off-topic. For last two weeks I've been visiting New Zealand (nomadic access). And last few days I'm in Australia. Let me tell you that it is unblievable that these two countries are on the same submarine cable systems. NZ looks from nomadic perspective as US or NL, but I've been more lucky to get connectivity in Beijing that in Melbourne neighborhood :-(
participants (12)
-
Alastair Johnson -
Ben Butler -
Frank Bulk -
Geoff Huston -
Mark Newton -
Martin Barry -
Michal Krsek -
Mikael Abrahamsson -
Paul Vixie -
Sean Donelan -
Simon Lyall -
Tom Vest