Cost of transit and options in APAC
Hi Nanog, As we extend our reach into Asia, we're finding that our typical carriers (see: upstreams of AS36692) who provide service to us in North America and Europe are not able to offer us service in Asia either (1) at all or (2) at prices remotely resembling our pricing in NA and EU. For example: Level(3) simply has no presence in Asia and on the pricing side, NTT, GBLX, Verizon and others' pricing is many times higher than their NA and EU pricing. In most cases, it's 10 or more times higher. Additionally, some of the networks seem to market their network based on their reach into the US, rather than their reach into actual users in Asia, which is what we're looking for. So my question is, what are non-APAC-based networks doing as they expand into Asia for transit beyond peering with whomever will peer with them to get close to actual users in Asia? Are people using regional carriers? Are people just paying the "crazy" (compared to US pricing) bandwidth costs? Are people doing peering-only setups out there? Any help would be useful -- hopefully this is on-topic for NANOG, which I think it is, since I'm curious how NA operators deal with these challenges as they expand into APAC. I'm happy to summarize responses later if there is interest. Thanks, David
Nice to see this change.... APAC has been obliged to pay the cost to peer with the US (long distance links are expensive). Now that US wants to peer with Asia, pricing may become more balanced... ----- Original Message ----- From: "David Ulevitch" <david@ulevitch.com> To: nanog@merit.edu Sent: Thursday, 12 August, 2010 7:00:12 AM Subject: Cost of transit and options in APAC Hi Nanog, As we extend our reach into Asia, we're finding that our typical carriers (see: upstreams of AS36692) who provide service to us in North America and Europe are not able to offer us service in Asia either (1) at all or (2) at prices remotely resembling our pricing in NA and EU. For example: Level(3) simply has no presence in Asia and on the pricing side, NTT, GBLX, Verizon and others' pricing is many times higher than their NA and EU pricing. In most cases, it's 10 or more times higher. Additionally, some of the networks seem to market their network based on their reach into the US, rather than their reach into actual users in Asia, which is what we're looking for. So my question is, what are non-APAC-based networks doing as they expand into Asia for transit beyond peering with whomever will peer with them to get close to actual users in Asia? Are people using regional carriers? Are people just paying the "crazy" (compared to US pricing) bandwidth costs? Are people doing peering-only setups out there? Any help would be useful -- hopefully this is on-topic for NANOG, which I think it is, since I'm curious how NA operators deal with these challenges as they expand into APAC. I'm happy to summarize responses later if there is interest. Thanks, David
On 8/11/10 12:29 PM, Franck Martin wrote:
Nice to see this change....
APAC has been obliged to pay the cost to peer with the US (long distance links are expensive). Now that US wants to peer with Asia, pricing may become more balanced...
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific. The answer has more to do with the maturity of comms infrastructure, the cost of captial, and regulatory or monopoloy capture than it does with some artifical lack of price equilibrium.
----- Original Message ----- From: "David Ulevitch" <david@ulevitch.com> To: nanog@merit.edu Sent: Thursday, 12 August, 2010 7:00:12 AM Subject: Cost of transit and options in APAC
Hi Nanog,
As we extend our reach into Asia, we're finding that our typical carriers (see: upstreams of AS36692) who provide service to us in North America and Europe are not able to offer us service in Asia either (1) at all or (2) at prices remotely resembling our pricing in NA and EU. For example: Level(3) simply has no presence in Asia and on the pricing side, NTT, GBLX, Verizon and others' pricing is many times higher than their NA and EU pricing. In most cases, it's 10 or more times higher.
Additionally, some of the networks seem to market their network based on their reach into the US, rather than their reach into actual users in Asia, which is what we're looking for.
So my question is, what are non-APAC-based networks doing as they expand into Asia for transit beyond peering with whomever will peer with them to get close to actual users in Asia?
Are people using regional carriers? Are people just paying the "crazy" (compared to US pricing) bandwidth costs? Are people doing peering-only setups out there? Any help would be useful -- hopefully this is on-topic for NANOG, which I think it is, since I'm curious how NA operators deal with these challenges as they expand into APAC.
I'm happy to summarize responses later if there is interest.
Thanks, David
"...the cost of captial, and regulatory or monopoloy capture than it does with some artifical lack of price equilibrium." now that sounds like fodder for a different list ;) On Wed, Aug 11, 2010 at 12:53 PM, Joel Jaeggli <joelja@bogus.com> wrote:
On 8/11/10 12:29 PM, Franck Martin wrote:
Nice to see this change....
APAC has been obliged to pay the cost to peer with the US (long distance links are expensive). Now that US wants to peer with Asia, pricing may become more balanced...
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
The answer has more to do with the maturity of comms infrastructure, the cost of captial, and regulatory or monopoloy capture than it does with some artifical lack of price equilibrium.
----- Original Message ----- From: "David Ulevitch" <david@ulevitch.com> To: nanog@merit.edu Sent: Thursday, 12 August, 2010 7:00:12 AM Subject: Cost of transit and options in APAC
Hi Nanog,
As we extend our reach into Asia, we're finding that our typical carriers (see: upstreams of AS36692) who provide service to us in North America and Europe are not able to offer us service in Asia either (1) at all or (2) at prices remotely resembling our pricing in NA and EU. For example: Level(3) simply has no presence in Asia and on the pricing side, NTT, GBLX, Verizon and others' pricing is many times higher than their NA and EU pricing. In most cases, it's 10 or more times higher.
Additionally, some of the networks seem to market their network based on their reach into the US, rather than their reach into actual users in Asia, which is what we're looking for.
So my question is, what are non-APAC-based networks doing as they expand into Asia for transit beyond peering with whomever will peer with them to get close to actual users in Asia?
Are people using regional carriers? Are people just paying the "crazy" (compared to US pricing) bandwidth costs? Are people doing peering-only setups out there? Any help would be useful -- hopefully this is on-topic for NANOG, which I think it is, since I'm curious how NA operators deal with these challenges as they expand into APAC.
I'm happy to summarize responses later if there is interest.
Thanks, David
-- Respectfully, Chris Hart Developer / System Administrator Insuremonkey.com 2080 E. Flamingo, Suite 223 Las Vegas, NV 89119
On 11 Aug 10, at 2:53 PM, Joel Jaeggli wrote:
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
Obviously I can't speak for the providers in question, but I'd guess that the cost for transit in AP is strongly related to the cost of long-haul transport. Once upon a time, the majority of Internet traffic in AP countries *did* originate in the US. Does anybody have data that this is changing? Cheers, -Benson
On 11 Aug 2010, at 22:03, Benson Schliesser wrote:
On 11 Aug 10, at 2:53 PM, Joel Jaeggli wrote:
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
Obviously I can't speak for the providers in question, but I'd guess that the cost for transit in AP is strongly related to the cost of long-haul transport. Once upon a time, the majority of Internet traffic in AP countries *did* originate in the US. Does anybody have data that this is changing?
Cheers, -Benson
Well, it would be hard to change because who would host in country when it costs so much to do so. It'd be much cheaper to host out of the country thereby exasperating the whole problem. -- Leigh Porter
On Wed, 11 Aug 2010, Leigh Porter wrote:
Well, it would be hard to change because who would host in country when it costs so much to do so. It'd be much cheaper to host out of the country thereby exasperating the whole problem.
Well some of us have no choice. We provide hosted video conferencing solutions that require us to be closer to our subscribers. Some providers will lower their rates if you can show them most of your traffic will stay local.
<> Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com
My feeling is that the Chinese market suffice to its own needs, now that all the major websites have their equivalent in Chinese and are more popular than the Chinese translation of US/EU based web sites. I have heard of large data Centres being built in AP. Google spoke at one time to do its own trans-pacific link, because it could not find anything suitable. I guess the location of akamai caches could be telling... I may also suspect economical models of trans-pacific fiber may be different from economical model for trans-atlantic fiber, which would explain difference in costs. I have heard of things like that, but don't have firm data. So it is empirical data, but I think things are changing. Understanding the landscape and the reason behind the costs may help negotiate a better deal. Finally, have you considered peering with Australia to see if it gives you access to the AP market at better cost? ----- Original Message ----- From: "Benson Schliesser" <bensons@queuefull.net> To: "Joel Jaeggli" <joelja@bogus.com>, "Franck Martin" <franck@genius.com>, "nanog" <nanog@merit.edu> Sent: Thursday, 12 August, 2010 9:03:34 AM Subject: Re: Cost of transit and options in APAC On 11 Aug 10, at 2:53 PM, Joel Jaeggli wrote:
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
Obviously I can't speak for the providers in question, but I'd guess that the cost for transit in AP is strongly related to the cost of long-haul transport. Once upon a time, the majority of Internet traffic in AP countries *did* originate in the US. Does anybody have data that this is changing? Cheers, -Benson
On 8/11/10 2:03 PM, Benson Schliesser wrote:
On 11 Aug 10, at 2:53 PM, Joel Jaeggli wrote:
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
Obviously I can't speak for the providers in question, but I'd guess that the cost for transit in AP is strongly related to the cost of long-haul transport.
Start with something that can be effectively isolated from the transpacific path. Gotten a quote for a 1Gbe or 10Gbe between two cities in India recently? map that onto what you'd pay for a similar path in the US, count the extra zeros (once you account for the fact that INR is 46.91 to the dollar).
Once upon a time, the majority of Internet traffic in AP countries *did* originate in the US. Does anybody have data that this is changing?
Cheers, -Benson
On 11 Aug 10, at 5:15 PM, Joel Jaeggli wrote:
Obviously I can't speak for the providers in question, but I'd guess that the cost for transit in AP is strongly related to the cost of long-haul transport.
Start with something that can be effectively isolated from the transpacific path.
Gotten a quote for a 1Gbe or 10Gbe between two cities in India recently?
That could be useful. Sadly, I have no first-hand knowledge of these costs. How does in-country transport compare to trans-Pacific transport cost? (i.e. on a per Mbps per kilometer or similar metric) I assume it's cheaper in-country / in-region compared to trans-oceanic. Is this true? Once that's known, I'd also look at the opposite: excluding any last-mile transport costs, what is the price per Mbps for transit service? That transit price has to accommodate both the local/in-region transport as well as trans-oceanic transport costs borne by the provider. If the locale of traffic is shifting, then the provider's transport costs are also shifting from one category to another. My advice to anybody looking to buy AP regional transit, assuming that trans-oceanic bandwidth is more expensive than regional bandwidth, is to negotiate a lower price in exchange for only in-region routes. If my assumption about bandwidth costs is backwards, then you're out of luck. (Maybe we need lower taxes, higher bribes, or more competition..?) Cheers, -Benson
On Wed, Aug 11, 2010 at 05:41:16PM -0500, Benson Schliesser wrote:
On 11 Aug 10, at 5:15 PM, Joel Jaeggli wrote:
Obviously I can't speak for the providers in question, but I'd guess that the cost for transit in AP is strongly related to the cost of long-haul transport.
Start with something that can be effectively isolated from the transpacific path.
Gotten a quote for a 1Gbe or 10Gbe between two cities in India recently?
That could be useful. Sadly, I have no first-hand knowledge of these costs. How does in-country transport compare to trans-Pacific transport cost? (i.e. on a per Mbps per kilometer or similar metric) I assume it's cheaper in-country / in-region compared to trans-oceanic. Is this true?
This is not an assumption you can make safely depending on the country and specific sub-region you are talking about. If you go back to mid 90s, the situation was much the same in Europe, which was why US East coast was the default location for IP traffic exchange for Europe until the costs started to change. -dorian
On 12 Aug 10, at 7:26 AM, Dorian Kim wrote:
Sadly, I have no first-hand knowledge of these costs. How does in-country transport compare to trans-Pacific transport cost? (i.e. on a per Mbps per kilometer or similar metric) I assume it's cheaper in-country / in-region compared to trans-oceanic. Is this true?
This is not an assumption you can make safely depending on the country and specific sub-region you are talking about.
That's fair; I'm not surprised to hear it. But I'm curious about the details. In specific cases like India and China, what is the underlying contributor to higher relative transport costs? (i.e. taxes, ROW fees, extraordinary shipping or operational costs, inadequate competition, low supply, greed, etc) Further, how does the situation compare to past examples like Europe? I doubt the AP region is forever doomed to exchange traffic via the US, but can't quite anticipate change without first understanding the current environment. Network interconnects are designed the way they are, today, for a reason. If anybody has insight they can share on the situation I would appreciate it. Cheers, -Benson
On Thu, 12 Aug 2010, Benson Schliesser wrote:
Further, how does the situation compare to past examples like Europe?
Countries in Europe are all in different phases of competition and pricing. There is at least 10x difference in transit prices across Europe, with central and northern Europe being the cheapest, and southern and eastern being the most expensive. I agree totally with Mr Gilmore that it's all about competition. When there are 3+ (preferrably 4+) providers or something and the market is de-regulated then you get huge downward pressure on price and you soon hit levels of 5-20% operating margin and a mature market. I still remember back around 2000-2001 when we purchased 30 megs of transit from Ebone at around 120 USD per megabit/s, 2-3 years later the price was down to ~10-20 USD/megabit/s and then it's slowly decreased over time from that, basically as new faster technology came online so things could be done cheaper and at grander scale (you need the same amount of people to run a 155 megabit/s network as a 5*10G network, so price per bit goes down. APAC needs to go thru this as well, but things are generally still heavily regulated and the people are still adopting internet use instead of the situation in most of Europe and US where "everybody who wants Internet connection has one". It's when mass market deployment and deregulation happen together that sensible pricing occurs. Oh, competition needs to happen at all levels, from fiber in the ground to end user access. You can't have any single entity having a (de facto) monopoly/duopoly on any part of the chain. You need 4+ here as well (or a neutral party who just do one part and does it well, like municipal fiber rented at decent prices to anyone who wants to rent). -- Mikael Abrahamsson email: swmike@swm.pp.se
It always amaze me how the word de-regulated is so misused. When there is a monopoly the regulation is in fact very very light: Acme co is the monopoly and government cash in dividends/license fees and just check they don't do anything really silly. When there is competition this is when you have a heavy regulation, because all the players have now to play nicely with each others (access to each other network, anti competitive practices, keeping your number, roaming, inter-exchange, spectrum allocation,...). When competition comes the regulatory unit becomes bigger. In short competition increase regulation, it certainly does not decrease it. ----- Original Message ----- From: "Mikael Abrahamsson" <swmike@swm.pp.se> To: "Benson Schliesser" <bensons@queuefull.net> Cc: "nanog" <nanog@merit.edu> Sent: Friday, 13 August, 2010 5:20:04 PM Subject: Re: Cost of transit and options in APAC On Thu, 12 Aug 2010, Benson Schliesser wrote:
Further, how does the situation compare to past examples like Europe?
Countries in Europe are all in different phases of competition and pricing. There is at least 10x difference in transit prices across Europe, with central and northern Europe being the cheapest, and southern and eastern being the most expensive. I agree totally with Mr Gilmore that it's all about competition. When there are 3+ (preferrably 4+) providers or something and the market is de-regulated then you get huge downward pressure on price and you soon hit levels of 5-20% operating margin and a mature market.
Around 40% of our low-end/budget VPS hosting customers are based in APAC. It's common for departing customers to cite the primary reason as seeking lower latency to their regions. Sent while on the go, please excuse terseness. On Aug 11, 2010, at 17:03, Benson Schliesser <bensons@queuefull.net> wrote:
On 11 Aug 10, at 2:53 PM, Joel Jaeggli wrote:
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
Obviously I can't speak for the providers in question, but I'd guess that the cost for transit in AP is strongly related to the cost of long-haul transport. Once upon a time, the majority of Internet traffic in AP countries *did* originate in the US. Does anybody have data that this is changing?
Cheers, -Benson
On Wed, Aug 11, 2010 at 12:53:18PM -0700, Joel Jaeggli wrote:
On 8/11/10 12:29 PM, Franck Martin wrote:
Nice to see this change....
APAC has been obliged to pay the cost to peer with the US (long distance links are expensive). Now that US wants to peer with Asia, pricing may become more balanced...
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
Because the percentage of traffic that actually stays in India, as compared to that which transits the Pacific, is miniscule. If you're asking for enough bandwidth / throwing enough money around, I'm sure you could get an Indian-only deal, but you'd need to make it worth the while for the provider to setup the config, given that either way they'll be getting your money, and you won't be using a lot of transpacific traffic. Note also that it's unlikely that the provider will be getting a differentiated rate from their upstreams for internal traffic, and you may have to settle for peering-only access (if your chosen provider is connected to any peering points). - Matt -- Ruby's the only language I've ever used that feels like it was designed by a programmer, and not by a hardware engineer (Java, C, C++), an academic theorist (Lisp, Haskell, OCaml), or an editor of PC World (Python). -- William Morgan
On Aug 11, 2010, at 10:01 PM, Matthew Palmer wrote:
On Wed, Aug 11, 2010 at 12:53:18PM -0700, Joel Jaeggli wrote:
On 8/11/10 12:29 PM, Franck Martin wrote:
Nice to see this change....
APAC has been obliged to pay the cost to peer with the US (long distance links are expensive). Now that US wants to peer with Asia, pricing may become more balanced...
I think the question is more like why am I being quoted $100 A megabit in India for transit in India? Not why am I being charged for for the transport cost across the pacific.
Because the percentage of traffic that actually stays in India, as compared to that which transits the Pacific, is miniscule. If you're asking for enough bandwidth / throwing enough money around, I'm sure you could get an Indian-only deal, but you'd need to make it worth the while for the provider to setup the config, given that either way they'll be getting your money, and you won't be using a lot of transpacific traffic. Note also that it's unlikely that the provider will be getting a differentiated rate from their upstreams for internal traffic, and you may have to settle for peering-only access (if your chosen provider is connected to any peering points).
You do not need to throw a lot of money around. Lots of places in Asia give you separate in-country and "international" rates, and charge you accordingly. People have been talking about distance-sensitive pricing for IP traffic for years, without realizing people have been doing it in Asia for years. Back to topic of why prices are high in some places (and it is not just Asia), it is trivial to prove objectively that monopoly power keeps prices ridiculously high. Before anyone jumps on me, there are many reasons for high prices. Monopoly power is only one, but clearly and obviously the biggest one. When I say "objectively", I mean it. Look at any country which has gone through any type of transition from "gov't owned monopoly telco" to "competition-based market". South Africa instantly springs to mind. Prices are still high, but have dropped, what, 75% in just a year or two once the monopoly power was broken? And this is after a decade or more of little to no decrease. Of course, this does not mean .za will have $1/Mbps transit like the US any time soon. As I said, there are other factors - geography, scale, local economy, even import policies, etc. But getting prices to go from US$2000/Mbps to, say, $100/Mbps is more important than the $100 -> $1 drop. (Hrmm, I wonder who will say "the first is only 20 times, the second is 100 times!" to prove me wrong? :) Plus there are a myriad of factors keeping that last step from happening, not just one. So wich do you think is more important, the monopoly power or the dozens of other factors? That said, this is not really on-topic for NANOG. So if you would like to argue the point, please e-mail privately, or let's take it to another list. End of day, the important thing is to break the monopoly. After that, prices will almost always drop, then you can work on other stuff. -- TTFN, patrick
+10 Once you pass a threshold of affordability (by breaking the monopoly), then the network use explodes and other issues can be worked out by more or less by consumer pressure (and economies of scale)... You need to reach "Packet Storm" level. ----- Original Message ----- From: "Patrick W. Gilmore" <patrick@ianai.net> To: "NANOG list" <nanog@nanog.org> Sent: Friday, 13 August, 2010 6:29:02 AM Subject: Re: Cost of transit and options in APAC Back to topic of why prices are high in some places (and it is not just Asia), it is trivial to prove objectively that monopoly power keeps prices ridiculously high. Before anyone jumps on me, there are many reasons for high prices. Monopoly power is only one, but clearly and obviously the biggest one. When I say "objectively", I mean it. Look at any country which has gone through any type of transition from "gov't owned monopoly telco" to "competition-based market". South Africa instantly springs to mind. Prices are still high, but have dropped, what, 75% in just a year or two once the monopoly power was broken? And this is after a decade or more of little to no decrease. Of course, this does not mean .za will have $1/Mbps transit like the US any time soon. As I said, there are other factors - geography, scale, local economy, even import policies, etc. But getting prices to go from US$2000/Mbps to, say, $100/Mbps is more important than the $100 -> $1 drop. (Hrmm, I wonder who will say "the first is only 20 times, the second is 100 times!" to prove me wrong? :) Plus there are a myriad of factors keeping that last step from happening, not just one. So wich do you think is more important, the monopoly power or the dozens of other factors? That said, this is not really on-topic for NANOG. So if you would like to argue the point, please e-mail privately, or let's take it to another list. End of day, the important thing is to break the monopoly. After that, prices will almost always drop, then you can work on other stuff. -- TTFN, patrick
participants (12)
-
Benson Schliesser -
Christopher Hart -
David Ulevitch -
Dorian Kim -
Franck Martin -
Gregg Berkholtz -
Joel Jaeggli -
Leigh Porter -
Matthew Palmer -
Mikael Abrahamsson -
Nathan Stratton -
Patrick W. Gilmore