CDNs should pay eyeball networks, too.
Yesterday I received the following mail, from a CDN: ---->8---- Greetings, Limelight Networks periodically reviews its interconnection strategy to ensure the quality and integrity of its interconnection between all its partners. We have recently updated our requirements for settlement-free peering which are posted at http://www.as22822.net/ This letter is to notify you that yyy no longer meets our minimum requirements. If yyy would like to maintain our current interconnectivity, there will be settlement associated with doing so. If you are interested in pursuing this option, please reply back to this email indicating such. Should your company decline this option, or if we do not have an agreement regarding the settlement in place prior to May 31st 2012, Limelight Networks will terminate the peering sessions on that day, with this letter serving as 30 day notice. Sincerely, ----8<---- The same mail was sent out last year, about end of April 2011, to another ISP I'm working with. They got depeered, but the ISP which received the mail above yesterday didn't meet the requirements last year either. I totally understand that some companies might not be able to handle sub-5Mbps peering sessions, be it technical or organisational, but >=100Mbps should be worth any effort, as long as it improves the network. In this particular case I'm talking about >=600Mbps of traffic send out by Limelight to "my" eyeballs, not mentioning their fairly small footprint in Germany in comparison to other CDNs. These points aside, we are talking about a Content *Delivery* Network here. There are CDNs out there who burn to improve their customer experience (both the content creators and the content receiver) at high cost. Having a Tier1 attitude and telling eyeball networks with <1Gbps of traffic exchanged to bugger off or pay is not one of the ways to improve this. At the end of the day I'm going to charge CDNs who want to deliver their customers content to my eyeballs and make me pay (about 2USD per Mbps, with a minimum of 1Gbps). -dominik
On May 1, 2012, at 9:39 AM, Dominik Bay wrote:
Yesterday I received the following mail, from a CDN:
**snip**
Should your company decline this option, or if we do not have an agreement regarding the settlement in place prior to May 31st 2012, Limelight Networks will terminate the peering sessions on that day, with this letter serving as 30 day notice.
Sincerely,
While I can understand having some peering requirements, the goal of any CDN should be to have the best reach possible. Without knowing if this is PI peering or just across a IX it is hard to judge what their (or your) costs are. If it is IX it would seem irrational to cut off someone who you are doing meaningful traffic with.
----8<---- In this particular case I'm talking about >=600Mbps of traffic send out by Limelight to "my" eyeballs, not mentioning their fairly small footprint in Germany in comparison to other CDNs.
600Mbps would appear to be meaningful traffic. Without the other side of the story it's hard to get a full grip on the situation. It is always possible that these are business (not network) decisions where there is a certain level of income expected from a division.
These points aside, we are talking about a Content *Delivery* Network here. There are CDNs out there who burn to improve their customer experience (both the content creators and the content receiver) at high cost. Having a Tier1 attitude and telling eyeball networks with <1Gbps of traffic exchanged to bugger off or pay is not one of the ways to improve this.
I do agree here, again on an IX level. They still have the choice to backhaul to you or hot potato your traffic wherever they feel like it. From the data you have provide it appears to be a lose-lose situation to de-peer you.
At the end of the day I'm going to charge CDNs who want to deliver their customers content to my eyeballs and make me pay (about 2USD per Mbps, with a minimum of 1Gbps).
That may be what they are doing "if at all possible try to monetize it".
On 05/01/2012 07:07 PM, Steven Noble wrote:
While I can understand having some peering requirements, the goal of any CDN should be to have the best reach possible. Without knowing if this is PI peering or just across a IX it is hard to judge what their (or your) costs are. If it is IX it would seem irrational to cut off someone who you are doing meaningful traffic with.
This is over an IXP, I should've pointed that out.
----8<---- In this particular case I'm talking about >=600Mbps of traffic send out by Limelight to "my" eyeballs, not mentioning their fairly small footprint in Germany in comparison to other CDNs.
600Mbps would appear to be meaningful traffic. Without the other side of the story it's hard to get a full grip on the situation. It is always possible that these are business (not network) decisions where there is a certain level of income expected from a division.
These points aside, we are talking about a Content *Delivery* Network here. There are CDNs out there who burn to improve their customer experience (both the content creators and the content receiver) at high cost. Having a Tier1 attitude and telling eyeball networks with <1Gbps of traffic exchanged to bugger off or pay is not one of the ways to improve this.
I do agree here, again on an IX level. They still have the choice to backhaul to you or hot potato your traffic wherever they feel like it. From the data you have provide it appears to be a lose-lose situation to de-peer you.
At the end of the day I'm going to charge CDNs who want to deliver their customers content to my eyeballs and make me pay (about 2USD per Mbps, with a minimum of 1Gbps).
That may be what they are doing "if at all possible try to monetize it".
Agreed on all points. There's at least nothing technical which needs solving.
On 5/1/12, Dominik Bay <db@rrbone.net> wrote:
Yesterday I received the following mail, from a CDN:
---->8---- Greetings,
Limelight Networks [has] recently updated our requirements for settlement-free peering
This letter is to notify you that yyy no longer meets our minimum requirements.
Proposed solution: Greetings, Where settlement-free peering has been offered but rejected, YYY moves all data traffic transiting that AS through a single minimum-cost Internet connection (cough Cogent cough) with the attendant impact to reliability. We appreciate the notice of depeering and will endeavor to identify and advise those making paid use of our respective services as to the impending impact to their activities. On the technical side you can only easily enforce that for outbound traffic. Essentially filter the routes containing their AS except from your minimum cost link. Intentionally degraded service can be better than flat refusal. Communication is two-way so even though the CDN sends more than it receives this is still a credible threat. Your customer sees perfectly good connectivity everywhere else and it isn't a complete disconnect so your customer assumes its "their" Internet connection rather than his.
I totally understand that some companies might not be able to handle sub-5Mbps peering sessions, be it technical or organisational, but >=100Mbps should be worth any effort, as long as it improves the network.
If I'm willing to go to your location, buy the card for your router and pay you for the staff hours to set it up, there should be *no* situation in which I'm willing to accept your traffic from an upstream Internet link but am unwilling to engage in otherwise settlement-free peering with you. Your customers have paid you to connect to me and my customers have paid me to connect to you. Double-billing the activity by either of us collecting money from the other is just plain wrong. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On May 1, 2012, at 13:26 , William Herrin wrote:
On 5/1/12, Dominik Bay <db@rrbone.net> wrote:
Yesterday I received the following mail, from a CDN:
---->8---- Greetings,
Limelight Networks [has] recently updated our requirements for settlement-free peering
I love the fact Dominik says "from a CDN", then leaves Limelight's name in the text. :)
If I'm willing to go to your location, buy the card for your router and pay you for the staff hours to set it up, there should be *no* situation in which I'm willing to accept your traffic from an upstream Internet link but am unwilling to engage in otherwise settlement-free peering with you.
I disagree with this. In fact, I can think of several possible cases where this would not hold, both using pure business and pure technical justifications. Generalizations are difficult in complex situations, and this is most definitely a complex situation.
Your customers have paid you to connect to me and my customers have paid me to connect to you. Double-billing the activity by either of us collecting money from the other is just plain wrong.
Wrong? My rule is: Your network, your decision. (Anyone who is paying attention knows my decision, but I it would be quite silly to assume my decision is right for all networks in all situations.) Asking for settlements is not illegal, or even immoral. Moreover, this is an operational list. "Right" and "wrong" are not really part of the discussion. But even if they were, this is not not "just plain wrong". "It's just business" is a much better way to say it, and in business, trying to make more money is the _point_, not wrong. Whether this is a good way to make money is left as an exercise to the reader. Instead, let's focus on the operational impact. Will the reduced complexity on these networks result in improved performance? Irrelevant to performance? Decreased performance? Maybe even whether that change in performance is an acceptable trade for the lower CapEx/OpEx? This is relevant since business requirements are the foundation for operational discussions. Can't buy more 10G ports if the business doesn't support it. Etc., etc. But right vs. wrong in a peering dispute? I think not. -- TTFN, patrick
On 05/01/2012 08:08 PM, Patrick W. Gilmore wrote:
Instead, let's focus on the operational impact. Will the reduced complexity on these networks result in improved performance? Irrelevant to performance? Decreased performance? Maybe even whether that change in performance is an acceptable trade for the lower CapEx/OpEx? This is relevant since business requirements are the foundation for operational discussions. Can't buy more 10G ports if the business doesn't support it.
While it would be easier to troubleshoot, I might decrease performance by the same or likely the same metrics. Peering via IXP/PNI pro: - more, maybe better paths - more ways to loadbalance traffic over various PNI/IXP - added redundancy cons: - needs grow to maintain sessions - maintain a contact database for the specific networks - tools needed to pinpoint issues on node, PNI / IXP, network level "Feeding" via some bigger peer networks oder classic transit pro: - single point of contact - less sessions to maintain (say 400 sessions for all bigger europe cities instead of 200 sessions per IXP) - easier view on traffic flows (depends on your tools though) cons: - if a single network breaks, it might have a bigger impact - not able to (easily) mitigate issues (ceasing announcements to a peer vs. turning down a transit session) - troubleshooting mostly outsourced to a 3rd party It depends on what one needs, and what one wants to pay. -dominik
In a message written on Tue, May 01, 2012 at 08:23:07PM +0200, Dominik Bay wrote:
"Feeding" via some bigger peer networks oder classic transit
You have made the assumption that their choice is peering with your network or sending it out transit. They may in fact peer with your upstream. That makes their choice peer with you, or peer with your upstream. Peering with your upstream may allow them to reach many people like you for cost of managing only a single peering session, as compared to maintaining a few dozen. Also, many networks have minimum volume amounts for peering relationships. They may be able to get settlement free peering with your upstream by having some minimum traffic level that they would not have if they peer with some of the individual customers behind that upstream. Peering with you may drop them below the threshold, causing them to pay for transit on 10's of Gigs of traffic. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On 05/01/2012 09:17 PM, Leo Bicknell wrote:
In a message written on Tue, May 01, 2012 at 08:23:07PM +0200, Dominik Bay wrote:
"Feeding" via some bigger peer networks oder classic transit
You have made the assumption that their choice is peering with your network or sending it out transit. They may in fact peer with your upstream.
No, I also assume 'bigger peer networks', usually a smaller ISPs upstream with a bigger regional footprint.
That makes their choice peer with you, or peer with your upstream. Peering with your upstream may allow them to reach many people like you for cost of managing only a single peering session, as compared to maintaining a few dozen.
Agreed, more reach with a single session, but with pros there are cons, as mentioned in my last mail. If they choose to accept these limitations, that's fine for me. But one should be aware, especially when aiming towards low latency, high bandwidth connections to eyeballs.
Also, many networks have minimum volume amounts for peering relationships. They may be able to get settlement free peering with your upstream by having some minimum traffic level that they would not have if they peer with some of the individual customers behind that upstream. Peering with you may drop them below the threshold, causing them to pay for transit on 10's of Gigs of traffic.
Agreed, but this is more a monetary optimization, not a technical optimization. I don't agree on any peering request, especially if this would force traffic on paths which are preferred but sub-optimal. A situation which wouldn't be optimal can be this one: Say a network in Sweden and Estonia reach them via one of their upstreams which do have a direct connection. They are both a member of an IXP in, say, Amsterdam. Both networks exchange their prefixes, and set localpref at this IXP. This would make the path worse than before, as traffic takes a detour, and might cost more due to backhauling traffic.
On Tue, May 1, 2012 at 2:17 PM, Leo Bicknell <bicknell@ufp.org> wrote:
In a message written on Tue, May 01, 2012 at 08:23:07PM +0200, Dominik Bay wrote:
"Feeding" via some bigger peer networks oder classic transit
You have made the assumption that their choice is peering with your network or sending it out transit. They may in fact peer with your upstream.
That makes their choice peer with you, or peer with your upstream. Peering with your upstream may allow them to reach many people like you for cost of managing only a single peering session, as compared to maintaining a few dozen.
Also, many networks have minimum volume amounts for peering relationships. They may be able to get settlement free peering with your upstream by having some minimum traffic level that they would not have if they peer with some of the individual customers behind that upstream. Peering with you may drop them below the threshold, causing them to pay for transit on 10's of Gigs of traffic.
Lets be honest. There are a million reasons we can all come up with to try and justify something like this but 99% of the time it is just the larger isp trying to throw their weight around in the name of greed. In the end, the customers of both companies are the ones who suffer and ultimately pay (figuratively and literally) for it.
On May 1, 2012, at 16:24 , Jerry Dent wrote:
Lets be honest. There are a million reasons we can all come up with to try and justify something like this but 99% of the time it is just the larger isp trying to throw their weight around in the name of greed. In the end, the customers of both companies are the ones who suffer and ultimately pay (figuratively and literally) for it.
1) Welcome to business 101. 2) Is that bad? 3) I think your 99% number is very inflated. But then 99% of stats are made up. :) -- TTFN, patrick
On Tue, May 1, 2012 at 3:30 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
On May 1, 2012, at 16:24 , Jerry Dent wrote:
Lets be honest. There are a million reasons we can all come up with to try and justify something like this but 99% of the time it is just the larger isp trying to throw their weight around in the name of greed. In the end, the customers of both companies are the ones who suffer and ultimately pay (figuratively and literally) for it.
1) Welcome to business 101.
2) Is that bad?
Can be for the end users if they wind up on a less direct network path.
3) I think your 99% number is very inflated. But then 99% of stats are made up. :)
Probably.
In a message written on Tue, May 01, 2012 at 03:45:29PM -0500, Jerry Dent wrote:
Can be for the end users if they wind up on a less direct network path.
"Direct" is not the only measure. I would take a 4-hop, 10GE, no packet loss path over a 1-hop, 1GE, 5% packet loss path any day of the week. "Shorter" {hops, latency, as-path} does not mean a higher quality end user experience. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Tue, May 1, 2012 at 3:54 PM, Leo Bicknell <bicknell@ufp.org> wrote:
In a message written on Tue, May 01, 2012 at 03:45:29PM -0500, Jerry Dent wrote:
Can be for the end users if they wind up on a less direct network path.
"Direct" is not the only measure.
I would take a 4-hop, 10GE, no packet loss path over a 1-hop, 1GE, 5% packet loss path any day of the week.
"Shorter" {hops, latency, as-path} does not mean a higher quality end user experience.
I was using "Direct" as a generic term. And if the issue was link performance, company A would have sent company B a "shape up or we'll de-peer" message rather than a "pay up or we'll de-peer" message.
On 5/1/12, Patrick W. Gilmore <patrick@ianai.net> wrote:
On May 1, 2012, at 13:26 , William Herrin wrote:
If I'm willing to go to your location, buy the card for your router and pay you for the staff hours to set it up, there should be *no* situation in which I'm willing to accept your traffic from an upstream Internet link but am unwilling to engage in otherwise settlement-free peering with you.
I disagree with this. In fact, I can think of several possible cases where this would not hold, both using pure business and pure technical justifications.
Hi Patrick, Please educate me. I'd be happy to adopt a more nuanced view.
Your customers have paid you to connect to me and my customers have paid me to connect to you. Double-billing the activity by either of us collecting money from the other is just plain wrong.
Wrong? My rule is: Your network, your decision.
Yes, wrong. Some decisions fall in to areas covered by general ethics. You sell a customer a red ball when you know you can only deliver green balls, it's a lie. Unethical. Wrong. You work for a company (W2 salary) and in the course of your work contract something to another company where you're an officer it's a conflict of interest. Unethical. Wrong. A customer pays you to build a piece of software by the hour. Another comes along and asks for the same software. You bill both for each hour. Double billing. Unethical. Wrong. A customer pays you to deliver a packet to "the Internet." You talk to the packet's destination and say, "Hey, I'll deliver it to you directly but only if you pay me. Otherwise I'll just toss it out in a random direction and hope it gets there." Double billing. Unethical. Wrong. None of these things is necessarily illegal although like spam some of them are illegal under specific conditions. Yet all of them (and spam) are Wrong. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On May 1, 2012, at 14:43 , William Herrin wrote:
On 5/1/12, Patrick W. Gilmore <patrick@ianai.net> wrote:
On May 1, 2012, at 13:26 , William Herrin wrote:
If I'm willing to go to your location, buy the card for your router and pay you for the staff hours to set it up, there should be *no* situation in which I'm willing to accept your traffic from an upstream Internet link but am unwilling to engage in otherwise settlement-free peering with you.
I disagree with this. In fact, I can think of several possible cases where this would not hold, both using pure business and pure technical justifications.
Hi Patrick,
Please educate me. I'd be happy to adopt a more nuanced view.
First, define "upstream Internet link" - i.e. upstream to whom? If I peer with your upstream, then peering with you could easily drop me below their requirements, causing me to lose a much bigger peer for what I may consider no real benefit. Let's assume you meant upstream to me. Many people negotiate volume discounts, peering with you may drop me to a lower tier, which may cost me more than your peering saves me. Let's further assume it is upstream to me. Perhaps I am trying to turn that upstream into a peer. This devolves into the first situation. Irrelevant on what you meant by upstream, I may have operational issues such as space & power, that disallow me from adding another port pointed at you in the location you want the port. I don't care if you buy the card, if there is no slot or power for the card, buying another rack - if possible in the first place! - may not be worth the benefit of peering with you. And I haven't even covered the CapEx & OpEx of adding a chassis to deal with your port irrespective of the power & space. Etc. Hopefully you get the gist.
Your customers have paid you to connect to me and my customers have paid me to connect to you. Double-billing the activity by either of us collecting money from the other is just plain wrong.
Wrong? My rule is: Your network, your decision.
Yes, wrong. Some decisions fall in to areas covered by general ethics.
You sell a customer a red ball when you know you can only deliver green balls, it's a lie. Unethical. Wrong.
You work for a company (W2 salary) and in the course of your work contract something to another company where you're an officer it's a conflict of interest. Unethical. Wrong.
I see your point here, but neither of those examples are peering related.
A customer pays you to build a piece of software by the hour. Another comes along and asks for the same software. You bill both for each hour. Double billing. Unethical. Wrong.
This is far less clear. Should the second customer get it for free just because you already wrote the code? Or is it the fact you billed "an hour" instead of "software" that bothers you? Not that it matters, since this has nothing to do with peering.
A customer pays you to deliver a packet to "the Internet." You talk to the packet's destination and say, "Hey, I'll deliver it to you directly but only if you pay me. Otherwise I'll just toss it out in a random direction and hope it gets there." Double billing. Unethical. Wrong.
I think if you look closely at any transit contract, almost none of them (cough Akamai cough) guarantee delivery t the end user. They typically only guarantee delivery to the edge of their own network, and make zero promises about _which_ edge they will use to get to the next network. Plus those "guarantees" are weaker than almost any other guarantee in any industry. Of course, if someone sold me "full transit" and could not reach a significant fraction of the Internet, I'd claim breach of contract. But A != B. So the straw man above _may_ be morally wrong (I'm not 100% certain of it, need to ruminate some more), it doesn't actually exist in the real world. And certainly is not related to the original post. Besides, "double billing" is not unethical in your example above. Using your exact straw man, if I go to the destination, make that threat, and they cave, I haven't harmed my customer at all. If they don't cave and I send them the packet directly anyway, I still haven't harmed my customer. Double-billing in-and-of itself is not morally wrong. All that said, if a provider sucks, change providers. <Cue discussion about being unable to change providers and how wrong that is. :->
None of these things is necessarily illegal although like spam some of them are illegal under specific conditions. Yet all of them (and spam) are Wrong.
We can agree that spam is wrong. :) -- TTFN, patrick
On Tue, May 1, 2012 at 3:06 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
On May 1, 2012, at 14:43 , William Herrin wrote:
On 5/1/12, Patrick W. Gilmore <patrick@ianai.net> wrote:
On May 1, 2012, at 13:26 , William Herrin wrote:
If I'm willing to go to your location, buy the card for your router and pay you for the staff hours to set it up, there should be *no* situation in which I'm willing to accept your traffic from an upstream Internet link but am unwilling to engage in otherwise settlement-free peering with you.
I disagree with this. In fact, I can think of several possible cases where this would not hold, both using pure business and pure technical justifications.
Hi Patrick,
Please educate me. I'd be happy to adopt a more nuanced view.
First, define "upstream Internet link" - i.e. upstream to whom?
Hi Patrick, By "upstream" I meant "via any other path." If we're mutually willing to talk at all, we should be willing to talk directly.
If I peer with your upstream, then peering with you could easily drop me below their requirements, causing me to lose a much bigger peer for what I may consider no real benefit.
Two wrongs. It's OK for me to behave unreasonably to you because I'm just passing it on from someone else.
Let's assume you meant upstream to me. Many people negotiate volume discounts, peering with you may drop me to a lower tier, which may cost me more than your peering saves me.
I'll have to think about this one.
I may have operational issues such as space & power, that disallow me from adding another port pointed at you in the location you want the port. I don't care if you buy the card, if there is no slot or power for the card, buying another rack - if possible in the first place! - may not be worth the benefit of peering with you. And I haven't even covered the CapEx & OpEx of adding a chassis to deal with your port irrespective of the power & space.
Fair point; I was too specific. If I'm willing to cover the -direct costs- (such as an extra router card or man hours, extra rack space, etc.) associated with peering with me at a location in which you have already made the choice to peer with others I think it unreasonable of you to refuse given that each of our respective customers has paid us to deliver packets to the other. Here's another one you didn't raise, but I'll throw it out there anyway: Why should I be able to demand to peer with your entire network at any location which happens to be convenient for me? I shouldn't. I should be able to demand to peer with that fraction of your network which is consistent with any other peering activity you have at that location. If you peer at 5 geographically dispersed locations and I only want to peer with you at one, I would think it imminently reasonable for you to restrict the session to that portion of your network which ordinarily transits to peers there. I wouldn't even be offended by an offer for discount transit to the rest in addition to the settlement free part.
I think if you look closely at any transit contract, almost none of them (cough Akamai cough) guarantee delivery t the end user. They typically only guarantee delivery to the edge of their own network, and make zero promises about _which_
I think that's disingenuous. Customers are buying "Internet" service, not "the networks we feel like connecting you to" service. Both you and your customer know it despite any weasel words in the contract. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
"A customer pays you to build a piece of software by the hour. Another comes along and asks for the same software. You bill both for each hour. Double billing. Unethical. Wrong. A customer pays you to deliver a packet to "the Internet." You talk to the packet's destination and say, "Hey, I'll deliver it to you directly but only if you pay me. Otherwise I'll just toss it out in a random direction and hope it gets there." Double billing. Unethical. Wrong." Neither of these is unethical or wrong in any way. What are you supposed to do, write software from scratch every time? That's just silly. On Tue, May 1, 2012 at 11:43 AM, William Herrin <bill@herrin.us> wrote:
On 5/1/12, Patrick W. Gilmore <patrick@ianai.net> wrote:
On May 1, 2012, at 13:26 , William Herrin wrote:
If I'm willing to go to your location, buy the card for your router and pay you for the staff hours to set it up, there should be *no* situation in which I'm willing to accept your traffic from an upstream Internet link but am unwilling to engage in otherwise settlement-free peering with you.
I disagree with this. In fact, I can think of several possible cases where this would not hold, both using pure business and pure technical justifications.
Hi Patrick,
Please educate me. I'd be happy to adopt a more nuanced view.
Your customers have paid you to connect to me and my customers have paid me to connect to you. Double-billing the activity by either of us collecting money from the other is just plain wrong.
Wrong? My rule is: Your network, your decision.
Yes, wrong. Some decisions fall in to areas covered by general ethics.
You sell a customer a red ball when you know you can only deliver green balls, it's a lie. Unethical. Wrong.
You work for a company (W2 salary) and in the course of your work contract something to another company where you're an officer it's a conflict of interest. Unethical. Wrong.
A customer pays you to build a piece of software by the hour. Another comes along and asks for the same software. You bill both for each hour. Double billing. Unethical. Wrong.
A customer pays you to deliver a packet to "the Internet." You talk to the packet's destination and say, "Hey, I'll deliver it to you directly but only if you pay me. Otherwise I'll just toss it out in a random direction and hope it gets there." Double billing. Unethical. Wrong.
None of these things is necessarily illegal although like spam some of them are illegal under specific conditions. Yet all of them (and spam) are Wrong.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
On 5/1/12, Mike Hale <eyeronic.design@gmail.com> wrote:
"A customer pays you to build a piece of software by the hour. Another comes along and asks for the same software. You bill both for each hour. Double billing. Unethical. Wrong. [...] Neither of these is unethical or wrong in any way. What are you supposed to do, write software from scratch every time? That's just silly.
Hi Mike, If one of the customers happens to be the U.S. Government, it's not only unethical it's a crime. It's usually a felony. You can do time. The product was man hours. You've sold them once. You can't sell them again. The customers can agree to share the cost up front. But then you're not billing the same hours twice, you're billing half an hour to one customer and half to another. When you finish the software you can sell the software to a second customer. But you *may not* tie your price to the hours used to produce it for the first. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
"If one of the customers happens to be the U.S. Government, it's not only unethical it's a crime. It's usually a felony. You can do time. The product was man hours. You've sold them once. You can't sell them again." You're assuming the contract is simply for work hours. Generally speaking, and from my experience, it isn't. The contract is for an app that does X, not 20 hours toward building an app that does X. "But you *may not* tie your price to the hours used to produce it for the first." Sure you can. How else do you determine what the software's going to cost if you're not going to factor in development? On Tue, May 1, 2012 at 2:06 PM, William Herrin <bill@herrin.us> wrote:
On 5/1/12, Mike Hale <eyeronic.design@gmail.com> wrote:
"A customer pays you to build a piece of software by the hour. Another comes along and asks for the same software. You bill both for each hour. Double billing. Unethical. Wrong. [...] Neither of these is unethical or wrong in any way. What are you supposed to do, write software from scratch every time? That's just silly.
Hi Mike,
If one of the customers happens to be the U.S. Government, it's not only unethical it's a crime. It's usually a felony. You can do time. The product was man hours. You've sold them once. You can't sell them again.
The customers can agree to share the cost up front. But then you're not billing the same hours twice, you're billing half an hour to one customer and half to another. When you finish the software you can sell the software to a second customer. But you *may not* tie your price to the hours used to produce it for the first.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
Let's keep our eye on the ball, people. Did the original post have any operational consequences? IMHO, it has many. Some are even interesting to the wider audience. So why are we discussing how you bill the US gov't? -- TTFN, patrick P.S. Bill, it is clear you have a point, but you are really stretching it. And it is not relevant to the discussion at hand. On May 1, 2012, at 17:13 , Mike Hale wrote:
"If one of the customers happens to be the U.S. Government, it's not only unethical it's a crime. It's usually a felony. You can do time. The product was man hours. You've sold them once. You can't sell them again." You're assuming the contract is simply for work hours. Generally speaking, and from my experience, it isn't. The contract is for an app that does X, not 20 hours toward building an app that does X.
[snip]
On Tue, 01 May 2012 17:16:38 -0400, "Patrick W. Gilmore" said:
P.S. Bill, it is clear you have a point, but you are really stretching it. And it is not relevant to the discussion at hand.
Oh, I dunno. Double billing 2 customers for development and double billing 2 customers for transporting packets seem related ;)
On Tue, 01 May 2012 14:13:01 -0700, Mike Hale said:
"But you *may not* tie your price to the hours used to produce it for the first."
The above was William Herrin's comment (quoting level fixed by me). Mike - please get mail software that does correct quoting. It's 2012, and proper quoting has been understood since the mid 80s. There's *really* no excuse for using software that can't get quoting and citing right.
Sure you can. How else do you determine what the software's going to cost if you're not going to factor in development?
You missed the point - having given customer #1 an invoice that included a line item for 1,432 hours of R&D at $221/hour, you're treading on thin ice if you present another customer an invoice that includes a line item for the same 1,432 hours of R&D (absent an agreement between the two customers to share the costs, etc). And if you've *collected* that $316,472 from the one customer, it's somewhere between sleazy and skanky to include that $316K in the costs that need to be amortized over the next N sales of the software.
Mike - please get mail software that does correct quoting. It's 2012, and proper quoting has been understood since the mid 80s. There's *really* no excuse for using software that can't get quoting and citing right. *eye roll* Really? You wasted 36 words on this?
And if you've *collected* that $316,472 from the one customer, it's somewhere between sleazy and skanky to include that $316K in the costs that need to be amortized over the next N sales of the software. It's neither sleazy nor skanky. It's called profit. I get what you're saying, but it's a silly argument because, while you're not going to bill the same "hours" (as a unit) twice, you sure as hell are going to bill over and over again for the same work...you'd be stupid not to.
On Tue, May 1, 2012 at 2:20 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Tue, 01 May 2012 14:13:01 -0700, Mike Hale said:
"But you *may not* tie your price to the hours used to produce it for the first."
The above was William Herrin's comment (quoting level fixed by me).
Mike - please get mail software that does correct quoting. It's 2012, and proper quoting has been understood since the mid 80s. There's *really* no excuse for using software that can't get quoting and citing right.
Sure you can. How else do you determine what the software's going to cost if you're not going to factor in development?
You missed the point - having given customer #1 an invoice that included a line item for 1,432 hours of R&D at $221/hour, you're treading on thin ice if you present another customer an invoice that includes a line item for the same 1,432 hours of R&D (absent an agreement between the two customers to share the costs, etc).
And if you've *collected* that $316,472 from the one customer, it's somewhere between sleazy and skanky to include that $316K in the costs that need to be amortized over the next N sales of the software.
-- 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
On Tue, 01 May 2012 14:27:50 -0700, Mike Hale said:
Mike - please get mail software that does correct quoting. It's 2012, and proper quoting has been understood since the mid 80s. There's *really* no excuse for using software that can't get quoting and citing right. *eye roll* Really? You wasted 36 words on this?
Wasn't wasted - this time your message included > in the right places. :) And yes, proper attribution *is* important.
On 5/1/2012 5:20 PM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 01 May 2012 14:13:01 -0700, Mike Hale said:
"But you *may not* tie your price to the hours used to produce it for the first." The above was William Herrin's comment (quoting level fixed by me).
Mike - please get mail software that does correct quoting. It's 2012, and proper quoting has been understood since the mid 80s. There's *really* no excuse for using software that can't get quoting and citing right.
Sure you can. How else do you determine what the software's going to cost if you're not going to factor in development? You missed the point - having given customer #1 an invoice that included a line item for 1,432 hours of R&D at $221/hour, you're treading on thin ice if you present another customer an invoice that includes a line item for the same 1,432 hours of R&D (absent an agreement between the two customers to share the costs, etc).
And if you've *collected* that $316,472 from the one customer, it's somewhere between sleazy and skanky to include that $316K in the costs that need to be amortized over the next N sales of the software.
From an accounting perspective, every R&D effort that I have seen or been a part of was not billed to any customer. R&D has always, in my experience, been an internal charge against a company's own profits.
As to hourly software development, I have never seen or been a part of a software development effort where the final work product was not owned by the entity paying for the hourly development. The contract software developer can't sell the same software twice because they don't own it to sell it twice. It is possible that my software development experience, while broad, has missed a model where R&D is billed to customers or software isn't owned by those who paid to have it developed... If a company expends their own resources to develop a piece of software (i.e. a product) regardless of whether through R&D (internal development) or contracting with someone else to develop the product (external development), then they can set the price of that product to whatever they like. There is nothing sleazy or skanky if the price of the product multiplied by the number of copies sold is positive after subtracting the price of the development of the product - we call that profit. -DMM
On Tue, 01 May 2012 18:03:06 -0400, David Miller said:
From an accounting perspective, every R&D effort that I have seen or been a part of was not billed to any customer. R&D has always, in my experience, been an internal charge against a company's own profits.
RIght - and when pricing the result of the R&D, you take into account the internal charges and estimated sales volume to determine a target price point. R&D was $316K and you estimate you can sell 100 of them, you're going to have to charge at least $3,160 a copy to make a profit on the project. You sank $150M into R&D and think you'll sell a million units, you'll need to charge $150 to break even. And so on. The point is that if you already got *paid* the $316K, it's morally wrong to include it in the calculation of "sunk costs we need to recoup to turn a profit".
On Tue, May 1, 2012 at 6:03 PM, David Miller <dmiller@tiggee.com> wrote:
From an accounting perspective, every R&D effort that I have seen or been a part of was not billed to any customer. R&D has always, in my experience, been an internal charge against a company's own profits.
Hi David, That's called "internal" research and development or "IR&D". It's distinguished from research and prototyping as an exploratory effort under contract to a customer. For example, all grants under organizations like NSF, ONR or DARPA the latter type of R&D. They want to know if something is possible so they pay knowledge domain experts to find out. Or more commonly, the experts submit a proposal that says, "We think this is possible. It would be very good for you if it is. Won't you please pay us to find out?"
As to hourly software development, I have never seen or been a part of a software development effort where the final work product was not owned by the entity paying for the hourly development. The contract software developer can't sell the same software twice because they don't own it to sell it twice.
That depends on the contract. By law, the folks who wrote the software (or the company who paid their salaries) own it. By law, contract != salary. Some contracts specify that the copyrights and other IP will be signed over upon completion. With most of the contracts I've worked, the customer doesn't get the copyright, but I make no claim that's typical. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Tue, May 01, 2012 at 02:43:50PM -0400, William Herrin wrote:
On 5/1/12, Patrick W. Gilmore <patrick@ianai.net> wrote:
On May 1, 2012, at 13:26 , William Herrin wrote:
If I'm willing to go to your location, buy the card for your router and pay you for the staff hours to set it up, there should be *no* situation in which I'm willing to accept your traffic from an upstream Internet link but am unwilling to engage in otherwise settlement-free peering with you.
I disagree with this. In fact, I can think of several possible cases where this would not hold, both using pure business and pure technical justifications.
Hi Patrick,
Please educate me. I'd be happy to adopt a more nuanced view.
Your customers have paid you to connect to me and my customers have paid me to connect to you. Double-billing the activity by either of us collecting money from the other is just plain wrong.
Wrong? My rule is: Your network, your decision.
Yes, wrong. Some decisions fall in to areas covered by general ethics.
You are high. If I've entered into contracts with multiple parties to deliver their traffic, there is no 'double dipping' or 'double billing'. Ignore any sort of traffic *type*. To assert that some transit entity with two customers paying to reach the universe (happens to include each party reciprocally) should go out of their way to discount for such customer to customer traffic is both madness of bellhead-accounting level and a quick route to bankruptcy. Stupid and nothing YOU would certainly do for your customers...?
You sell a customer a red ball when you know you can only deliver green balls, it's a lie. Unethical. Wrong.
Utterly irrelevant to the discussion.
You work for a company (W2 salary) and in the course of your work contract something to another company where you're an officer it's a conflict of interest. Unethical. Wrong.
Utterly irrelevant to the discussion.
A customer pays you to build a piece of software by the hour. Another comes along and asks for the same software. You bill both for each hour. Double billing. Unethical. Wrong.
Utterly irrelevant to the discussion.
A customer pays you to deliver a packet to "the Internet." You talk to the packet's destination and say, "Hey, I'll deliver it to you directly but only if you pay me. Otherwise I'll just toss it out in a random direction and hope it gets there." Double billing. Unethical. Wrong.
Mindset of an inexperienced small provider who believes "the Internet" is something 'out there' and your infrastructure isn't part of it. Your customers give you traffic at your handoff to deliver to/from others. Some of those others amy also be your customers; are you giving them free traffic customer-to-customer [complex accounting at your egress to peers and transit] or are you billing them for traffic transferred [port stats facing the customer]? Discussion of applicability of one model or another to this or that type of business is an interesting academic exercise, but if you aren't willing to give your customers free transit across your service to each other, your position is at best disingenuous. At worst, hypocritical. Cheers! Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE / NewNOG
On 5/1/12, Patrick W. Gilmore <patrick@ianai.net> wrote:
I love the fact Dominik says "from a CDN", then leaves Limelight's name in the text. :) [snip]
So a CDN made the mistake of attempting to monetize an existing peering arrangement without first having a signed peering arrangement in place for the existing peering with a Mutual NDA prohibiting any dissemination details of any of their communications/ future negotiations/etc of peering arrangement or traffic exchanged to unrelated third parties?? The lack of a NDA that would prohibit identifying a CDN and their e-mail notification of depeering, whether their name was mentioned or not, sounds quite un-Tier1 like to me. -- -JH
Right on Thanks, Ameen Pishdadi On May 1, 2012, at 11:39 AM, Dominik Bay <db@rrbone.net> wrote:
Yesterday I received the following mail, from a CDN:
---->8---- Greetings,
Limelight Networks periodically reviews its interconnection strategy to ensure the quality and integrity of its interconnection between all its partners. We have recently updated our requirements for settlement-free peering which are posted at http://www.as22822.net/
This letter is to notify you that yyy no longer meets our minimum requirements. If yyy would like to maintain our current interconnectivity, there will be settlement associated with doing so. If you are interested in pursuing this option, please reply back to this email indicating such.
Should your company decline this option, or if we do not have an agreement regarding the settlement in place prior to May 31st 2012, Limelight Networks will terminate the peering sessions on that day, with this letter serving as 30 day notice.
Sincerely, ----8<----
The same mail was sent out last year, about end of April 2011, to another ISP I'm working with. They got depeered, but the ISP which received the mail above yesterday didn't meet the requirements last year either. I totally understand that some companies might not be able to handle sub-5Mbps peering sessions, be it technical or organisational, but >=100Mbps should be worth any effort, as long as it improves the network.
In this particular case I'm talking about >=600Mbps of traffic send out by Limelight to "my" eyeballs, not mentioning their fairly small footprint in Germany in comparison to other CDNs.
These points aside, we are talking about a Content *Delivery* Network here. There are CDNs out there who burn to improve their customer experience (both the content creators and the content receiver) at high cost. Having a Tier1 attitude and telling eyeball networks with <1Gbps of traffic exchanged to bugger off or pay is not one of the ways to improve this.
At the end of the day I'm going to charge CDNs who want to deliver their customers content to my eyeballs and make me pay (about 2USD per Mbps, with a minimum of 1Gbps).
-dominik
participants (12)
-
Ameen Pishdadi -
David Miller -
Dominik Bay -
Jerry Dent -
Jimmy Hess -
Joe Provo -
Leo Bicknell -
Mike Hale -
Patrick W. Gilmore -
Steven Noble -
Valdis.Kletnieks＠vt.edu -
William Herrin