I've been doing a lot of reading and thinking on what the best solution is for an application that initially requires approximately 15Mbps outgoing and 8Mbps incoming (as viewed from my router), and talks with 500000 unique hosts daily (i.e. has fairly wide coverage of the net). The application involves at least thirty machines, so colocation is likely to be cost-prohibitive. A single T3, or frac T3 isn't an option because there isn't a single provider that I can trust for the availability we want. Even ignoring availability, I seriously doubt that any provider can consistently fill a single customer's T3 pipe these days.
From stuff I've seen here and elsewhere I think the most important reason for this is congestion at NAPs making it impossible to suck (or shove) lots of bandwidth at anything but your provider's backbone. Taking all of this into account, I'm really leaning towards a solution that involves lots of small pipes to lots of providers. Essentially eliminating the need for 90% of our packets to traverse NAPs by using each backbone mostly for their own customers.
Now that I've done my homework, I'd like to hear comments from some of the more experienced folk here. I haven't considered yet the maintenance/logistical cost of managing 15 T1s to 6 or 7 providers vs. the "ease" of two frac-T3s to two providers.
From a provider's point of view, if a site wanted to connect, and was willing to sign a use-policy saying they wouldn't use the connection for transit to other providers (i.e. would only ask for customer BGP and only route to the nets you provide in BGP updates), would that site have lower costs associated with it? (that you could pass on?)
Thanks, Dean
On 21 Jul 1996, Dean Gaudet wrote:
of this into account, I'm really leaning towards a solution that involves lots of small pipes to lots of providers. Essentially eliminating the need for 90% of our packets to traverse NAPs by using each backbone mostly for their own customers.
Are you sure you can't accomplish this with your own national backbone and private interconnects with the major providers similar to what Sprint/MCI are doing to keep traffic off the NAP's? On the other hand, maybe you could be the customer that establishes the distributed web server scenario I discussed earlier. If you have read through http://www.ix.digital.com you will not that not only are they running an exchange point but they are also running a web farm of sorts at the same location. Chances are good that this web-farm-at-the-XP concept will become the rule rather than the exception. Note that in Digital's model it would be possible to connect to larger ISP's without requiring traffic to flow through the XP itself. Michael Dillon - ISP & Internet Consulting Memra Software Inc. - Fax: +1-604-546-3049 http://www.memra.com - E-mail: michael@memra.com
On the other hand, maybe you could be the customer that establishes the distributed web server scenario I discussed earlier. If you have read through http://www.ix.digital.com you will not that not only are they running an exchange point but they are also running a web farm of sorts at the same location. Chances are good that this web-farm-at-the-XP concept will become the rule rather than the exception. Note that in Digital's model it would be possible to connect to larger ISP's without requiring traffic to flow through the XP itself.
At Digital's Palo Alto IX (URL as Michael said above), we view the exchange point as a place where peering relationships are implemented. Web farmers who are not ISPs (I'm not trying to start a debate on what-is-an-ISP) can only peer with ISPs at the GIGAswitch if the web farmer is able to find an ISP that wishes to provide connectivity in that manner. I would hope that they wouldn't. My personal feeling is that the provision of service should be implemented on a separate port of the ISP's router - this provides both the ISP and the web farmer with a measurable point of demarcation independent of the IX. If the web farmer paid for an Ethernet or whatever interface, they'd get an Ethernet or whatever interface, and the bandwidth available to the customer on that port would not vary with other traffic as it would if the web farmer were competing with the ISP's peers for an interface attached to the GIGAswitch. Should the web farmer purchase connectivity from other ISPs, their purchases can be implemented as cross-connects to ISP routers (assuming the address space can be advertised, the topology of the web farmer's network can handle it, etc., etc., etc.). ISPs might also wish to implement certain peering relationships with cross-connects rather than consume bandwidth on their interface to the GIGAswitch. To us, cross-connects are cross-connects, whether they connect ISPs to web farmers or ISPs to ISPs. Stephen - ----- Stephen Stuart stuart@pa.dec.com Network Systems Laboratory Digital Equipment Corporation
On Sun, 21 Jul 1996, Stephen Stuart wrote:
At Digital's Palo Alto IX (URL as Michael said above), we view the exchange point as a place where peering relationships are implemented. Web farmers who are not ISPs (I'm not trying to start a debate on what-is-an-ISP) can only peer with ISPs at the GIGAswitch if the web farmer is able to find an ISP that wishes to provide connectivity in that manner. I would hope that they wouldn't.
You sound overly negative here. I understand that you see a BIG difference between a web server hooked up to a Gigaswitch port and a web server in an ISP's rack getting to the Gigaswitch through the ISP's FDDI or 100baseTx or whatever. But potential users of the XP location don't think that way. If you want to see what your bosses are advertising, read the press release at http://www.ix.digital.com/press.html
Should the web farmer purchase connectivity from other ISPs, their purchases can be implemented as cross-connects to ISP routers (assuming the address space can be advertised, the topology of the web farmer's network can handle it, etc., etc., etc.).
This is the key thing here. You are providing the physical facilities to locate the servers as well as the wiring required to get any sort of connection desired to the ISP's in the same building including zero-mile T1's and DS3's. And you are promoting this service. This is what is new and I expect that either Digital will do a few more of these in other locations and/or other XP locations will also expand their facilities to do this. Over the next 5 years XP's like this will be found in every major North American city and the big content providers *WILL* have resolved some technology that allows them to serve content from the topologically closest server. Michael Dillon - ISP & Internet Consulting Memra Software Inc. - Fax: +1-604-546-3049 http://www.memra.com - E-mail: michael@memra.com
On Sun, 21 Jul 1996, Stephen Stuart wrote:
At Digital's Palo Alto IX (URL as Michael said above), we view the exchange point as a place where peering relationships are implemented. Web farmers who are not ISPs (I'm not trying to start a debate on what-is-an-ISP) can only peer with ISPs at the GIGAswitch if the web farmer is able to find an ISP that wishes to provide connectivity in that manner. I would hope that they wouldn't.
You sound overly negative here. I understand that you see a BIG difference between a web server hooked up to a Gigaswitch port and a web server in an ISP's rack getting to the Gigaswitch through the ISP's FDDI or 100baseTx or whatever. But potential users of the XP location don't think that way. If you want to see what your bosses are advertising, read the press release at http://www.ix.digital.com/press.html
I certainly wasn't trying to sound negative; I know what the press release says, having reviewed it before it went out. The press release doesn't talk about "connections to the IX" are implemented. This is an implementation issue; if a web farmer wished to create a topology where they were speaking BGP to ISPs with whom they arranged connectivity, I would not stand in the way. There are benefits to that topology that require a level of sophistication that some of the web farmers to whom I've spoken are prepared to implement and maintain. Many are not, though. I hope to see topologies matched to customers in a way that gives all customers the level of flexibility and autonomy they desire. I'm not the only voice in the design of such topologies, though, and my opinion is not necessarily The Truth. The ISPs and the web farmers are the customers, and at the end of the day, I hope to have done what produces the most customer satisfaction.
Should the web farmer purchase connectivity from other ISPs, their purchases can be implemented as cross-connects to ISP routers (assuming the address space can be advertised, the topology of the web farmer's network can handle it, etc., etc., etc.).
This is the key thing here. You are providing the physical facilities to locate the servers as well as the wiring required to get any sort of connection desired to the ISP's in the same building including zero-mile T1's and DS3's. And you are promoting this service. This is what is new and I expect that either Digital will do a few more of these in other locations and/or other XP locations will also expand their facilities to do this. Over the next 5 years XP's like this will be found in every major North American city and the big content providers *WILL* have resolved some technology that allows them to serve content from the topologically closest server.
We are in agreement here. Stephen - ----- Stephen Stuart stuart@pa.dec.com Network Systems Laboratory Digital Equipment Corporation
My personal feeling is that the provision of service should be implemented on a separate port of the ISP's router - this provides both the ISP and the web farmer with a measurable point of demarcation
My personal feeling is that any Unix machines should be provisioned on separate IP space from the routers. And that perhaps a default- monitoring program (which we run ourselves anyway) might be run to make sure that the machines aren't defaulting to anyone if the machines are hooked up to the same switched fabric as the routers (which I think is a bad idea). I'm not arguing that getting transit via an XP fabric is a bad idea - as long as the XP provider gets beefy enough switching and as long as the transit providers have enough ports into their routers from the XP fabric, it's a fine idea - much better than lots of rack-rack cables. But boxes w/ hard drives are a different story IMO.
independent of the IX. If the web farmer paid for an Ethernet or whatever interface, they'd get an Ethernet or whatever interface, and the bandwidth available to the customer on that port would not vary with other traffic as it would if the web farmer were competing with the ISP's peers for an interface attached to the GIGAswitch. Should the web farmer purchase connectivity from other ISPs, their purchases can be implemented as cross-connects to ISP routers (assuming the address space can be advertised, the topology of the web farmer's network can handle it, etc., etc., etc.).
ISPs might also wish to implement certain peering relationships with cross-connects rather than consume bandwidth on their interface to the GIGAswitch. To us, cross-connects are cross-connects, whether they connect ISPs to web farmers or ISPs to ISPs.
Stephen
Avi
On Mon, 22 Jul 1996, Avi Freedman wrote:
I'm not arguing that getting transit via an XP fabric is a bad idea - as long as the XP provider gets beefy enough switching and as long as the transit providers have enough ports into their routers from the XP fabric, it's a fine idea - much better than lots of rack-rack cables.
But boxes w/ hard drives are a different story IMO.
Agreed. IMO, the best place for these things is one hops away from an XP, either in a provider's network or behind a content provider's router that's connected to the XP. -dorian
On Mon, 22 Jul 1996, Avi Freedman wrote:
Agreed. IMO, the best place for these things is one hops away from an XP, either in a provider's network or behind a content provider's router that's connected to the XP.
Yes, 100%, I would pick MAE-East. MAE-East is the closest you are going to get to the world, and there is more data going over it then any other NAP. You may also want to look at putting a server up at a few NAPs, this is what we are doing now for one of our users. This will be a HUGE www audio provider, and they need fast access with as little delay as possible. We will be providing them colo at MAE-East, MAE-West, and Chicago NAP. The people they are accessing the site will then hit a image map and get served out of the closest server. Nathan Stratton CEO, NetRail, Inc. Tracking the future today! --------------------------------------------------------------------------- Phone (703)524-4800 NetRail, Inc. Fax (703)534-5033 2007 N. 15 St. Suite 5 Email sales@netrail.net Arlington, Va. 22201 WWW http://www.netrail.net/ Access: (703) 524-4802 guest --------------------------------------------------------------------------- "Therefore do not worry about tomorrow, for tomorrow will worry about itself. Each day has enough trouble of its own." Matthew 6:34
On Mon, 22 Jul 1996, Avi Freedman wrote:
Agreed. IMO, the best place for these things is one hops away from an XP, either in a provider's network or behind a content provider's router that's connected to the XP.
Yes, 100%, I would pick MAE-East. MAE-East is the closest you are going to get to the world, and there is more data going over it then any other NAP. You may also want to look at putting a server up at a few NAPs, this is what we are doing now for one of our users. This will be a HUGE www audio provider, and they need fast access with as little delay as possible. We will be providing them colo at MAE-East, MAE-West, and Chicago NAP. The people they are accessing the site will then hit a image map and get served out of the closest server.
The fact that MAE-East is the busiest exchange point in the world should push you to locate servers elsewhere, IMHO. There are many providers who really don't want more traffic at MAE-East. With all the overload happening now and the lack of additional connections to share traffic, do you really want to add significant content to the bargain? Karl ----------------------------------------------------------------------------- Karl Mueller karl@best.net (415) 944-8228 Sr. Network Engineer Best Internet Communications, Inc. -----------------------------------------------------------------------------
On Mon, 22 Jul 1996, Karl Mueller wrote:
Yes, 100%, I would pick MAE-East. MAE-East is the closest you are going to get to the world, and there is more data going over it then any other NAP. You may also want to look at putting a server up at a few NAPs, this is what we are doing now for one of our users. This will be a HUGE www audio provider, and they need fast access with as little delay as possible. We will be providing them colo at MAE-East, MAE-West, and Chicago NAP. The people they are accessing the site will then hit a image map and get served out of the closest server.
The fact that MAE-East is the busiest exchange point in the world should push you to locate servers elsewhere, IMHO. There are many providers who really don't want more traffic at MAE-East. With all the overload happening now and the lack of additional connections to share traffic, do you really want to add significant content to the bargain?
The delay at MAE-East is because some larger providers have just 1 DS3 connection when they need more. The gigaswitch can do almost 1000 times what it is doing now. The OC-3s that are connecting the gigaswitches together also have plenty of room on them. Nathan Stratton CEO, NetRail, Inc. Tracking the future today! --------------------------------------------------------------------------- Phone (703)524-4800 NetRail, Inc. Fax (703)534-5033 2007 N. 15 St. Suite 5 Email sales@netrail.net Arlington, Va. 22201 WWW http://www.netrail.net/ Access: (703) 524-4802 guest --------------------------------------------------------------------------- "Therefore do not worry about tomorrow, for tomorrow will worry about itself. Each day has enough trouble of its own." Matthew 6:34
The delay at MAE-East is because some larger providers have just 1 DS3 connection when they need more. The gigaswitch can do almost 1000 times what it is doing now. The OC-3s that are connecting the gigaswitches together also have plenty of room on them.
Nathan Stratton CEO, NetRail, Inc. Tracking the future today!
It's possible that some providers may not have the time or resources to increase their bandwidth to MAE-East; also, it's possible that some of them may decide not to, choosing instead private interconnects or other exchange points. The point being made was that MAE-East is not necessarily the best place to be for robustness and performance; the point stands even though, as you assert, MAE-East is (still) the largest public exchange point in the known multiverse. Avi
Nathan Stratton writes:
The delay at MAE-East is because some larger providers have just 1 DS3 connection when they need more. The gigaswitch can do almost 1000 times what it is doing now. The OC-3s that are connecting the gigaswitches together also have plenty of room on them.
That was not the point he was trying to make. The fact that mae-east is the busiest peering point should be a reason to stay _away_ from it and try to spread the bandwidth load around to other peering points. The busier mae-east gets, the more trouble the entire net gets into when it blows up (which has been known to happen). You have said many times that the gigaswitch is not anywhere near capacity, and you are probably right. However, what you repeatedly forget to realize is that there are many other factors that need to be considered. Alec -- +------------------------------------+--------------------------------------+ |Alec Peterson - chuckie@panix.com | Panix Public Access Internet and UNIX| |Network Administrator | New York City, NY | +------------------------------------+--------------------------------------+
private interconnects with the major providers similar to what Sprint/MCI are doing to keep traffic off the NAP's?
On the other hand, maybe you could be the customer that establishes the distributed web server scenario I discussed earlier. If you have read through http://www.ix.digital.com you will not that not only are they running an exchange point but they are also running a web farm of sorts at the same location. Chances are good that this web-farm-at-the-XP concept will become the rule rather than the exception. Note that in Digital's model it would be possible to connect to larger ISP's without requiring traffic to flow through the XP itself.
The differences between digitals exchange and MAE-LA seem to be limited to these items: Digital is escorted MAE-LA is not (things are caged at MAELA) Digital has its generator installed MAE-LA will have its installed in Dec Digital has -48vdc ready MAE-LA has exausted its -48vdc capacity and is adding more Digital has its gigaswitch installed MAE-LA has not, since there is not yet a demand In other ways, they are functionally identical. Both have multicarrier access Both support public and private interconnections Both support commercial access Both havee colocation services Both have UPS,preaction fire, cooling, the whole environmental pkg Oops. forgot these: Digital has a large marketing budget MAE-LA does not Digital is 20-25% more expensive MAE-LA is based on cost recovery Note that MAE-LA, like MAE-West, is not an MFS only activity. They have thier own rates, the other players have theirs. http://www.isi.edu/div7/mla -- --bill
The differences between digitals exchange and MAE-LA seem to be limited to these items:
Digital is escorted MAE-LA is not (things are caged at MAELA)
Digital is escorted if your racks are in the common area. If you buy enough racks (like three), we'll cage off the space and you'll on longer require escort. Stephen - ----- Stephen Stuart stuart@pa.dec.com Network Systems Laboratory Digital Equipment Corporation
On Sun, 21 Jul 1996, Michael Dillon wrote:
On 21 Jul 1996, Dean Gaudet wrote:
of this into account, I'm really leaning towards a solution that involves lots of small pipes to lots of providers. Essentially eliminating the need for 90% of our packets to traverse NAPs by using each backbone mostly for their own customers.
Are you sure you can't accomplish this with your own national backbone and private interconnects with the major providers similar to what Sprint/MCI are doing to keep traffic off the NAP's?
I believe the current ante to get into this game is DS3 infrastructure with presence at 3 exchange points. This is a slightly different order of magnitude than what's Dean seems to be talking about. -dorian
I'm going to skip commenting on most of Dean's note and simply answer one of his questions that nobody else seems to have touched upon...
Now that I've done my homework, I'd like to hear comments from some of the more experienced folk here.
Glad to volunteer them. :-)
I haven't considered yet the maintenance/logistical cost of managing 15 T1s to 6 or 7 providers vs. the "ease" of two frac-T3s to two providers.
I would suggest you do so very soon; it will have a major impact on the way you conceptualize and design your operation. Keep in mind that a fractional T3 vs a T1 from most reasonably sized ISP's isn't that much different, typically around $1000/month for a T1 vs $3000 to $4000/month for a fractional T3 line. With the T1 lines, you're limited in how much data you can push out, and you don't really have any upgrade path as far as bandwidth goes. Also, keep in mind the routing issues involved with 15 T1's versus 3 T3's are VERY different; unless you have a VERY good routing guru locally, I wouldn't want to attempt juggling 15 T1's to different AS's as a first effort venture. :-)
From a provider's point of view, if a site wanted to connect, and was willing to sign a use-policy saying they wouldn't use the connection for transit to other providers (i.e. would only ask for customer BGP and only route to the nets you provide in BGP updates), would that site have lower costs associated with it? (that you could pass on?)
Of course! In fact, transit is a separate line item when you order; all you need to do is not ask for transit, and save the associated cost. Then all you have access to is our internal customers and backbone, we don't announce your routes out at all. Of course, I should have prefaced this by mentioning that I'm a network engineer, not a sales person, but since we're the ones that told sales how to sell this stuff... :)
Thanks, Dean
Matt Petach, speaking almost somewhat officially for InterNex Information Services (but just for this post!) mpetach@internex.net
On 21 Jul 1996, Dean Gaudet wrote:
The application involves at least thirty machines, so colocation is likely to be cost-prohibitive. A single T3, or frac T3 isn't an option because there isn't a single provider that I can trust for the availability we want. Even ignoring availability, I seriously doubt that any provider can consistently fill a single customer's T3 pipe these days.
of this into account, I'm really leaning towards a solution that involves lots of small pipes to lots of providers. Essentially eliminating the need for 90% of our packets to traverse NAPs by using each backbone mostly for their own customers.
I haven't considered yet the maintenance/logistical cost of managing 15 T1s to 6 or 7 providers vs. the "ease" of two frac-T3s to two providers.
Given that it sounds like you are budgeting for slightly more than a DS3 worth of bandwidth, in connectivity cost, then the way to do multi T1 is to pick a set of ISPs that comprise a good percentage of the net, and get N x T1 to the ISPs based on your best guess break down of traffic. If it's just the whole Internet you are aiming for, then something like 4 T1s to MCI, 3 T1s to Sprint, 3 T1s to UUNET, 2 to ANS, 1 to AGIS and 2 to others is an example. Note that this is a wild guesstimate of the percentage of Internet traffic sinks. What you need to think about with this scenario are the following: 1) cost of procuring 15 T1s vs. DS3/fract DS3. 2) logistics of support. 3) infrastructure issues. 1) is pretty cut and dry. 2) is the largest "hidden" cost. With this set up, you need a competent net engineer or two to babysit this so that the packets are flowing in the right direction. You also ideally need significant automation of your router configurations so that you can pull correct info and generate configs that match a very fluid reality of today's routing. You'll also need a decent NOC to deal with the 6-7 ISP NOCs and possibly different carrier's NOCs when trouble hits. I find that much of trouble shooting involves live body at an end of a phone much more than a brain. 3) you need to have different equipment to do this. It costs more to provision hardware and to do 15 T1s then it does for 1-2 DS3/fract DS3. This doesn't even go to redundancy and sparing issues. Something like Cisco 7200 might this better, but I'm not sure. The other scenario of two fract DS3 alleviates the problem #2 and #3, but still doesn't make them go away altogether. You also need to pick providers with interesting enough traffic sinks so that you can load balance effectively (as effectively as you can get in that situation anyway) in a somewhat straigh forward fashion. ( like taking internal customer routes from ISP A and the rest via ISP B.
From a provider's point of view, if a site wanted to connect, andwas willing to sign a use-policy saying they wouldn't use the connection for transit to other providers (i.e. would only ask for customer BGP and only route to the nets you provide in BGP updates), would that site have lower costs associated with it? (that you could pass on?)
It would seem to me that, as long as the site is an interesting enough traffic source, and the ISP can recoup whatever cost of offering that connection + margin(or not). Speaking only for CICNet, given that the site is an interesting traffic source, we'd gradly offer a connection for what it costs to provide that connection, if such request came to us. Hope this helps, -dorian
On Mon, 22 Jul 1996, Dorian R. Kim wrote:
From a provider's point of view, if a site wanted to connect, andwas willing to sign a use-policy saying they wouldn't use the connection for transit to other providers (i.e. would only ask for customer BGP and only route to the nets you provide in BGP updates), would that site have lower costs associated with it? (that you could pass on?) It would seem to me that, as long as the site is an interesting enough traffic source, and the ISP can recoup whatever cost of offering that connection + margin(or not). Speaking only for CICNet, given that the site is an interesting traffic source, we'd gradly offer a connection for what it costs to provide that connection, if such request came to us.
Who else will do this? MCI, Sprint? This seems like a new and growing market. Is it being added to the service menus of various backbones? We'd love to talk to any backbone that will do this. Chris Caputo President, Altopia Corporation
participants (11)
-
Alec H. Peterson -
Avi Freedman -
bmanning＠isi.edu -
Chris Caputo -
dgaudet＠hotwired.com -
Dorian R. Kim -
Karl Mueller -
Matthew Petach -
Michael Dillon -
Nathan Stratton -
Stephen Stuart