Current trends in capacity planning and oversubscription
While the answer is always it depends, I was wondering what the current rules of thumb university network engineers are using for capacity planning and oversubscription for resnets and admin networks? For K-12, SETDA (http://www.setda.org/web/guest/2020/broadband) is recommending: - An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff - Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff How does that compare with university and enterprise network rules of thumb?
On 11/10/2010 12:26 AM, Sean Donelan wrote:
While the answer is always it depends, I was wondering what the current rules of thumb university network engineers are using for capacity planning and oversubscription for resnets and admin networks?
For K-12, SETDA (http://www.setda.org/web/guest/2020/broadband) is recommending:
- An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff - Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff
How does that compare with university and enterprise network rules of thumb?
I can tell you that no one I know in K-12 or Higher Education has the capacity for 1GE per 1000 students. Same goes for the WAN connection. More than likely *if* they have fiber from every school to a central location it's a single pair and they couldn't afford to go 10GE. Though I have answered some RFPs where they wanted to know what it would cost. (Usually 2.5 * 1GE cost.) I believe my Alma mater just reached 150Mbps for the 6,500 students + faculty. Sure beats the 30Mbps they started with when I was a Freshmen. I also just finished an RFP where a campus of at least 4,000 only had 100Meg. This was in a suburban environment with at least a dozen carriers competing.
On Wed, 10 Nov 2010 07:42:32 EST, ML said:
- An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff
30K students here, 2x10GE to the outside world.
- Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff
Not applicable when you're mostly just one campus, unless you want to discuss per-building feeds. Most of our buildings have 1 or more 1G uplinks to a 10GE core network, and some larger ones have their own 10G uplink. There's a few outlier buildings that have a dozen or so people that still have 100M uplinks.
I can tell you that no one I know in K-12 or Higher Education has the capacity for 1GE per 1000 students. Same goes for the WAN connection.
We're working on deploying 1G to the desktop. Some buildings are going faster than others - the ones we wired first are now the hardest, because there's a lot of Cat-3 in them. The later ones that already have Cat-6 will be easier. With something like 120 buildings to do, it's going to take a while.
On Tue, Nov 9, 2010 at 10:26 PM, Sean Donelan <sean@donelan.com> wrote:
While the answer is always it depends, I was wondering what the current rules of thumb university network engineers are using for capacity planning and oversubscription for resnets and admin networks?
For K-12, SETDA (http://www.setda.org/web/guest/2020/broadband) is recommending:
- An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff - Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff
How does that compare with university and enterprise network rules of thumb?
As someone else has said I've never seen K-12 with remotely that high of a ratio, or, really, any educational institution. UofM here in Missoula, MT doesn't have anywhere near those ratios for internet services nor for the campus network. I don't have any exact details but I'm pretty certain there's no 10 Gig-E there. I'm not even sure if the building-to-building links are 1 Gig-E in all cases. Actually...I'm not sure anywhere has that high of a ratio here in the states, at least for wired connectivity. The carriers here all keep the prices nice and high to preserve their profit margins in the face of losing their long distance and traditional POTS cash cows as people move more to cell phones and other non POTS carriers.
I am on the technology committee of the college I attended (Whitman) and they currently have a 200 mbit/sec via gigE link for a campus of just under 2000 and every building has at least 1X gigE into their backbone. They are in a rural area (walla walla, wa) but they don't generally have more than 100 or 150 mbit/sec of usage, fitting nicely in the below recommendations. -----Original Message----- From: Michael Loftis [mailto:mloftis@wgops.com] Sent: Wednesday, November 10, 2010 8:48 AM To: Sean Donelan Cc: nanog Subject: Re: Current trends in capacity planning and oversubscription On Tue, Nov 9, 2010 at 10:26 PM, Sean Donelan <sean@donelan.com> wrote:
While the answer is always it depends, I was wondering what the current rules of thumb university network engineers are using for capacity planning and oversubscription for resnets and admin networks?
For K-12, SETDA (http://www.setda.org/web/guest/2020/broadband) is recommending:
- An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff - Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff
How does that compare with university and enterprise network rules of thumb?
Michael Loftis expunged (mloftis@wgops.com):
Actually...I'm not sure anywhere has that high of a ratio here in the states, at least for wired connectivity.
I would say that's highly dependent on your geographical location. In Montana I could see that as being true, but not in NYC, for example... -Steve
From: Steve Meuse > Sent: Wednesday, November 10, 2010 9:31 AM To: Michael Loftis Cc: nanog Subject: Re: Current trends in capacity planning and oversubscription
Michael Loftis expunged (mloftis@wgops.com):
Actually...I'm not sure anywhere has that high of a ratio here in
the
states, at least for wired connectivity.
I would say that's highly dependent on your geographical location. In Montana I could see that as being true, but not in NYC, for example...
-Steve
I'd say it also depends on the nature of studies at the school, too. A general liberal arts school might have different needs than one heavy on science and engineering. Throw in some specialized resource like a radio telescope or nuclear research lab and it changes even more as those research data are shared among other collaborators at other institutions.
On Nov 10, 2010, at 12:40 56PM, George Bonser wrote:
From: Steve Meuse > Sent: Wednesday, November 10, 2010 9:31 AM To: Michael Loftis Cc: nanog Subject: Re: Current trends in capacity planning and oversubscription
Michael Loftis expunged (mloftis@wgops.com):
Actually...I'm not sure anywhere has that high of a ratio here in
the
states, at least for wired connectivity.
I would say that's highly dependent on your geographical location. In Montana I could see that as being true, but not in NYC, for example...
-Steve
I'd say it also depends on the nature of studies at the school, too. A general liberal arts school might have different needs than one heavy on science and engineering. Throw in some specialized resource like a radio telescope or nuclear research lab and it changes even more as those research data are shared among other collaborators at other institutions.
It also depends on how much file-sharing the students are doing... --Steve Bellovin, http://www.cs.columbia.edu/~smb
On Wed, Nov 10, 2010 at 10:31 AM, Steve Meuse <smeuse@mara.org> wrote:
Michael Loftis expunged (mloftis@wgops.com):
Actually...I'm not sure anywhere has that high of a ratio here in the states, at least for wired connectivity.
I would say that's highly dependent on your geographical location. In Montana I could see that as being true, but not in NYC, for example...
It might be more dependent upon the level of competition int he bandwidth market. Here in the mountain west (MT included) you either pay the ILEC their exorbitant fees for your last mile, or, you pay to trench. Only a couple years ago Qwest was quoting $8k+/mo for a DS3, and this was where their Cisco ONS 15454 was in the same room. I don't even want to know how much they'd charge you if you had to pay any real line mileage. Luckily int hat particular building there were/are other options, but almost any other place, you don't (that building has a lot of antennae on top, a couple placed by myself)
-Steve
Sometimes it is a hard sell, but the factor most overlooked when designing high speed networks is that of designing for low latency. Bandwidth and over/under subscription are only part of the network design. Low latency networks (regional RTTs of 1-5 milliseconds; campus RTTs in the sub millisecond range for GiGE, and microsecond range for 10 GiGE) by their nature solve a lot of QoS problems, often relegating QoS as a method to be used in emergency cases only, such as DoS attacks. Here is something I have been thinking is not too far over the horizon: commodity 48 port x 10/100 GiGE switches; 10 GiG NICs on the desktop; commodity-priced 10,40, or 100 GiGE WAN links; while user bandwidth requirements rise at a much slower rate. Some switch vendors even today have 48 port 10 GiGE switches in the $25K range. My rule of thumb is that 1 GiGE link is an order of magnitude lower latency than a 100 Mbps link, and a 10 GiGE link is an order of magnitude lower latency than a GiGE link. -----Original Message----- From: Sean Donelan [mailto:sean@donelan.com] Sent: Tuesday, November 09, 2010 9:26 PM To: nanog@nanog.org Subject: Current trends in capacity planning and oversubscription While the answer is always it depends, I was wondering what the current rules of thumb university network engineers are using for capacity planning and oversubscription for resnets and admin networks? For K-12, SETDA (http://www.setda.org/web/guest/2020/broadband) is recommending: - An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff - Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff How does that compare with university and enterprise network rules of thumb?
On Nov 9, 2010, at 11:26 PM, Sean Donelan wrote:
While the answer is always it depends, I was wondering what the current rules of thumb university network engineers are using for capacity planning and oversubscription for resnets and admin networks?
For K-12, SETDA (http://www.setda.org/web/guest/2020/broadband) is recommending:
- An external Internet connection to the Internet Service Provider of at least 100 Mbps per 1,000 students/staff - Internal wide area network connections from the district to each school and between schools of at least 1 Gbps per 1,000 students/staff
How does that compare with university and enterprise network rules of thumb?
Page 10 of the presentation on the link you provided says those are the numbers they recommend "for the next 5-7 years". Perhaps they meant to say "in 5 to 7 years"? "for the next 2-3 years" they recommend 10 Mbps per 1,000 people to an ISP and 100 Mbps per 1,000 people between schools. Our campus is already using almost 3 times the ISP bandwidth recommended "for the next 2-3 years". If we take our current ISP bandwidth and increase it by 50% every year for 5 years it would be about twice the 100 Mbps per 1,000 students/staff recommendation. --- Bruce Curtis bruce.curtis@ndsu.edu Certified NetAnalyst II 701-231-8527 North Dakota State University
On Wed, 10 Nov 2010, Curtis, Bruce wrote:
If we take our current ISP bandwidth and increase it by 50% every year for 5 years it would be about twice the 100 Mbps per 1,000 students/staff recommendation.
Is 50% growth each year typical these days? In the dot-com boom days, people said 100% growth, other people have suggested 20% may be more reasonable now. A problem with government network capacity planning/growth forecasts is you will be stuck with whatever you choose, too high or too low, for many years because the budget cycle is so long. It would be great if there was some actual data available. But it seems more typical to benchmark/compare to do network capacity planning with other government agencies, so we end up with X-Mbps per Y,000 people. Yes, I know it depends. 1,000 people downloading data from LHC experiments will be different from an administrative school office. The difference is the people using LHC data usually have someone who can figure out network capacity planning, while the people in an administrative school office may not have anyone. So what is a reasonable network capacity for 1,000 students now and in 5 years.
On Fri, 12 Nov 2010 18:52:20 EST, Sean Donelan said:
The difference is the people using LHC data usually have someone who can figure out network capacity planning, while the people in an administrative school office may not have anyone.
So what is a reasonable network capacity for 1,000 students now and in 5 years.
Just as LHC people and a school are different, I'm willing to bet that bandwidth "requirements" per student are different based on the school and its policies, and that they're to a large extent self-fullfilling. The "requirement" at a small liberal arts school with a fascist network usage policy "we block all bittorrent and any protocols we don't understand (i.e. most of them), and no network access in the dorms", will be different from a large engineering school that says "We'll provide bandwidth so you can explore, experiment, and learn, and we'll let you know if you're named in a copyright complaint". So "reasonable" bandwidth ends up depending on what the network admin thinks "reasonable" use of the network is...
On 11/13/2010 12:37 PM, Valdis.Kletnieks@vt.edu wrote:
On Fri, 12 Nov 2010 18:52:20 EST, Sean Donelan said:
So what is a reasonable network capacity for 1,000 students now and in 5 years. Just as LHC people and a school are different, I'm willing to bet that bandwidth "requirements" per student are different based on the school and its policies, and that they're to a large extent self-fullfilling.
There are "a number" of factors at play, but looking at a given snapshot of network activity, some of the relevant metrics are the number of active hosts [count inside clients if you are a content "consumer", count outside clients of you are a content "provider"], the number of active connections, and compare that to the bandwidth consumption. It doesn't really matter how many students you have, it's how many are online and active at a given time.
The "requirement" at a small liberal arts school with a fascist network usage policy "we block all bittorrent and any protocols we don't understand (i.e. most of them), and no network access in the dorms", will be different from a large engineering school that says "We'll provide bandwidth so you can explore, experiment, and learn, and we'll let you know if you're named in a copyright complaint".
That is true to a degree in terms of your degree of network neutrality. We do block most P2P, so I am not so certain of the bandwidth consumption it would take these days if left open. I do, however, know that other traffic has simply risen to take it's place. Well over half of our bandwidth (less P2P) is streaming media, and a growing chunk of that is Netflix. I could take 50Mbps and divide it among a thousand web surfers, or a hundred Youtube viewers, or a few dozen Netflix folks, or one of our Athletics Dragonfly video servers. You can size it for a given population and a given application mix, then be hit by the Obama inauguration, the Michael Jackson funeral, the World Cup playoffs, or just a good healthy Patch Tuesday or Apple MacOS update.
So "reasonable" bandwidth ends up depending on what the network admin thinks "reasonable" use of the network is...
And then there are limits to your ability to predict the "reasonable" demands :-) Which is where things like 95th percentile numbers start to come in handy. Another interesting metric is looking at "per user" consumption over time. Just as the applications have certain fringe cases that consume an inordinate of bandwidth, you also have a few fringe case uses that will account for an inordinate amount of bandwidth. If your goal is to accomodate these folks equally with the rest of the population, your bandwidth requirements will be MUCH higher to accomodate them. So with all that said... for what it's worth... for the last 24hours, our (sustained) inbound bandwidth (peak/avg) was 317Mbps / 107Mbps, inside concurrently active host counts 2017/986, concurrent connection counts 22.6K/10.8K. That is with 11K students, 1500 fac/staff, and the numbers being considerably higher during the week. The 95th percentile number is ~250Mbps for a typical weekday (24h), but we do "cap" dorm traffic during classroom hours. To backup another point brought up in this thread, the demand has roughly doubled annually the last 3 years. Jeff
I've been tracking Internet bandwidth usage for several schools (RESNET-L reports), and I've seen it as low at 2.1 kbps/FTE (5+ years ago) to higher than 300 kbps/FTE. Most of the schools are between 30 to 60 kbps/FTE at this time. Very broadly speaking, rural and smaller schools are on the low end, while state schools with tens of thousands of students are on the higher end. Internet2 plays in the mix as well. In terms of growth rates, while our ISP serves only one college, even though they've been growing their bandwidth pipe at low-double digits per year, they are maxed out most days from mid-day to 1 am. If most ISPs see end-user traffic grow 50 to 80% per year, I can't see why schools would be much out of that range. Frank -----Original Message----- From: Sean Donelan [mailto:sean@donelan.com] Sent: Friday, November 12, 2010 5:52 PM To: NANOG list Subject: Re: Current trends in capacity planning and oversubscription On Wed, 10 Nov 2010, Curtis, Bruce wrote:
If we take our current ISP bandwidth and increase it by 50% every year for 5 years it would be about twice the 100 Mbps per 1,000 students/staff recommendation.
Is 50% growth each year typical these days? In the dot-com boom days, people said 100% growth, other people have suggested 20% may be more reasonable now. A problem with government network capacity planning/growth forecasts is you will be stuck with whatever you choose, too high or too low, for many years because the budget cycle is so long. It would be great if there was some actual data available. But it seems more typical to benchmark/compare to do network capacity planning with other government agencies, so we end up with X-Mbps per Y,000 people. Yes, I know it depends. 1,000 people downloading data from LHC experiments will be different from an administrative school office. The difference is the people using LHC data usually have someone who can figure out network capacity planning, while the people in an administrative school office may not have anyone. So what is a reasonable network capacity for 1,000 students now and in 5 years.
On Nov 12, 2010, at 5:52 PM, Sean Donelan wrote:
On Wed, 10 Nov 2010, Curtis, Bruce wrote:
If we take our current ISP bandwidth and increase it by 50% every year for 5 years it would be about twice the 100 Mbps per 1,000 students/staff recommendation.
Is 50% growth each year typical these days? In the dot-com boom days, people said 100% growth, other people have suggested 20% may be more reasonable now.
We did see a lower rate of growth after the dot-com boom/bust. However the rate of growth picked up with the popularity of video streaming sites. This site mentions 40 to 50% growth last year and has references to other papers that mention similar growth rates (although some of those papers may now be several years old.) http://www.dtc.umn.edu/mints/home.php So to answer the question I would say that 40 to 50% growth is typical these days, it has been for us. I assume that it will continue for a few years but I'm less confidant speculating that it would still be 40 to 50% in 5 to 7 years. But I wouldn't bet against it either.
A problem with government network capacity planning/growth forecasts is you will be stuck with whatever you choose, too high or too low, for many years because the budget cycle is so long.
It would be great if there was some actual data available. But it seems more typical to benchmark/compare to do network capacity planning with other government agencies, so we end up with X-Mbps per Y,000 people. Yes, I know it depends. 1,000 people downloading data from LHC experiments will be different from an administrative school office. The difference is the people using LHC data usually have someone who can figure out network capacity planning, while the people in an administrative school office may not have anyone.
So what is a reasonable network capacity for 1,000 students now and in 5 years.
--- Bruce Curtis bruce.curtis@ndsu.edu Certified NetAnalyst II 701-231-8527 North Dakota State University
participants (12)
-
Curtis, Bruce -
Frank Bulk -
George Bonser -
Holmes,David A -
Jeff Kell -
John van Oppen -
Michael Loftis -
ML -
Sean Donelan -
Steve Meuse -
Steven Bellovin -
Valdis.Kletnieks＠vt.edu