If a new operator or city is building a greenfield access network from the ground up, what software and hardware is needed in the core network to provide and manage residential and business internet services similar to the likes of AT&T, Comcast, and Google Fiber? Television and Telephone services are not to be considered only internet. Assume hypothetically the operator already has the following in place: 10 GPON OLTs Chassis from an access vendor in 10 POPs around town (each POP has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. Dark fiber going from the POP locations back to the core location Assume a 32:1 way split, and each OLT chassis has enough ports populated to serve the area. 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged mode. Assume you are building a network designed to serve 10,000 subs All the fiber splitters, ducts, fiber, etc connecting the OLTs to the ONTs is already in place ASN from ARIN /20 of IPv4 space and /32 of IPv6 space from ARIN 4 burstable 10G internet connections from 4 tier 1 internet providers Questions are: What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive? Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch? How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? Dynamic IP addresses? DHCP? How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts? If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else? What about DNS? Is a firewall needed in the core? What else is needed? Is there a guide out there somewhere? I know many cities are looking at building their own network, and have similar questions. Access vendors are willing to sell gear all day long, but then they leave it up to the operator/city to answer these harder questions. How would you build a access network from the ground up if you had the resources and time to do so? Would you even use GPON? Even if GPON was not used and another access technology like AE, VDSL2, or wireless was used I think many of these questions would be the same.
"What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive?" Definitely aggregate into a switch first unless you want to run a Layer 3 switch as your router, which I don't recommend. "Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch?" Your math is a little backwards, its very unlikely that you're going to have 40 Gbps of Internet (or other interconnection) for the router to actually have to process. What is the average provisioned speed for each of the 10k PON ports? What over subscription rate are you planning for? What, if anything, will you be carrying on net, ie bandwidth consumption that won't come from or go to the public Internet? Your own video, voice, or other service are examples of things that are often on net. In any case you're probably in the ASR family with Cisco and I can't remember the equivalent from Juniper. How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? CGN is the option of last resort IMO, but you may have to consider it. A better approach is to see if your backbone providers will agree to give some blocks that you can announce and use those blocks for dynamic customers only. Your static IP customers should come from your direct ARIN allotment in case you need to choose a new backbone provider, which is extremely common over time. "Dynamic IP addresses? DHCP?" DHCP with enforcement from the shelves. All the major OLT vendors support doing this so that a customer can only use the address assigned to him by DHCP and nothing else, except for those customers that you choose to hard code. Make most of your "static" customers actually DHCP reservations and only hard code those that you must. "How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts?" DHCP, with Option 82 logging for the circuit ID is the better path than a BRAS (PPPoE) these days. Here's a paper we put together on that topic a while back: http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPP... Depending on your OLT vendor you can either use their built in port isolation or QinQ tagging, both are reliable and scalable, just ask your vendor which is the best option for your specific gear. "If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else?" I wouldn't have those two services connected personally, though there are hooks for some of the CGN boxes to talk to DHCP servers. I would hope you can get another 6k addresses and avoid the need for CGN altogether. Having said that, have you tested your OLTs and ONTs for IPv6 interoperability? If they don't handle it well then you're going to have to think about alternatives like 6RD (http://en.wikipedia.org/wiki/IPv6_rapid_deployment) For DHCP at your scale you can run ISC DHCP ( http://www.isc.org/downloads/dhcp/) which is the most common open source DHCP daemon if you someone who can take care of a Linux server, parse the Option 82 information for logging, and handle the configuration of the DHCP daemon itself. Otherwise you might want to look at commercial products designed for the service provider market like Incongito's BCC and Cisco's BAC (CNR replacement) http://www.incognito.com/products/broadband-command-center/ http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-acc... "What about DNS? Is a firewall needed in the core? What else is needed?" There are two kinds of DNS, caching (recursive) and authoritative. The first is what your customers will use to resolve things on the Internet and the second is used to provide caching name servers on the Internet with information about domains you control (are authoritative for). The first needs good performance, availability, and scalability since your customers will use your caching name servers constantly. Most people can run BIND at your scale, again if you have someone with Linux experience, but there are other alternatives. PowerDNS has both caching and authoritative modules and there are some commercial offerings out there both as cloud hosting and local deployments. Your backbone provider will also often have caching name servers your customers can use, but the quality varies quite a bit. You can also, especially at first, leverage some of the free offerings like Google's DNS. I don't recommend firewalls for service provider networks, but you should make sure your gear can run (and is configured to do so) BCP 38. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor@gmail.com> wrote:
If a new operator or city is building a greenfield access network from the ground up, what software and hardware is needed in the core network to provide and manage residential and business internet services similar to the likes of AT&T, Comcast, and Google Fiber? Television and Telephone services are not to be considered only internet.
Assume hypothetically the operator already has the following in place: 10 GPON OLTs Chassis from an access vendor in 10 POPs around town (each POP has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. Dark fiber going from the POP locations back to the core location Assume a 32:1 way split, and each OLT chassis has enough ports populated to serve the area. 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged mode. Assume you are building a network designed to serve 10,000 subs All the fiber splitters, ducts, fiber, etc connecting the OLTs to the ONTs is already in place ASN from ARIN /20 of IPv4 space and /32 of IPv6 space from ARIN 4 burstable 10G internet connections from 4 tier 1 internet providers
Questions are:
What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive? Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch? How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? Dynamic IP addresses? DHCP? How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts? If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else? What about DNS? Is a firewall needed in the core? What else is needed?
Is there a guide out there somewhere? I know many cities are looking at building their own network, and have similar questions. Access vendors are willing to sell gear all day long, but then they leave it up to the operator/city to answer these harder questions.
How would you build a access network from the ground up if you had the resources and time to do so? Would you even use GPON? Even if GPON was not used and another access technology like AE, VDSL2, or wireless was used I think many of these questions would be the same.
Scott, Thanks for the long post. We will use a layer 2 10G aggregation switch then to aggregate the chassis at the core location. Do you have any recommendations on 10G switches? Yes I realize the math is a little backwards as this is all hypothetical at this point. We would provision each ONT as a shared 1Gbps offering similar to Google Fiber. We know there will be a large amount of oversubscription as no one really uses a full Gbps or anywhere close to it. I just wanted to stress the point that carrier redundancy at the 10G level would be a requirement for the core router, and it should of course have 10G links going to the uplinks on the aggregation switch. I think the Cisco ASR9k and the Juniper MX line will do well. Not sure if there are any others that can handle this level of traffic on the BGP side? So we have a 10G aggregation switch to aggregate the chassis uplink connections, and a 10G router BGP capable router. I really liked your article on DHCP vs PPP for DSL networks. We definitely agree the way to go is with a DHCP server. A couple of items your article left as big questions: 1. The article mentioned DHCP doesn't do the other part of what PPPoE or PPPoA does, which is generate RADIUS accounting records that give us the bandwidth information. So that’s one of the main challenges in switching to a DHCP based system. So, how do you handle bandwidth tracking in an all DHCP environment then? If I want to track how many GB a customer used last month, or the average Mbps used how do you do so? 2. I liked your option 82 example, and that works well for DSL networks where one port is tied to one customer. But how does option 82 work when you have multiple customers hanging off a GPON port? What does GPON use a subport identifier? 3. You mentioned, DHCP is again, not a authentication protocol. So what handles authentication then if only DHCP is used, and there are no usernames and passwords? I guess for DSL networks you can enable or disable the port to allow or disallow access, and Option 82 for identification? I assume you wouldn't want to shut off the GPON OLT port if one customer wasn't paying their bill as it would affect the other customers on that port. I assume access vendors allow you to shut down the sub port or ONT in this situation for GPON? Still that seems messy having to login to a shelf or EMS system or API to an EMS system especially if you have multiple access vendors in a network. Is there a way to do authentication with DHCP? What about open networks like wifi where anyone can connect, so you don't have the ability to turn of the port or disable the end device? 4. I don't think anyone is buying a BRAS anymore, but looks like Cisco, Juniper, and ALU have what they call BGN, Broadband Subscriber Management, and other similar software. How are these different from BRAS functionality? So it looks like there are open source and commercial solutions for DHCP and DNS. Some providers like Infloblox seems to integrate all these into one. So if we have a core router that speaks BGP, a 10G aggregation switch to aggregate the the chassis, and a device like Infloblox or the other commercial solutions you mentioned that do DHCP/DNS, is there anything else that is needed besides the access gear already mentioned in the assumptions? Are these large and expensive commercial BGN/Broadband Subscriber management products a thing of the past or still very relevant in todays environment? On Thu, Jul 31, 2014 at 8:54 AM, Scott Helms <khelms@zcorum.com> wrote:
"What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive?"
Definitely aggregate into a switch first unless you want to run a Layer 3 switch as your router, which I don't recommend.
"Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch?"
Your math is a little backwards, its very unlikely that you're going to have 40 Gbps of Internet (or other interconnection) for the router to actually have to process. What is the average provisioned speed for each of the 10k PON ports? What over subscription rate are you planning for? What, if anything, will you be carrying on net, ie bandwidth consumption that won't come from or go to the public Internet? Your own video, voice, or other service are examples of things that are often on net. In any case you're probably in the ASR family with Cisco and I can't remember the equivalent from Juniper.
How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option?
CGN is the option of last resort IMO, but you may have to consider it. A better approach is to see if your backbone providers will agree to give some blocks that you can announce and use those blocks for dynamic customers only. Your static IP customers should come from your direct ARIN allotment in case you need to choose a new backbone provider, which is extremely common over time.
"Dynamic IP addresses? DHCP?"
DHCP with enforcement from the shelves. All the major OLT vendors support doing this so that a customer can only use the address assigned to him by DHCP and nothing else, except for those customers that you choose to hard code. Make most of your "static" customers actually DHCP reservations and only hard code those that you must.
"How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts?"
DHCP, with Option 82 logging for the circuit ID is the better path than a BRAS (PPPoE) these days. Here's a paper we put together on that topic a while back:
http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPP...
Depending on your OLT vendor you can either use their built in port isolation or QinQ tagging, both are reliable and scalable, just ask your vendor which is the best option for your specific gear.
"If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else?"
I wouldn't have those two services connected personally, though there are hooks for some of the CGN boxes to talk to DHCP servers. I would hope you can get another 6k addresses and avoid the need for CGN altogether. Having said that, have you tested your OLTs and ONTs for IPv6 interoperability? If they don't handle it well then you're going to have to think about alternatives like 6RD (http://en.wikipedia.org/wiki/IPv6_rapid_deployment)
For DHCP at your scale you can run ISC DHCP ( http://www.isc.org/downloads/dhcp/) which is the most common open source DHCP daemon if you someone who can take care of a Linux server, parse the Option 82 information for logging, and handle the configuration of the DHCP daemon itself. Otherwise you might want to look at commercial products designed for the service provider market like Incongito's BCC and Cisco's BAC (CNR replacement)
http://www.incognito.com/products/broadband-command-center/
http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-acc...
"What about DNS? Is a firewall needed in the core? What else is needed?"
There are two kinds of DNS, caching (recursive) and authoritative. The first is what your customers will use to resolve things on the Internet and the second is used to provide caching name servers on the Internet with information about domains you control (are authoritative for). The first needs good performance, availability, and scalability since your customers will use your caching name servers constantly. Most people can run BIND at your scale, again if you have someone with Linux experience, but there are other alternatives. PowerDNS has both caching and authoritative modules and there are some commercial offerings out there both as cloud hosting and local deployments. Your backbone provider will also often have caching name servers your customers can use, but the quality varies quite a bit. You can also, especially at first, leverage some of the free offerings like Google's DNS. I don't recommend firewalls for service provider networks, but you should make sure your gear can run (and is configured to do so) BCP 38.
Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------
On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor@gmail.com> wrote:
If a new operator or city is building a greenfield access network from the ground up, what software and hardware is needed in the core network to provide and manage residential and business internet services similar to the likes of AT&T, Comcast, and Google Fiber? Television and Telephone services are not to be considered only internet.
Assume hypothetically the operator already has the following in place: 10 GPON OLTs Chassis from an access vendor in 10 POPs around town (each POP has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. Dark fiber going from the POP locations back to the core location Assume a 32:1 way split, and each OLT chassis has enough ports populated to serve the area. 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged mode. Assume you are building a network designed to serve 10,000 subs All the fiber splitters, ducts, fiber, etc connecting the OLTs to the ONTs is already in place ASN from ARIN /20 of IPv4 space and /32 of IPv6 space from ARIN 4 burstable 10G internet connections from 4 tier 1 internet providers
Questions are:
What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive? Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch? How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? Dynamic IP addresses? DHCP? How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts? If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else? What about DNS? Is a firewall needed in the core? What else is needed?
Is there a guide out there somewhere? I know many cities are looking at building their own network, and have similar questions. Access vendors are willing to sell gear all day long, but then they leave it up to the operator/city to answer these harder questions.
How would you build a access network from the ground up if you had the resources and time to do so? Would you even use GPON? Even if GPON was not used and another access technology like AE, VDSL2, or wireless was used I think many of these questions would be the same.
On Thu, Jul 31, 2014 at 12:07 PM, Colton Conor <colton.conor@gmail.com> wrote:
Scott,
Thanks for the long post.
We will use a layer 2 10G aggregation switch then to aggregate the chassis at the core location. Do you have any recommendations on 10G switches?
Not really, just stick with one of the major brands and you _should_ be fine.
Yes I realize the math is a little backwards as this is all hypothetical at this point. We would provision each ONT as a shared 1Gbps offering similar to Google Fiber. We know there will be a large amount of oversubscription as no one really uses a full Gbps or anywhere close to it. I just wanted to stress the point that carrier redundancy at the 10G level would be a requirement for the core router, and it should of course have 10G links going to the uplinks on the aggregation switch. I think the Cisco ASR9k and the Juniper MX line will do well. Not sure if there are any others that can handle this level of traffic on the BGP side?
That's reasonable IMO and yes, I think the Juniper MX can handle that as well as some other functions for you related to subscriber management if you want. The MX line has a full BRAS set of capabilities built into it that it inherited from the older ERX line, but they're commonly deployed without using any of them of as well.
So we have a 10G aggregation switch to aggregate the chassis uplink connections, and a 10G router BGP capable router.
I really liked your article on DHCP vs PPP for DSL networks. We definitely agree the way to go is with a DHCP server. A couple of items your article left as big questions:
1. The article mentioned DHCP doesn't do the other part of what PPPoE or PPPoA does, which is generate RADIUS accounting records that give us the bandwidth information. So that’s one of the main challenges in switching to a DHCP based system. So, how do you handle bandwidth tracking in an all DHCP environment then? If I want to track how many GB a customer used last month, or the average Mbps used how do you do so?
There are a few ways to get at that problem. You can use Netflow/IPFIX collection to gather the usage from your router, accepting that you're only going to get information on layer 3 traffic, which generally isn't a problem. You will need to match the IPs up against your Option 82 parsing which will give you the circuit ID, IP address, and WAN MAC of the ONT. You can also poll your shelves via SNMP, CLI, TL-1, and/or Netconf to collect the data and put it into a database in much the same way you can use RADIUS accounting data.
2. I liked your option 82 example, and that works well for DSL networks where one port is tied to one customer. But how does option 82 work when you have multiple customers hanging off a GPON port? What does GPON use a subport identifier?
Yep, the different vendors implement it slightly differently, usually the ONT MAC/serial will be included or the ONT ID will be included. Talk with your vendor, all the major OLT vendors are very familiar with Option 82 and in many cases they can tailor what their boxes send to make it easier for you.
3. You mentioned, DHCP is again, not a authentication protocol. So what handles authentication then if only DHCP is used, and there are no usernames and passwords? I guess for DSL networks you can enable or disable the port to allow or disallow access, and Option 82 for identification? I assume you wouldn't want to shut off the GPON OLT port if one customer wasn't paying their bill as it would affect the other customers on that port. I assume access vendors allow you to shut down the sub port or ONT in this situation for GPON? Still that seems messy having to login to a shelf or EMS system or API to an EMS system especially if you have multiple access vendors in a network. Is there a way to do authentication with DHCP? What about open networks like wifi where anyone can connect, so you don't have the ability to turn of the port or disable the end device? 4. I don't think anyone is buying a BRAS anymore, but looks like Cisco, Juniper, and ALU have what they call BGN, Broadband Subscriber Management, and other similar software. How are these different from BRAS functionality?
First, if you can manage it turn on DOCSIS provisioning of your GPON network. AFAIK only Calix has announced this functionality, but I expect the others to follow suit now that there is an official effort at CableLabs to allow that. http://www.lightreading.com/cable-video/docsis/calix-launches-docsis-provisi... The notion of managing ports and profiles via (an ever changing) shelf API is one of the main reasons that telco billing systems cost so much compared to cable billing systems. If you can't swing DPoG then you're kind of stuck, either you can implement the API your vendor supplies with your billing system, manage the profile assignment manually (yuck), or just provision everyone with the same speed (only works for data only deployments), or go down the older route of putting in a BRAS and making sure the ONTs you're deploying have a PPPoE client embedded in them. Having to deploy an external router for each customer, which I've seen some operators do, makes your install costs higher and makes troubleshooting harder.
So it looks like there are open source and commercial solutions for DHCP and DNS. Some providers like Infloblox seems to integrate all these into one.
So if we have a core router that speaks BGP, a 10G aggregation switch to aggregate the the chassis, and a device like Infloblox or the other commercial solutions you mentioned that do DHCP/DNS, is there anything else that is needed besides the access gear already mentioned in the assumptions? Are these large and expensive commercial BGN/Broadband Subscriber management products a thing of the past or still very relevant in todays environment?
They're not very relevant, once the the OLT vendors realized they could snoop the DHCP session and enforce what the server provided the need for subscriber management pieces really dropped. You've listed the bare essentials for a functional network, there are lots of things that are helpful or useful, but what you have is functional. Having said all of that, there are some relatively unobtrusive ways to have some level of authentication, I just don't think they're very valuable.
On Thu, Jul 31, 2014 at 8:54 AM, Scott Helms <khelms@zcorum.com> wrote:
"What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive?"
Definitely aggregate into a switch first unless you want to run a Layer 3 switch as your router, which I don't recommend.
"Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch?"
Your math is a little backwards, its very unlikely that you're going to have 40 Gbps of Internet (or other interconnection) for the router to actually have to process. What is the average provisioned speed for each of the 10k PON ports? What over subscription rate are you planning for? What, if anything, will you be carrying on net, ie bandwidth consumption that won't come from or go to the public Internet? Your own video, voice, or other service are examples of things that are often on net. In any case you're probably in the ASR family with Cisco and I can't remember the equivalent from Juniper.
How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option?
CGN is the option of last resort IMO, but you may have to consider it. A better approach is to see if your backbone providers will agree to give some blocks that you can announce and use those blocks for dynamic customers only. Your static IP customers should come from your direct ARIN allotment in case you need to choose a new backbone provider, which is extremely common over time.
"Dynamic IP addresses? DHCP?"
DHCP with enforcement from the shelves. All the major OLT vendors support doing this so that a customer can only use the address assigned to him by DHCP and nothing else, except for those customers that you choose to hard code. Make most of your "static" customers actually DHCP reservations and only hard code those that you must.
"How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts?"
DHCP, with Option 82 logging for the circuit ID is the better path than a BRAS (PPPoE) these days. Here's a paper we put together on that topic a while back:
http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPP...
Depending on your OLT vendor you can either use their built in port isolation or QinQ tagging, both are reliable and scalable, just ask your vendor which is the best option for your specific gear.
"If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else?"
I wouldn't have those two services connected personally, though there are hooks for some of the CGN boxes to talk to DHCP servers. I would hope you can get another 6k addresses and avoid the need for CGN altogether. Having said that, have you tested your OLTs and ONTs for IPv6 interoperability? If they don't handle it well then you're going to have to think about alternatives like 6RD (http://en.wikipedia.org/wiki/IPv6_rapid_deployment )
For DHCP at your scale you can run ISC DHCP ( http://www.isc.org/downloads/dhcp/) which is the most common open source DHCP daemon if you someone who can take care of a Linux server, parse the Option 82 information for logging, and handle the configuration of the DHCP daemon itself. Otherwise you might want to look at commercial products designed for the service provider market like Incongito's BCC and Cisco's BAC (CNR replacement)
http://www.incognito.com/products/broadband-command-center/
http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-acc...
"What about DNS? Is a firewall needed in the core? What else is needed?"
There are two kinds of DNS, caching (recursive) and authoritative. The first is what your customers will use to resolve things on the Internet and the second is used to provide caching name servers on the Internet with information about domains you control (are authoritative for). The first needs good performance, availability, and scalability since your customers will use your caching name servers constantly. Most people can run BIND at your scale, again if you have someone with Linux experience, but there are other alternatives. PowerDNS has both caching and authoritative modules and there are some commercial offerings out there both as cloud hosting and local deployments. Your backbone provider will also often have caching name servers your customers can use, but the quality varies quite a bit. You can also, especially at first, leverage some of the free offerings like Google's DNS. I don't recommend firewalls for service provider networks, but you should make sure your gear can run (and is configured to do so) BCP 38.
Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------
On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor@gmail.com> wrote:
If a new operator or city is building a greenfield access network from the ground up, what software and hardware is needed in the core network to provide and manage residential and business internet services similar to the likes of AT&T, Comcast, and Google Fiber? Television and Telephone services are not to be considered only internet.
Assume hypothetically the operator already has the following in place: 10 GPON OLTs Chassis from an access vendor in 10 POPs around town (each POP has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. Dark fiber going from the POP locations back to the core location Assume a 32:1 way split, and each OLT chassis has enough ports populated to serve the area. 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged mode. Assume you are building a network designed to serve 10,000 subs All the fiber splitters, ducts, fiber, etc connecting the OLTs to the ONTs is already in place ASN from ARIN /20 of IPv4 space and /32 of IPv6 space from ARIN 4 burstable 10G internet connections from 4 tier 1 internet providers
Questions are:
What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive? Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch? How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? Dynamic IP addresses? DHCP? How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts? If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else? What about DNS? Is a firewall needed in the core? What else is needed?
Is there a guide out there somewhere? I know many cities are looking at building their own network, and have similar questions. Access vendors are willing to sell gear all day long, but then they leave it up to the operator/city to answer these harder questions.
How would you build a access network from the ground up if you had the resources and time to do so? Would you even use GPON? Even if GPON was not used and another access technology like AE, VDSL2, or wireless was used I think many of these questions would be the same.
I have read both the Juniper MX and Cisco ASR9K do support this advanced BRAS functionality, what Juniper calls Subscriber Feature Management and what Cisco calls BGN. These software functions run on the router itself, however the are not free or included with the base chassis. To enable these you must pay a hefty fee. So you are saying that these advanced feature packs that the largest networking markers in the world sell are really not needed anymore due to advancements on the access vendor side of the house?
From the reading I have done about these solutions, it is kind of like PPPoE with a radius setup, but instead DHCP option 82 with a radius setup. These routers are also capable of running a local DHCP server, but I am not sure if that is recommended.
The DPoE DOCSIS provisioning of your GPON network is interesting, but is that really relevant for a new provider if they don't have cable CMTS systems already deployed. Sure, it makes sense for the cable compaines who have already bought billing systems and are used to living in a DOCSIS world. But if you were starting fresh from the group up are you recommending we look at GPON providers like Calix because they support DPoE so we can buy DOCIS billing systems? That is an interesting concept. On Thu, Jul 31, 2014 at 12:59 PM, Scott Helms <khelms@zcorum.com> wrote:
On Thu, Jul 31, 2014 at 12:07 PM, Colton Conor <colton.conor@gmail.com> wrote:
Scott,
Thanks for the long post.
We will use a layer 2 10G aggregation switch then to aggregate the chassis at the core location. Do you have any recommendations on 10G switches?
Not really, just stick with one of the major brands and you _should_ be fine.
Yes I realize the math is a little backwards as this is all hypothetical at this point. We would provision each ONT as a shared 1Gbps offering similar to Google Fiber. We know there will be a large amount of oversubscription as no one really uses a full Gbps or anywhere close to it. I just wanted to stress the point that carrier redundancy at the 10G level would be a requirement for the core router, and it should of course have 10G links going to the uplinks on the aggregation switch. I think the Cisco ASR9k and the Juniper MX line will do well. Not sure if there are any others that can handle this level of traffic on the BGP side?
That's reasonable IMO and yes, I think the Juniper MX can handle that as well as some other functions for you related to subscriber management if you want. The MX line has a full BRAS set of capabilities built into it that it inherited from the older ERX line, but they're commonly deployed without using any of them of as well.
So we have a 10G aggregation switch to aggregate the chassis uplink connections, and a 10G router BGP capable router.
I really liked your article on DHCP vs PPP for DSL networks. We definitely agree the way to go is with a DHCP server. A couple of items your article left as big questions:
1. The article mentioned DHCP doesn't do the other part of what PPPoE or PPPoA does, which is generate RADIUS accounting records that give us the bandwidth information. So that’s one of the main challenges in switching to a DHCP based system. So, how do you handle bandwidth tracking in an all DHCP environment then? If I want to track how many GB a customer used last month, or the average Mbps used how do you do so?
There are a few ways to get at that problem. You can use Netflow/IPFIX collection to gather the usage from your router, accepting that you're only going to get information on layer 3 traffic, which generally isn't a problem. You will need to match the IPs up against your Option 82 parsing which will give you the circuit ID, IP address, and WAN MAC of the ONT. You can also poll your shelves via SNMP, CLI, TL-1, and/or Netconf to collect the data and put it into a database in much the same way you can use RADIUS accounting data.
2. I liked your option 82 example, and that works well for DSL networks where one port is tied to one customer. But how does option 82 work when you have multiple customers hanging off a GPON port? What does GPON use a subport identifier?
Yep, the different vendors implement it slightly differently, usually the ONT MAC/serial will be included or the ONT ID will be included. Talk with your vendor, all the major OLT vendors are very familiar with Option 82 and in many cases they can tailor what their boxes send to make it easier for you.
3. You mentioned, DHCP is again, not a authentication protocol. So what handles authentication then if only DHCP is used, and there are no usernames and passwords? I guess for DSL networks you can enable or disable the port to allow or disallow access, and Option 82 for identification? I assume you wouldn't want to shut off the GPON OLT port if one customer wasn't paying their bill as it would affect the other customers on that port. I assume access vendors allow you to shut down the sub port or ONT in this situation for GPON? Still that seems messy having to login to a shelf or EMS system or API to an EMS system especially if you have multiple access vendors in a network. Is there a way to do authentication with DHCP? What about open networks like wifi where anyone can connect, so you don't have the ability to turn of the port or disable the end device? 4. I don't think anyone is buying a BRAS anymore, but looks like Cisco, Juniper, and ALU have what they call BGN, Broadband Subscriber Management, and other similar software. How are these different from BRAS functionality?
First, if you can manage it turn on DOCSIS provisioning of your GPON network. AFAIK only Calix has announced this functionality, but I expect the others to follow suit now that there is an official effort at CableLabs to allow that.
http://www.lightreading.com/cable-video/docsis/calix-launches-docsis-provisi...
The notion of managing ports and profiles via (an ever changing) shelf API is one of the main reasons that telco billing systems cost so much compared to cable billing systems. If you can't swing DPoG then you're kind of stuck, either you can implement the API your vendor supplies with your billing system, manage the profile assignment manually (yuck), or just provision everyone with the same speed (only works for data only deployments), or go down the older route of putting in a BRAS and making sure the ONTs you're deploying have a PPPoE client embedded in them. Having to deploy an external router for each customer, which I've seen some operators do, makes your install costs higher and makes troubleshooting harder.
So it looks like there are open source and commercial solutions for DHCP and DNS. Some providers like Infloblox seems to integrate all these into one.
So if we have a core router that speaks BGP, a 10G aggregation switch to aggregate the the chassis, and a device like Infloblox or the other commercial solutions you mentioned that do DHCP/DNS, is there anything else that is needed besides the access gear already mentioned in the assumptions? Are these large and expensive commercial BGN/Broadband Subscriber management products a thing of the past or still very relevant in todays environment?
They're not very relevant, once the the OLT vendors realized they could snoop the DHCP session and enforce what the server provided the need for subscriber management pieces really dropped. You've listed the bare essentials for a functional network, there are lots of things that are helpful or useful, but what you have is functional. Having said all of that, there are some relatively unobtrusive ways to have some level of authentication, I just don't think they're very valuable.
On Thu, Jul 31, 2014 at 8:54 AM, Scott Helms <khelms@zcorum.com> wrote:
"What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive?"
Definitely aggregate into a switch first unless you want to run a Layer 3 switch as your router, which I don't recommend.
"Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch?"
Your math is a little backwards, its very unlikely that you're going to have 40 Gbps of Internet (or other interconnection) for the router to actually have to process. What is the average provisioned speed for each of the 10k PON ports? What over subscription rate are you planning for? What, if anything, will you be carrying on net, ie bandwidth consumption that won't come from or go to the public Internet? Your own video, voice, or other service are examples of things that are often on net. In any case you're probably in the ASR family with Cisco and I can't remember the equivalent from Juniper.
How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option?
CGN is the option of last resort IMO, but you may have to consider it. A better approach is to see if your backbone providers will agree to give some blocks that you can announce and use those blocks for dynamic customers only. Your static IP customers should come from your direct ARIN allotment in case you need to choose a new backbone provider, which is extremely common over time.
"Dynamic IP addresses? DHCP?"
DHCP with enforcement from the shelves. All the major OLT vendors support doing this so that a customer can only use the address assigned to him by DHCP and nothing else, except for those customers that you choose to hard code. Make most of your "static" customers actually DHCP reservations and only hard code those that you must.
"How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts?"
DHCP, with Option 82 logging for the circuit ID is the better path than a BRAS (PPPoE) these days. Here's a paper we put together on that topic a while back:
http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPP...
Depending on your OLT vendor you can either use their built in port isolation or QinQ tagging, both are reliable and scalable, just ask your vendor which is the best option for your specific gear.
"If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else?"
I wouldn't have those two services connected personally, though there are hooks for some of the CGN boxes to talk to DHCP servers. I would hope you can get another 6k addresses and avoid the need for CGN altogether. Having said that, have you tested your OLTs and ONTs for IPv6 interoperability? If they don't handle it well then you're going to have to think about alternatives like 6RD ( http://en.wikipedia.org/wiki/IPv6_rapid_deployment)
For DHCP at your scale you can run ISC DHCP ( http://www.isc.org/downloads/dhcp/) which is the most common open source DHCP daemon if you someone who can take care of a Linux server, parse the Option 82 information for logging, and handle the configuration of the DHCP daemon itself. Otherwise you might want to look at commercial products designed for the service provider market like Incongito's BCC and Cisco's BAC (CNR replacement)
http://www.incognito.com/products/broadband-command-center/
http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-acc...
"What about DNS? Is a firewall needed in the core? What else is needed?"
There are two kinds of DNS, caching (recursive) and authoritative. The first is what your customers will use to resolve things on the Internet and the second is used to provide caching name servers on the Internet with information about domains you control (are authoritative for). The first needs good performance, availability, and scalability since your customers will use your caching name servers constantly. Most people can run BIND at your scale, again if you have someone with Linux experience, but there are other alternatives. PowerDNS has both caching and authoritative modules and there are some commercial offerings out there both as cloud hosting and local deployments. Your backbone provider will also often have caching name servers your customers can use, but the quality varies quite a bit. You can also, especially at first, leverage some of the free offerings like Google's DNS. I don't recommend firewalls for service provider networks, but you should make sure your gear can run (and is configured to do so) BCP 38.
Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------
On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor@gmail.com> wrote:
If a new operator or city is building a greenfield access network from the ground up, what software and hardware is needed in the core network to provide and manage residential and business internet services similar to the likes of AT&T, Comcast, and Google Fiber? Television and Telephone services are not to be considered only internet.
Assume hypothetically the operator already has the following in place: 10 GPON OLTs Chassis from an access vendor in 10 POPs around town (each POP has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. Dark fiber going from the POP locations back to the core location Assume a 32:1 way split, and each OLT chassis has enough ports populated to serve the area. 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged mode. Assume you are building a network designed to serve 10,000 subs All the fiber splitters, ducts, fiber, etc connecting the OLTs to the ONTs is already in place ASN from ARIN /20 of IPv4 space and /32 of IPv6 space from ARIN 4 burstable 10G internet connections from 4 tier 1 internet providers
Questions are:
What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive? Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch? How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? Dynamic IP addresses? DHCP? How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts? If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else? What about DNS? Is a firewall needed in the core? What else is needed?
Is there a guide out there somewhere? I know many cities are looking at building their own network, and have similar questions. Access vendors are willing to sell gear all day long, but then they leave it up to the operator/city to answer these harder questions.
How would you build a access network from the ground up if you had the resources and time to do so? Would you even use GPON? Even if GPON was not used and another access technology like AE, VDSL2, or wireless was used I think many of these questions would be the same.
On Thu, Jul 31, 2014 at 2:25 PM, Colton Conor <colton.conor@gmail.com> wrote:
I have read both the Juniper MX and Cisco ASR9K do support this advanced BRAS functionality, what Juniper calls Subscriber Feature Management and what Cisco calls BGN. These software functions run on the router itself, however the are not free or included with the base chassis. To enable these you must pay a hefty fee. So you are saying that these advanced feature packs that the largest networking markers in the world sell are really not needed anymore due to advancements on the access vendor side of the house? From the reading I have done about these solutions, it is kind of like PPPoE with a radius setup, but instead DHCP option 82 with a radius setup. These routers are also capable of running a local DHCP server, but I am not sure if that is recommended.
Yeah, that's it in a nutshell. There are several options, like matching on Option 82 or redirecting to a web page, but at the end of the day I don't believe they're worth the time or expense. Keep in mind that earlier in my career I was a huge proponent of BRAS architecture and I've put in everything from Nortel Shasta's to Lucent Terminators, to Redbacks, to Juniper ERXs and several more models I can't remember. Once you get past the whole lack of authentication, which was never very secure, and understand that you can depend on Option 82 to tell you where a session came from physically the rest is just finding away to count and account for bits. Oh, and I never recommend running the DHCP daemon on a piece of networking gear for service providers.
The DPoE DOCSIS provisioning of your GPON network is interesting, but is that really relevant for a new provider if they don't have cable CMTS systems already deployed. Sure, it makes sense for the cable compaines who have already bought billing systems and are used to living in a DOCSIS world. But if you were starting fresh from the group up are you recommending we look at GPON providers like Calix because they support DPoE so we can buy DOCIS billing systems? That is an interesting concept.
I'd strongly recommend finding a vendor that says they will support it on the shelves you're going to buy even if they don't today. Even if you're not doing DOCSIS cable modems and don't ever plan to the provisioning paradigm (DHCP, TFTP, ToD) is much simpler than the proprietary north bound (usually SOAP) API that direct integration requires. You can even build your own provisioning system with a little scripting and there are many more commercial options than there are for direct integration to the shelves.
On Thu, Jul 31, 2014 at 12:59 PM, Scott Helms <khelms@zcorum.com> wrote:
On Thu, Jul 31, 2014 at 12:07 PM, Colton Conor <colton.conor@gmail.com> wrote:
Scott,
Thanks for the long post.
We will use a layer 2 10G aggregation switch then to aggregate the chassis at the core location. Do you have any recommendations on 10G switches?
Not really, just stick with one of the major brands and you _should_ be fine.
Yes I realize the math is a little backwards as this is all hypothetical at this point. We would provision each ONT as a shared 1Gbps offering similar to Google Fiber. We know there will be a large amount of oversubscription as no one really uses a full Gbps or anywhere close to it. I just wanted to stress the point that carrier redundancy at the 10G level would be a requirement for the core router, and it should of course have 10G links going to the uplinks on the aggregation switch. I think the Cisco ASR9k and the Juniper MX line will do well. Not sure if there are any others that can handle this level of traffic on the BGP side?
That's reasonable IMO and yes, I think the Juniper MX can handle that as well as some other functions for you related to subscriber management if you want. The MX line has a full BRAS set of capabilities built into it that it inherited from the older ERX line, but they're commonly deployed without using any of them of as well.
So we have a 10G aggregation switch to aggregate the chassis uplink connections, and a 10G router BGP capable router.
I really liked your article on DHCP vs PPP for DSL networks. We definitely agree the way to go is with a DHCP server. A couple of items your article left as big questions:
1. The article mentioned DHCP doesn't do the other part of what PPPoE or PPPoA does, which is generate RADIUS accounting records that give us the bandwidth information. So that’s one of the main challenges in switching to a DHCP based system. So, how do you handle bandwidth tracking in an all DHCP environment then? If I want to track how many GB a customer used last month, or the average Mbps used how do you do so?
There are a few ways to get at that problem. You can use Netflow/IPFIX collection to gather the usage from your router, accepting that you're only going to get information on layer 3 traffic, which generally isn't a problem. You will need to match the IPs up against your Option 82 parsing which will give you the circuit ID, IP address, and WAN MAC of the ONT. You can also poll your shelves via SNMP, CLI, TL-1, and/or Netconf to collect the data and put it into a database in much the same way you can use RADIUS accounting data.
2. I liked your option 82 example, and that works well for DSL networks where one port is tied to one customer. But how does option 82 work when you have multiple customers hanging off a GPON port? What does GPON use a subport identifier?
Yep, the different vendors implement it slightly differently, usually the ONT MAC/serial will be included or the ONT ID will be included. Talk with your vendor, all the major OLT vendors are very familiar with Option 82 and in many cases they can tailor what their boxes send to make it easier for you.
3. You mentioned, DHCP is again, not a authentication protocol. So what handles authentication then if only DHCP is used, and there are no usernames and passwords? I guess for DSL networks you can enable or disable the port to allow or disallow access, and Option 82 for identification? I assume you wouldn't want to shut off the GPON OLT port if one customer wasn't paying their bill as it would affect the other customers on that port. I assume access vendors allow you to shut down the sub port or ONT in this situation for GPON? Still that seems messy having to login to a shelf or EMS system or API to an EMS system especially if you have multiple access vendors in a network. Is there a way to do authentication with DHCP? What about open networks like wifi where anyone can connect, so you don't have the ability to turn of the port or disable the end device? 4. I don't think anyone is buying a BRAS anymore, but looks like Cisco, Juniper, and ALU have what they call BGN, Broadband Subscriber Management, and other similar software. How are these different from BRAS functionality?
First, if you can manage it turn on DOCSIS provisioning of your GPON network. AFAIK only Calix has announced this functionality, but I expect the others to follow suit now that there is an official effort at CableLabs to allow that.
http://www.lightreading.com/cable-video/docsis/calix-launches-docsis-provisi...
The notion of managing ports and profiles via (an ever changing) shelf API is one of the main reasons that telco billing systems cost so much compared to cable billing systems. If you can't swing DPoG then you're kind of stuck, either you can implement the API your vendor supplies with your billing system, manage the profile assignment manually (yuck), or just provision everyone with the same speed (only works for data only deployments), or go down the older route of putting in a BRAS and making sure the ONTs you're deploying have a PPPoE client embedded in them. Having to deploy an external router for each customer, which I've seen some operators do, makes your install costs higher and makes troubleshooting harder.
So it looks like there are open source and commercial solutions for DHCP and DNS. Some providers like Infloblox seems to integrate all these into one.
So if we have a core router that speaks BGP, a 10G aggregation switch to aggregate the the chassis, and a device like Infloblox or the other commercial solutions you mentioned that do DHCP/DNS, is there anything else that is needed besides the access gear already mentioned in the assumptions? Are these large and expensive commercial BGN/Broadband Subscriber management products a thing of the past or still very relevant in todays environment?
They're not very relevant, once the the OLT vendors realized they could snoop the DHCP session and enforce what the server provided the need for subscriber management pieces really dropped. You've listed the bare essentials for a functional network, there are lots of things that are helpful or useful, but what you have is functional. Having said all of that, there are some relatively unobtrusive ways to have some level of authentication, I just don't think they're very valuable.
On Thu, Jul 31, 2014 at 8:54 AM, Scott Helms <khelms@zcorum.com> wrote:
"What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive?"
Definitely aggregate into a switch first unless you want to run a Layer 3 switch as your router, which I don't recommend.
"Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch?"
Your math is a little backwards, its very unlikely that you're going to have 40 Gbps of Internet (or other interconnection) for the router to actually have to process. What is the average provisioned speed for each of the 10k PON ports? What over subscription rate are you planning for? What, if anything, will you be carrying on net, ie bandwidth consumption that won't come from or go to the public Internet? Your own video, voice, or other service are examples of things that are often on net. In any case you're probably in the ASR family with Cisco and I can't remember the equivalent from Juniper.
How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option?
CGN is the option of last resort IMO, but you may have to consider it. A better approach is to see if your backbone providers will agree to give some blocks that you can announce and use those blocks for dynamic customers only. Your static IP customers should come from your direct ARIN allotment in case you need to choose a new backbone provider, which is extremely common over time.
"Dynamic IP addresses? DHCP?"
DHCP with enforcement from the shelves. All the major OLT vendors support doing this so that a customer can only use the address assigned to him by DHCP and nothing else, except for those customers that you choose to hard code. Make most of your "static" customers actually DHCP reservations and only hard code those that you must.
"How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts?"
DHCP, with Option 82 logging for the circuit ID is the better path than a BRAS (PPPoE) these days. Here's a paper we put together on that topic a while back:
http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPP...
Depending on your OLT vendor you can either use their built in port isolation or QinQ tagging, both are reliable and scalable, just ask your vendor which is the best option for your specific gear.
"If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else?"
I wouldn't have those two services connected personally, though there are hooks for some of the CGN boxes to talk to DHCP servers. I would hope you can get another 6k addresses and avoid the need for CGN altogether. Having said that, have you tested your OLTs and ONTs for IPv6 interoperability? If they don't handle it well then you're going to have to think about alternatives like 6RD ( http://en.wikipedia.org/wiki/IPv6_rapid_deployment)
For DHCP at your scale you can run ISC DHCP ( http://www.isc.org/downloads/dhcp/) which is the most common open source DHCP daemon if you someone who can take care of a Linux server, parse the Option 82 information for logging, and handle the configuration of the DHCP daemon itself. Otherwise you might want to look at commercial products designed for the service provider market like Incongito's BCC and Cisco's BAC (CNR replacement)
http://www.incognito.com/products/broadband-command-center/
http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-acc...
"What about DNS? Is a firewall needed in the core? What else is needed?"
There are two kinds of DNS, caching (recursive) and authoritative. The first is what your customers will use to resolve things on the Internet and the second is used to provide caching name servers on the Internet with information about domains you control (are authoritative for). The first needs good performance, availability, and scalability since your customers will use your caching name servers constantly. Most people can run BIND at your scale, again if you have someone with Linux experience, but there are other alternatives. PowerDNS has both caching and authoritative modules and there are some commercial offerings out there both as cloud hosting and local deployments. Your backbone provider will also often have caching name servers your customers can use, but the quality varies quite a bit. You can also, especially at first, leverage some of the free offerings like Google's DNS. I don't recommend firewalls for service provider networks, but you should make sure your gear can run (and is configured to do so) BCP 38.
Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------
On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor@gmail.com> wrote:
If a new operator or city is building a greenfield access network from the ground up, what software and hardware is needed in the core network to provide and manage residential and business internet services similar to the likes of AT&T, Comcast, and Google Fiber? Television and Telephone services are not to be considered only internet.
Assume hypothetically the operator already has the following in place: 10 GPON OLTs Chassis from an access vendor in 10 POPs around town (each POP has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. Dark fiber going from the POP locations back to the core location Assume a 32:1 way split, and each OLT chassis has enough ports populated to serve the area. 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged mode. Assume you are building a network designed to serve 10,000 subs All the fiber splitters, ducts, fiber, etc connecting the OLTs to the ONTs is already in place ASN from ARIN /20 of IPv4 space and /32 of IPv6 space from ARIN 4 burstable 10G internet connections from 4 tier 1 internet providers
Questions are:
What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive? Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch? How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? Dynamic IP addresses? DHCP? How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts? If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else? What about DNS? Is a firewall needed in the core? What else is needed?
Is there a guide out there somewhere? I know many cities are looking at building their own network, and have similar questions. Access vendors are willing to sell gear all day long, but then they leave it up to the operator/city to answer these harder questions.
How would you build a access network from the ground up if you had the resources and time to do so? Would you even use GPON? Even if GPON was not used and another access technology like AE, VDSL2, or wireless was used I think many of these questions would be the same.
Scott, Thank you for your input. What do you recommend for network segmentation? A VLAN per Chassis, a VLAN per service, or a VLAN per customer/port? When you say qinq VLANs are you referring to the CVLAN model? I am really interested to know how the largest providers, like Comcast, AT&T, and Verizon setup their residential networks. Little information is available on the internet besides the access platforms they use. On Thu, Jul 31, 2014 at 1:51 PM, Scott Helms <khelms@zcorum.com> wrote:
On Thu, Jul 31, 2014 at 2:25 PM, Colton Conor <colton.conor@gmail.com> wrote:
I have read both the Juniper MX and Cisco ASR9K do support this advanced BRAS functionality, what Juniper calls Subscriber Feature Management and what Cisco calls BGN. These software functions run on the router itself, however the are not free or included with the base chassis. To enable these you must pay a hefty fee. So you are saying that these advanced feature packs that the largest networking markers in the world sell are really not needed anymore due to advancements on the access vendor side of the house? From the reading I have done about these solutions, it is kind of like PPPoE with a radius setup, but instead DHCP option 82 with a radius setup. These routers are also capable of running a local DHCP server, but I am not sure if that is recommended.
Yeah, that's it in a nutshell. There are several options, like matching on Option 82 or redirecting to a web page, but at the end of the day I don't believe they're worth the time or expense. Keep in mind that earlier in my career I was a huge proponent of BRAS architecture and I've put in everything from Nortel Shasta's to Lucent Terminators, to Redbacks, to Juniper ERXs and several more models I can't remember. Once you get past the whole lack of authentication, which was never very secure, and understand that you can depend on Option 82 to tell you where a session came from physically the rest is just finding away to count and account for bits.
Oh, and I never recommend running the DHCP daemon on a piece of networking gear for service providers.
The DPoE DOCSIS provisioning of your GPON network is interesting, but is that really relevant for a new provider if they don't have cable CMTS systems already deployed. Sure, it makes sense for the cable compaines who have already bought billing systems and are used to living in a DOCSIS world. But if you were starting fresh from the group up are you recommending we look at GPON providers like Calix because they support DPoE so we can buy DOCIS billing systems? That is an interesting concept.
I'd strongly recommend finding a vendor that says they will support it on the shelves you're going to buy even if they don't today. Even if you're not doing DOCSIS cable modems and don't ever plan to the provisioning paradigm (DHCP, TFTP, ToD) is much simpler than the proprietary north bound (usually SOAP) API that direct integration requires. You can even build your own provisioning system with a little scripting and there are many more commercial options than there are for direct integration to the shelves.
On Thu, Jul 31, 2014 at 12:59 PM, Scott Helms <khelms@zcorum.com> wrote:
On Thu, Jul 31, 2014 at 12:07 PM, Colton Conor <colton.conor@gmail.com> wrote:
Scott,
Thanks for the long post.
We will use a layer 2 10G aggregation switch then to aggregate the chassis at the core location. Do you have any recommendations on 10G switches?
Not really, just stick with one of the major brands and you _should_ be fine.
Yes I realize the math is a little backwards as this is all hypothetical at this point. We would provision each ONT as a shared 1Gbps offering similar to Google Fiber. We know there will be a large amount of oversubscription as no one really uses a full Gbps or anywhere close to it. I just wanted to stress the point that carrier redundancy at the 10G level would be a requirement for the core router, and it should of course have 10G links going to the uplinks on the aggregation switch. I think the Cisco ASR9k and the Juniper MX line will do well. Not sure if there are any others that can handle this level of traffic on the BGP side?
That's reasonable IMO and yes, I think the Juniper MX can handle that as well as some other functions for you related to subscriber management if you want. The MX line has a full BRAS set of capabilities built into it that it inherited from the older ERX line, but they're commonly deployed without using any of them of as well.
So we have a 10G aggregation switch to aggregate the chassis uplink connections, and a 10G router BGP capable router.
I really liked your article on DHCP vs PPP for DSL networks. We definitely agree the way to go is with a DHCP server. A couple of items your article left as big questions:
1. The article mentioned DHCP doesn't do the other part of what PPPoE or PPPoA does, which is generate RADIUS accounting records that give us the bandwidth information. So that’s one of the main challenges in switching to a DHCP based system. So, how do you handle bandwidth tracking in an all DHCP environment then? If I want to track how many GB a customer used last month, or the average Mbps used how do you do so?
There are a few ways to get at that problem. You can use Netflow/IPFIX collection to gather the usage from your router, accepting that you're only going to get information on layer 3 traffic, which generally isn't a problem. You will need to match the IPs up against your Option 82 parsing which will give you the circuit ID, IP address, and WAN MAC of the ONT. You can also poll your shelves via SNMP, CLI, TL-1, and/or Netconf to collect the data and put it into a database in much the same way you can use RADIUS accounting data.
2. I liked your option 82 example, and that works well for DSL networks where one port is tied to one customer. But how does option 82 work when you have multiple customers hanging off a GPON port? What does GPON use a subport identifier?
Yep, the different vendors implement it slightly differently, usually the ONT MAC/serial will be included or the ONT ID will be included. Talk with your vendor, all the major OLT vendors are very familiar with Option 82 and in many cases they can tailor what their boxes send to make it easier for you.
3. You mentioned, DHCP is again, not a authentication protocol. So what handles authentication then if only DHCP is used, and there are no usernames and passwords? I guess for DSL networks you can enable or disable the port to allow or disallow access, and Option 82 for identification? I assume you wouldn't want to shut off the GPON OLT port if one customer wasn't paying their bill as it would affect the other customers on that port. I assume access vendors allow you to shut down the sub port or ONT in this situation for GPON? Still that seems messy having to login to a shelf or EMS system or API to an EMS system especially if you have multiple access vendors in a network. Is there a way to do authentication with DHCP? What about open networks like wifi where anyone can connect, so you don't have the ability to turn of the port or disable the end device? 4. I don't think anyone is buying a BRAS anymore, but looks like Cisco, Juniper, and ALU have what they call BGN, Broadband Subscriber Management, and other similar software. How are these different from BRAS functionality?
First, if you can manage it turn on DOCSIS provisioning of your GPON network. AFAIK only Calix has announced this functionality, but I expect the others to follow suit now that there is an official effort at CableLabs to allow that.
http://www.lightreading.com/cable-video/docsis/calix-launches-docsis-provisi...
The notion of managing ports and profiles via (an ever changing) shelf API is one of the main reasons that telco billing systems cost so much compared to cable billing systems. If you can't swing DPoG then you're kind of stuck, either you can implement the API your vendor supplies with your billing system, manage the profile assignment manually (yuck), or just provision everyone with the same speed (only works for data only deployments), or go down the older route of putting in a BRAS and making sure the ONTs you're deploying have a PPPoE client embedded in them. Having to deploy an external router for each customer, which I've seen some operators do, makes your install costs higher and makes troubleshooting harder.
So it looks like there are open source and commercial solutions for DHCP and DNS. Some providers like Infloblox seems to integrate all these into one.
So if we have a core router that speaks BGP, a 10G aggregation switch to aggregate the the chassis, and a device like Infloblox or the other commercial solutions you mentioned that do DHCP/DNS, is there anything else that is needed besides the access gear already mentioned in the assumptions? Are these large and expensive commercial BGN/Broadband Subscriber management products a thing of the past or still very relevant in todays environment?
They're not very relevant, once the the OLT vendors realized they could snoop the DHCP session and enforce what the server provided the need for subscriber management pieces really dropped. You've listed the bare essentials for a functional network, there are lots of things that are helpful or useful, but what you have is functional. Having said all of that, there are some relatively unobtrusive ways to have some level of authentication, I just don't think they're very valuable.
On Thu, Jul 31, 2014 at 8:54 AM, Scott Helms <khelms@zcorum.com> wrote:
"What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive?"
Definitely aggregate into a switch first unless you want to run a Layer 3 switch as your router, which I don't recommend.
"Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch?"
Your math is a little backwards, its very unlikely that you're going to have 40 Gbps of Internet (or other interconnection) for the router to actually have to process. What is the average provisioned speed for each of the 10k PON ports? What over subscription rate are you planning for? What, if anything, will you be carrying on net, ie bandwidth consumption that won't come from or go to the public Internet? Your own video, voice, or other service are examples of things that are often on net. In any case you're probably in the ASR family with Cisco and I can't remember the equivalent from Juniper.
How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option?
CGN is the option of last resort IMO, but you may have to consider it. A better approach is to see if your backbone providers will agree to give some blocks that you can announce and use those blocks for dynamic customers only. Your static IP customers should come from your direct ARIN allotment in case you need to choose a new backbone provider, which is extremely common over time.
"Dynamic IP addresses? DHCP?"
DHCP with enforcement from the shelves. All the major OLT vendors support doing this so that a customer can only use the address assigned to him by DHCP and nothing else, except for those customers that you choose to hard code. Make most of your "static" customers actually DHCP reservations and only hard code those that you must.
"How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts?"
DHCP, with Option 82 logging for the circuit ID is the better path than a BRAS (PPPoE) these days. Here's a paper we put together on that topic a while back:
http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPP...
Depending on your OLT vendor you can either use their built in port isolation or QinQ tagging, both are reliable and scalable, just ask your vendor which is the best option for your specific gear.
"If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else?"
I wouldn't have those two services connected personally, though there are hooks for some of the CGN boxes to talk to DHCP servers. I would hope you can get another 6k addresses and avoid the need for CGN altogether. Having said that, have you tested your OLTs and ONTs for IPv6 interoperability? If they don't handle it well then you're going to have to think about alternatives like 6RD ( http://en.wikipedia.org/wiki/IPv6_rapid_deployment)
For DHCP at your scale you can run ISC DHCP ( http://www.isc.org/downloads/dhcp/) which is the most common open source DHCP daemon if you someone who can take care of a Linux server, parse the Option 82 information for logging, and handle the configuration of the DHCP daemon itself. Otherwise you might want to look at commercial products designed for the service provider market like Incongito's BCC and Cisco's BAC (CNR replacement)
http://www.incognito.com/products/broadband-command-center/
http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-acc...
"What about DNS? Is a firewall needed in the core? What else is needed?"
There are two kinds of DNS, caching (recursive) and authoritative. The first is what your customers will use to resolve things on the Internet and the second is used to provide caching name servers on the Internet with information about domains you control (are authoritative for). The first needs good performance, availability, and scalability since your customers will use your caching name servers constantly. Most people can run BIND at your scale, again if you have someone with Linux experience, but there are other alternatives. PowerDNS has both caching and authoritative modules and there are some commercial offerings out there both as cloud hosting and local deployments. Your backbone provider will also often have caching name servers your customers can use, but the quality varies quite a bit. You can also, especially at first, leverage some of the free offerings like Google's DNS. I don't recommend firewalls for service provider networks, but you should make sure your gear can run (and is configured to do so) BCP 38.
Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------
On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor@gmail.com> wrote:
If a new operator or city is building a greenfield access network from the ground up, what software and hardware is needed in the core network to provide and manage residential and business internet services similar to the likes of AT&T, Comcast, and Google Fiber? Television and Telephone services are not to be considered only internet.
Assume hypothetically the operator already has the following in place: 10 GPON OLTs Chassis from an access vendor in 10 POPs around town (each POP has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. Dark fiber going from the POP locations back to the core location Assume a 32:1 way split, and each OLT chassis has enough ports populated to serve the area. 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged mode. Assume you are building a network designed to serve 10,000 subs All the fiber splitters, ducts, fiber, etc connecting the OLTs to the ONTs is already in place ASN from ARIN /20 of IPv4 space and /32 of IPv6 space from ARIN 4 burstable 10G internet connections from 4 tier 1 internet providers
Questions are:
What is the ideal way to aggregate the 40 10G connections from the uplinks of the chassis? I would guess a 10G switch since 10G ports on a router would be much more expensive? Which router is recommended to handle 4 10G internet connections with full tables, and then at least 4 10G ports going back to the 10G aggregation switch? How do you handle IP address management? a /20 is only 4096 IP addresses, but the network would have potentially 10,000 customers. Assume that getting more space from ARIN is not an option. Is CGN an option? Dynamic IP addresses? DHCP? How do you separate users and traffic? VLANs, Service VLANs, Per Customer VLANs, Usernames? Passwords? PPPoE? MAC Separation? Is a BRAS or BGN functionally really needed or are these older concepts? If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? The core router, a linux box, or something else? What about DNS? Is a firewall needed in the core? What else is needed?
Is there a guide out there somewhere? I know many cities are looking at building their own network, and have similar questions. Access vendors are willing to sell gear all day long, but then they leave it up to the operator/city to answer these harder questions.
How would you build a access network from the ground up if you had the resources and time to do so? Would you even use GPON? Even if GPON was not used and another access technology like AE, VDSL2, or wireless was used I think many of these questions would be the same.
1. The article mentioned DHCP doesn't do the other part of what PPPoE or PPPoA does, which is generate RADIUS accounting records that give us the bandwidth information. So that’s one of the main challenges in switching to a DHCP based system. So, how do you handle bandwidth tracking in an all DHCP environment then? If I want to track how many GB a customer used last month, or the average Mbps used how do you do so? A medium sized NMS could do 95th percentile usage on 10k ports. Normally I wouldn't want to use an NMS for billing usage but the capability is there. 2. I liked your option 82 example, and that works well for DSL networks where one port is tied to one customer. But how does option 82 work when you have multiple customers hanging off a GPON port? What does GPON use a subport identifier? The ONT can put an option-82 header on the packet and tag whichever port
3. You mentioned, DHCP is again, not a authentication protocol. So what handles authentication then if only DHCP is used, and there are no usernames and passwords? I guess for DSL networks you can enable or disable the port to allow or disallow access, and Option 82 for identification? I assume you wouldn't want to shut off the GPON OLT port if one customer wasn't paying their bill as it would affect the other customers on that port. I assume access vendors allow you to shut down the sub port or ONT in this situation for GPON? Still that seems messy having to login to a shelf or EMS system or API to an EMS system especially if you have multiple access vendors in a network. Is there a way to do authentication with DHCP? What about open networks like wifi where anyone can connect, so you don't have the ability to turn of the port or disable the end device? Most GPON vendors either support TR-69 or some other means to remote
On 7/31/2014 12:07 PM, Colton Conor wrote: the DHCP request came from. provision the ONTs. You can use the DHCP option-82 to identify who a customer is and then send their ONT a specific config. Like DOCSIS you could make a disable profile, or you could make them hop on a different VLAN that redirects all traffic to a billing page or something. There is also DPoE/DPoG (DOCSIS Provisioning of EPON/GPON) that converts DOCSIS provisioning into something PON can use.
4. I don't think anyone is buying a BRAS anymore, but looks like Cisco, Juniper, and ALU have what they call BGN, Broadband Subscriber Management, and other similar software. How are these different from BRAS functionality? I've got no experience with BRAS so I'm not sure. I think the ASR1k can do pppoe termination if you want a Cisco solution. So it looks like there are open source and commercial solutions for DHCP and DNS. Some providers like Infloblox seems to integrate all these into one.
Infoblox, Bluecat, 6connect, Incognito, Promptlink, VitalQIP, Cisco BAC There are a bunch of vendors and they all have their ups and downs. A DHCP system can be an expensive part of your network and it's a very critical one, so you might want to look at multiple offerings before deciding.
So if we have a core router that speaks BGP, a 10G aggregation switch to aggregate the the chassis, and a device like Infloblox or the other commercial solutions you mentioned that do DHCP/DNS, is there anything else that is needed besides the access gear already mentioned in the assumptions? Are these large and expensive commercial BGN/Broadband Subscriber management products a thing of the past or still very relevant in todays environment?
Make sure you've got your provisioning system planned out and working before you run with it. Your DHCP systems will tie heavily into your OSS so you'll need to work that piece out. If you use an NMS for billing reasons then that will need to tie into the OSS as well. It's always possible to roll out a network that just works, turn up a bunch of devices and then realize a critical piece is broken or badly designed. You don't want to be in a position where everything works except.... and you can't take it down because everyone is using it.
On Jul 31, 2014, at 8:23 PM, Colton Conor <colton.conor@gmail.com> wrote:
Is a firewall needed in the core?
No, quite the opposite: <https://app.box.com/s/a3oqqlgwe15j8svojvzl>
How would you build a access network from the ground up if you had the resources and time to do so?
I'd hire folks who have experience from both and architectural and operational perspectives, and who have the necessary local knowledge. Most of the question you're asking (except the one about iatrogenic stateful firewalls) are situationally-specific, and aren't really going to be answerable in detail via a mailing-list, no matter the depth and breadth of expertise of many of those participating in said email list. For example, you've asked nothing specifically about recursive or authoritative DNS infrastructure, although they're both key (you did mention DNS generically, which is good, but that's overly broad). Nothing about availability and resiliency and telemetry visibility and network hardening. Nothing about access policies, mitigation systems, quarantine systems, etc. Nothing about upstream transit requirements, nothing about peering goals and imperatives. Nothing about redundancy at any level/in any area/for any function. And so forth. I'm not criticizing you; I'm just trying to make the point that instead of concentrating on vendors and technologies and hardware and software, it's better to concentrate on *people* who have the requisite experience and expertise, and go from there. There are lots of specializations and subspecializations, and it's important to have folks who have broad experience spanning multiple areas, as well as others who know *everything* in a given area. While you can get some categorical advice, you can't really crowdsource the architecture, design, deployment, and operations of your network. ;> ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
Roland, I agree with everything you mentioned in your email. No matter how much money and resources you have, if you don't have the talent and people required to get the job done the project will fail. There a many outfits, like Scotts for example, that will handle most all of these issues for an operator that doesn't have the skills, talent, or personnel to deploy such a network on their own. I tried to keep the topics as broad as possible. No, I didn't go into detail about recursive or authoritative as I figured the general term DNS would cover both for the readers of this forum. The same with availability and resiliency and telemetry visibility and network hardening and the other detailed terms you have mentioned as I am making the assumption that this networking gear being talk about (carrier grade routers) would have most of these capabilities and people that would implement them (certified network engineers) would handle these issues. With that being said, we are not trying to crowdsource the architecture, design, deployment, and operations of our network. We are just seeking categorical advice as mentioned. If you ask this question to many of the network vendors that make these products they will try to oversell you on items you don't need. Just trying to cut through some of the marketing BS that the vendors produce, and see what people in the real world are actually deploying. On Thu, Jul 31, 2014 at 11:24 AM, Roland Dobbins <rdobbins@arbor.net> wrote:
On Jul 31, 2014, at 8:23 PM, Colton Conor <colton.conor@gmail.com> wrote:
Is a firewall needed in the core?
No, quite the opposite:
<https://app.box.com/s/a3oqqlgwe15j8svojvzl>
How would you build a access network from the ground up if you had the resources and time to do so?
I'd hire folks who have experience from both and architectural and operational perspectives, and who have the necessary local knowledge. Most of the question you're asking (except the one about iatrogenic stateful firewalls) are situationally-specific, and aren't really going to be answerable in detail via a mailing-list, no matter the depth and breadth of expertise of many of those participating in said email list.
For example, you've asked nothing specifically about recursive or authoritative DNS infrastructure, although they're both key (you did mention DNS generically, which is good, but that's overly broad). Nothing about availability and resiliency and telemetry visibility and network hardening. Nothing about access policies, mitigation systems, quarantine systems, etc. Nothing about upstream transit requirements, nothing about peering goals and imperatives. Nothing about redundancy at any level/in any area/for any function. And so forth.
I'm not criticizing you; I'm just trying to make the point that instead of concentrating on vendors and technologies and hardware and software, it's better to concentrate on *people* who have the requisite experience and expertise, and go from there. There are lots of specializations and subspecializations, and it's important to have folks who have broad experience spanning multiple areas, as well as others who know *everything* in a given area.
While you can get some categorical advice, you can't really crowdsource the architecture, design, deployment, and operations of your network.
;>
---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Equo ne credite, Teucri.
-- Laocoön
On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor@gmail.com> wrote:
If a new operator or city is building a greenfield access network from the ground up,
Hi Colton, We just had a long discussion in this forum to the effect that if a city builds a greenfield access network, it would be best limited to "layer 1" services. That is, deliver dark fiber and invite as many service providers as possible to light it with whatever services they're inclined to sell. Commercially, the L1 infrastructure presents the barrier to entry. That's why you don't have enough competitive commercial entities mooting the need to even discuss providing Internet as a municipal service. Even the smallest city is attractive to competitive commercial service providers when they can lease in-place L1 infrastructure ad hoc. This isn't as sexy as delivering gigabit Internet in the way roads aren't as sexy as the cars which drive on them but it relieves the city of having to make most of the hard-to-get-right decisions that could tank your effort and turn it into a boondoggle. Let commercial entities worry about what car will be popular next year and let commercial entities figure out which stores folks will drive those cars to. Just worry about where to build roads. On Thu, Jul 31, 2014 at 12:24 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
I'm not criticizing you; I'm just trying to make the point that instead of concentrating on vendors and technologies and hardware and software, it's better to concentrate on *people* who have the requisite experience and expertise, and go from there.
This. So much this. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> Can I solve your unusual networking challenges?
participants (5)
-
Colton Conor -
Robert Drake -
Roland Dobbins -
Scott Helms -
William Herrin