hey, Some time ago, many people noticed rapid IPv6 deployment growth in Estonia (from 0% to 5% in 4 weeks). We at 3249/Elion/Estonian Telecom were behind this, other operators don't have any serious IPv6 deployments at the moment. We rolled out v6 to everyone (both business and residential customers) with last-gen CPE, there was no hop-in our hop-out program - aim was to do it perfectly and without customers even noticing. I'm happy to say that we achieved this goal :) To satisfy general interest, I promised small (somehow it turned out longer than I expected) technical writeup how we enabled v6 for our subscribers. If you have any other questions, feel free to ask and I do my best to answer them. You can also skip the technical content and there are some statistics below. Our access network is mix of DSL/GPON/wimax/p2p-ETH and broadband service is deployed in shared service vlans. IPv6 traffic shares vlan with IPv4. Service vlans are transported over MPLS metro network using pseudowires and terminated in geo-clustered Alcatel 7750 BNG routers. Each subscriber is allocated up to 4 mixed v4 and v6 IP hosts. For v4 we are using the usual DHCP, for IPv6 we are using DHCPv6 with IA_PD only, no IA_NA is provided. Unfortunately DHCPv6 provides no way to signal IPv6 default-route thus we have to fall back to RA for default-route. RA does not include any on-link prefixes or DNS information. RAs are L2 unicasted to CPE MAC so no other CPE in service vlan picks up those RAs. To ensure rapid switchover between BNG routers, we are signalling virtual link-local address as default-route. We are using ALU internal DHCP/DHCPv6 servers to allocate leases but we also signal IP information from radius (in such case BNG "fakes" DHCP server) for static IP customers. Provided IPv6 prefix is always /56 and we keep the old lease for 24h even if the CPE is turned off (actual lease time is 30min). Unfortunately, IPv6 LDRA is not available on most of our access platforms so we have to rely on IPv4 session information for authentication. This linking is done in the radius server during subscriber authentication (excellent radiator + quite awful SQL queries :) - if subscriber has IPv4 session (that has been authenticated using DHCP opt82), same MAC address is allowed to have IPv6 session on exactly the same virtual BNG port. IPv4 and v6 session are both tied to same subscriber and share shapers, QOS etc. We were able to enable IPv6 only on our last-gen Inteno CPEs. They run modified OpenWrt and because it's linux - everything is possible :) In CPE, /56 is divided up to /64s, first one is currently reserved but we will configure it on loopback interface and use it for CPE management. Second /64 is configured on LAN and third is configured on public wifi SSID (if you choose to enable this option). In the LAN, IPv6 config is provided by RAs, we also support RDNSS and stateless DHCPv6 for DNS. There is also ingress IPv6 firewall in the CPE and configuration is modifiable by user. To make deployment as smooth as possible, we rolled out IPv6 capable CPE software first. Then, during the BNG platform refresh, we deployed L2 ACLs that dropped all IPv6 traffic based on 0x86dd ethertype. We then deployed IPv6 config to all BNGs and could verify everything before single v6 lease was handed out to the subscribers. Then, interface by interface, we replaced L2 ACL with one that only allowed 0x86dd for certain, supported, OUIs. This is the current situation and we are investigating ways to support 3rd party CPEs - main problem is unreliable IPv6 config in CPEs. Many don't enable DHCPv6 (or enable NA but no PD) but still pick up default-route from RA and happily signal it to LAN. Some others hammer our BNGs with NA request every 0.1 seconds etc. As statistics go, there are 30000+ active IPv6 subscribers (almost 15% of our customer base, based on our public numbers), 81% of them have have at least one IPv6 enabled device in the LAN, 70% have more than one. Most IPv6 traffic is generated by Google+Youtube, Facebook and Akamai. Not bad for a country with 1.3M people. Next up: mobile network :) -- tarko
Hello, folks! Tere from your customer FastVPS Eesti OU/AS198068! :) On Mon, Dec 22, 2014 at 6:27 PM, Tarko Tikan <tarko@lanparty.ee> wrote:
hey,
Some time ago, many people noticed rapid IPv6 deployment growth in Estonia (from 0% to 5% in 4 weeks). We at 3249/Elion/Estonian Telecom were behind this, other operators don't have any serious IPv6 deployments at the moment. We rolled out v6 to everyone (both business and residential customers) with last-gen CPE, there was no hop-in our hop-out program - aim was to do it perfectly and without customers even noticing. I'm happy to say that we achieved this goal :)
To satisfy general interest, I promised small (somehow it turned out longer than I expected) technical writeup how we enabled v6 for our subscribers. If you have any other questions, feel free to ask and I do my best to answer them. You can also skip the technical content and there are some statistics below.
Our access network is mix of DSL/GPON/wimax/p2p-ETH and broadband service is deployed in shared service vlans. IPv6 traffic shares vlan with IPv4.
Service vlans are transported over MPLS metro network using pseudowires and terminated in geo-clustered Alcatel 7750 BNG routers.
Each subscriber is allocated up to 4 mixed v4 and v6 IP hosts. For v4 we are using the usual DHCP, for IPv6 we are using DHCPv6 with IA_PD only, no IA_NA is provided. Unfortunately DHCPv6 provides no way to signal IPv6 default-route thus we have to fall back to RA for default-route. RA does not include any on-link prefixes or DNS information. RAs are L2 unicasted to CPE MAC so no other CPE in service vlan picks up those RAs. To ensure rapid switchover between BNG routers, we are signalling virtual link-local address as default-route.
We are using ALU internal DHCP/DHCPv6 servers to allocate leases but we also signal IP information from radius (in such case BNG "fakes" DHCP server) for static IP customers. Provided IPv6 prefix is always /56 and we keep the old lease for 24h even if the CPE is turned off (actual lease time is 30min).
Unfortunately, IPv6 LDRA is not available on most of our access platforms so we have to rely on IPv4 session information for authentication. This linking is done in the radius server during subscriber authentication (excellent radiator + quite awful SQL queries :) - if subscriber has IPv4 session (that has been authenticated using DHCP opt82), same MAC address is allowed to have IPv6 session on exactly the same virtual BNG port. IPv4 and v6 session are both tied to same subscriber and share shapers, QOS etc.
We were able to enable IPv6 only on our last-gen Inteno CPEs. They run modified OpenWrt and because it's linux - everything is possible :)
In CPE, /56 is divided up to /64s, first one is currently reserved but we will configure it on loopback interface and use it for CPE management. Second /64 is configured on LAN and third is configured on public wifi SSID (if you choose to enable this option).
In the LAN, IPv6 config is provided by RAs, we also support RDNSS and stateless DHCPv6 for DNS. There is also ingress IPv6 firewall in the CPE and configuration is modifiable by user.
To make deployment as smooth as possible, we rolled out IPv6 capable CPE software first. Then, during the BNG platform refresh, we deployed L2 ACLs that dropped all IPv6 traffic based on 0x86dd ethertype. We then deployed IPv6 config to all BNGs and could verify everything before single v6 lease was handed out to the subscribers.
Then, interface by interface, we replaced L2 ACL with one that only allowed 0x86dd for certain, supported, OUIs. This is the current situation and we are investigating ways to support 3rd party CPEs - main problem is unreliable IPv6 config in CPEs. Many don't enable DHCPv6 (or enable NA but no PD) but still pick up default-route from RA and happily signal it to LAN. Some others hammer our BNGs with NA request every 0.1 seconds etc.
As statistics go, there are 30000+ active IPv6 subscribers (almost 15% of our customer base, based on our public numbers), 81% of them have have at least one IPv6 enabled device in the LAN, 70% have more than one. Most IPv6 traffic is generated by Google+Youtube, Facebook and Akamai. Not bad for a country with 1.3M people.
Next up: mobile network :)
-- tarko
-- Sincerely yours, Pavel Odintsov
On 2014-12-22 16:27, Tarko Tikan wrote:
Our access network is mix of DSL/GPON/wimax/p2p-ETH and broadband service is deployed in shared service vlans. IPv6 traffic shares vlan with IPv4.
How do you protect customers from each other? There are many nasty IPv6 attacks you can do when on a shared VLAN. /Anders
hey,
How do you protect customers from each other?
There are many nasty IPv6 attacks you can do when on a shared VLAN.
Split-horizon (switchport protected in Cisco world). Customers can't send packets directly to each other, all communication has to go via BNG router. Obviously we protect L2 as well like limiting number of MACs per customers, make sure BNG MAC cannot be learned from customer ports etc. We don't use any L3 (both v4 and v6) inspection in ANs, everything happens in BNG. It's actually much better and logical for v6 as it is for v4. In v4 world you have to implement proxy-arp, in v6 world there is no need for customers to send packets to each others link-local WAN addresses and packets sent to PD addresses are by default routed via BNG. -- tarko
On 2014-12-27 17:27, Tarko Tikan wrote:
Split-horizon (switchport protected in Cisco world). Customers can't send packets directly to each other, all communication has to go via BNG router. Obviously we protect L2 as well like limiting number of MACs per customers, make sure BNG MAC cannot be learned from customer ports etc. We don't use any L3 (both v4 and v6) inspection in ANs, everything happens in BNG.
Ok seems simple enough. I assume you have a star-network below the BNG? Ie no rings or similar in the access network? /Anders
hey,
I assume you have a star-network below the BNG? Ie no rings or similar in the access network?
Most of our network below BNG is MPLS, so no, it's not a star per say. But as PWs are point-to-point, you are technically correct. Below MPLS there is some ethernet too and this is all strictly star/tree. I would encourage everyone to push MPLS as close to customer as possible, this makes redundancy, BNG placement etc. all much easier as you can use IGP/MPLS + PWs. -- tarko
Hi, On Sat, Dec 27, 2014 at 05:15:13PM +0100, Anders L??winger wrote:
On 2014-12-22 16:27, Tarko Tikan wrote:
Our access network is mix of DSL/GPON/wimax/p2p-ETH and broadband service is deployed in shared service vlans. IPv6 traffic shares vlan with IPv4.
How do you protect customers from each other?
There are many nasty IPv6 attacks you can do when on a shared VLAN.
true, but some (most) of them only apply in networks where multicasting/ND is fully supported which is not necessarily the case in the above type of networks. and, from what I understand, in their scenario RAs are not sent to link-local scope all nodes (ff02::1), so that would eliminate another attack vector (depending on the actual processing of RAs on the CPEs). best Enno
/Anders
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
On 2014-12-27 17:37, Enno Rey wrote:
true, but some (most) of them only apply in networks where multicasting/ND is fully supported which is not necessarily the case in the above type of networks.
Yes. I'm aware of the various types of solutions for security in IPv6 with shared VLANs. I was curious of what solution they used.
and, from what I understand, in their scenario RAs are not sent to link-local scope all nodes (ff02::1), so that would eliminate another attack vector (depending on the actual processing of RAs on the CPEs).
In P2P-Eth you can always remove the CPE and connect your hacker PC instead, and then start to inject RAs. Depending on the network this will be handled or not. Now it sounds they have a good solution in place, no L2 between customer ports. /Anders
The access boxes and BNG typically have protection mechanisms in place. Also even though customers are in a shared VLAN and IP subnet they aren't typically on the same broadcast domain. In the case of active Ethernet you use things like private Vlans or other access controls. Phil -----Original Message----- From: "Anders Löwinger" <anders@abundo.se> Sent: 12/27/2014 11:17 AM To: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: Estonian IPv6 deployment report On 2014-12-22 16:27, Tarko Tikan wrote:
Our access network is mix of DSL/GPON/wimax/p2p-ETH and broadband service is deployed in shared service vlans. IPv6 traffic shares vlan with IPv4.
How do you protect customers from each other? There are many nasty IPv6 attacks you can do when on a shared VLAN. /Anders
participants (5)
-
Anders Löwinger -
Enno Rey -
Pavel Odintsov -
Phil Bedard -
Tarko Tikan