Colton Conor wrote:
I know Arista is typically a switch manufacturer, but with their recently announced Arista 7500R Series and soon to be announced but already shipping 7280R Series Arista is officially getting into the routing game. The fixed 1U 7280R Series looks quite impressive. The 7500R series is your traditional chassis and line card based solution.
Both of these products have the ability to hold the full internet routing table, and Arista is working on MPLS features. Both of these new products use the latest Broadcom Jerico chipsets.
We (Netflix) have been deploying the previous gen (7500E) as edge routers for about two years in high traffic, low route count applications in our CDN, and have been working with Arista for almost as long to improve route scale so that we could turn off all our traditional routers. The features that enable full routes on Jericho are running in our production network today and we also have the 7500R and 7280R working with full tables. I can't speak to MPLS, but for our use case (all L3, very high-density 10/40/100G, BGP, IS-IS and light QoS), it's working well. So, yes, I'd say those two products are quite viable and competitive options in the edge router space.
Ryan, What routing platform were you coming from before? What features does Arista not have that you find limiting that the old platform did have? How does Astira's Sflow only compare to having Cisco Netflow or Juniper JFlow for traffic monitoring which I assume Netflix does alot of? On Wed, Apr 20, 2016 at 3:48 PM, Ryan Woolley <rwoolleynanog@gmail.com> wrote:
Colton Conor wrote:
I know Arista is typically a switch manufacturer, but with their recently announced Arista 7500R Series and soon to be announced but already shipping 7280R Series Arista is officially getting into the routing game. The fixed 1U 7280R Series looks quite impressive. The 7500R series is your traditional chassis and line card based solution.
Both of these products have the ability to hold the full internet routing table, and Arista is working on MPLS features. Both of these new products use the latest Broadcom Jerico chipsets.
We (Netflix) have been deploying the previous gen (7500E) as edge routers for about two years in high traffic, low route count applications in our CDN, and have been working with Arista for almost as long to improve route scale so that we could turn off all our traditional routers.
The features that enable full routes on Jericho are running in our production network today and we also have the 7500R and 7280R working with full tables.
I can't speak to MPLS, but for our use case (all L3, very high-density 10/40/100G, BGP, IS-IS and light QoS), it's working well.
So, yes, I'd say those two products are quite viable and competitive options in the edge router space.
IOS-XR on ASR 9k and Junos on MX. For our use case, there's no longer anything limiting as compared to those platforms. BGP policy is perhaps not as rich as you might be used to if your experience is with the sort of routers traditionally marketed to service providers, but I'm sure that will get better, and it's probably irrelevant if your policy is fairly static. You are correct that we do collect a lot of flow data, both via sflow and Netflow. We've been able to do everything we need with Arista's sflow implementation. On Mon, Apr 25, 2016 at 6:41 PM, Colton Conor <colton.conor@gmail.com> wrote:
Ryan,
What routing platform were you coming from before? What features does Arista not have that you find limiting that the old platform did have?
How does Astira's Sflow only compare to having Cisco Netflow or Juniper JFlow for traffic monitoring which I assume Netflix does alot of?
On Wed, Apr 20, 2016 at 3:48 PM, Ryan Woolley <rwoolleynanog@gmail.com> wrote:
I know Arista is typically a switch manufacturer, but with their recently announced Arista 7500R Series and soon to be announced but already shipping 7280R Series Arista is officially getting into the routing game. The fixed 1U 7280R Series looks quite impressive. The 7500R series is your traditional chassis and line card based solution.
Both of these products have the ability to hold the full internet routing table, and Arista is working on MPLS features. Both of these new
Colton Conor wrote: products
use the latest Broadcom Jerico chipsets.
We (Netflix) have been deploying the previous gen (7500E) as edge routers for about two years in high traffic, low route count applications in our CDN, and have been working with Arista for almost as long to improve route scale so that we could turn off all our traditional routers.
The features that enable full routes on Jericho are running in our production network today and we also have the 7500R and 7280R working with full tables.
I can't speak to MPLS, but for our use case (all L3, very high-density 10/40/100G, BGP, IS-IS and light QoS), it's working well.
So, yes, I'd say those two products are quite viable and competitive options in the edge router space.
Ryan, Curious if you have any thoughts on the longevity of the 7500R and 7280R survival's with IPv4 full tables? How full are you seeing the TCAM getting today (I'm assuming they are doing some form of selective download)? And if we are currently adding 100k/routes a year, how much longer will it last? -Peter
On Wed, Apr 27, 2016 at 4:41 PM, Peter Kranz <pkranz@unwiredltd.com> wrote:
Curious if you have any thoughts on the longevity of the 7500R and 7280R survival's with IPv4 full tables? How full are you seeing the TCAM getting today (I'm assuming they are doing some form of selective download)? And if we are currently adding 100k/routes a year, how much longer will it last?
I can't speak for Ryan or Netflix, but we (Arista) are stating our technique is good for 1M+ prefixes of IPv4+v6 combined. Internet right now is at between 575K and 635K IPv4 and between 28K and 35K IPv6 right now and its taken many many many years to get there, its foreseeable there's many years of growth there. Note that we don't do static partitioning between IPv4 and IPv6 and our how we do it has more headroom in it than we state, so we're confident. We're also not doing "selective download", this is every prefix in current table. What I can share is two different scenarios today: 1. a traditional internet edge router with multiple transit/peer providers, Internet as of right now, and a cloud customer that also has hundreds of thousands of prefixes internally Ryan's case might be different to others, but here are three scenarios deployed today: 1. a large hosting provider with full tables and many internal prefixes, 2. a cloud deployment. The former is at 854K IPv4 and 35K IPv6 of 'internet' as of a few weeks ago: 7500R# show ip route summary | grep Total Total Routes 575127 7500R# show ipv6 route summary | grep Total Total Routes 35511 7500R# show hardware capacity | grep Routing Forwarding Resources Usage Table Feature Chip Used Used Free Committed Best Case High Entries (%) Entries Entries Max Watermark Entries -------- ---------- --------- -------- ------ --------- ----------- ----------- --------- Routing Resource1 815 39% 1233 0 2048 817 Routing Resource2 469 45% 555 0 1024 471 Routing Resource3 14074 42% 18694 0 32768 14098 Routing V4Routes 696364 88% 89753 0 786432 697110 Routing V6Routes 0 0% 89753 0 786432 0 The latter is at 854K IPv4 + 45K IPv6: 7500R# show ip route summary | grep Total Total Routes 854393 7500R# show ipv6 route summary | grep Total Total Routes 45678 7500R# show hardware capacity | grep Routing Forwarding Resources Usage Table Feature Chip Used Used Free Committed Best Case High Entries (%) Entries Entries Max Watermark Entries -------- ---------- --------- -------- ------ --------- ----------- ----------- --------- Routing Resource1 1319 64% 729 0 2048 1320 Routing Resource2 809 79% 215 0 1024 814 Routing Resource3 24102 73% 8666 0 32768 24104 Routing V4Routes 644336 83% 124302 0 786432 644364 Routing V6Routes 17792 12% 124302 0 786432 17795 One could ask Geoff Huston where he thinks combined IPv4+v6 will exceed 1M entries but I would expect it to be many years away based on http://bgp.potaroo.net/ and we'd welcome discussions about if it you want to know our opinion [*] on how we're doing it will scale. What we're doing doesn't explode at 1M, there's headroom in it hence why we say "1M+". Again we're happy to talk about it, just ask your friendly arista person and if you don't know who to ask, ask me and i'll put you in touch with the right folks. cheers, lincoln. [*] ltd@arista.com
Well, Once you eliminate the ~160k superfluous prefixes (last time I checked)... This is a none issue. Some work on some sort summary function would keep those devices alive... but we all know there is more money to be made the faster the device become obsolete :( ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 04/28/16 01:33, lincoln dale wrote:
On Wed, Apr 27, 2016 at 4:41 PM, Peter Kranz <pkranz@unwiredltd.com> wrote:
Curious if you have any thoughts on the longevity of the 7500R and 7280R survival's with IPv4 full tables? How full are you seeing the TCAM getting today (I'm assuming they are doing some form of selective download)? And if we are currently adding 100k/routes a year, how much longer will it last?
I can't speak for Ryan or Netflix, but we (Arista) are stating our technique is good for 1M+ prefixes of IPv4+v6 combined. Internet right now is at between 575K and 635K IPv4 and between 28K and 35K IPv6 right now and its taken many many many years to get there, its foreseeable there's many years of growth there. Note that we don't do static partitioning between IPv4 and IPv6 and our how we do it has more headroom in it than we state, so we're confident. We're also not doing "selective download", this is every prefix in current table.
What I can share is two different scenarios today:
1. a traditional internet edge router with multiple transit/peer providers, Internet as of right now, and a cloud customer that also has hundreds of thousands of prefixes internally Ryan's case might be different to others, but here are three scenarios deployed today: 1. a large hosting provider with full tables and many internal prefixes, 2. a cloud deployment.
The former is at 854K IPv4 and 35K IPv6 of 'internet' as of a few weeks ago:
7500R# show ip route summary | grep Total Total Routes 575127 7500R# show ipv6 route summary | grep Total Total Routes 35511 7500R# show hardware capacity | grep Routing Forwarding Resources Usage
Table Feature Chip Used Used Free Committed Best Case High Entries (%) Entries Entries Max Watermark
Entries -------- ---------- --------- -------- ------ --------- ----------- ----------- --------- Routing Resource1 815 39% 1233 0 2048 817 Routing Resource2 469 45% 555 0 1024 471 Routing Resource3 14074 42% 18694 0 32768 14098 Routing V4Routes 696364 88% 89753 0 786432 697110 Routing V6Routes 0 0% 89753 0 786432 0
The latter is at 854K IPv4 + 45K IPv6:
7500R# show ip route summary | grep Total Total Routes 854393 7500R# show ipv6 route summary | grep Total Total Routes 45678 7500R# show hardware capacity | grep Routing Forwarding Resources Usage
Table Feature Chip Used Used Free Committed Best Case High Entries (%) Entries Entries Max Watermark
Entries -------- ---------- --------- -------- ------ --------- ----------- ----------- --------- Routing Resource1 1319 64% 729 0 2048 1320 Routing Resource2 809 79% 215 0 1024 814 Routing Resource3 24102 73% 8666 0 32768 24104 Routing V4Routes 644336 83% 124302 0 786432 644364 Routing V6Routes 17792 12% 124302 0 786432 17795
One could ask Geoff Huston where he thinks combined IPv4+v6 will exceed 1M entries but I would expect it to be many years away based on http://bgp.potaroo.net/ and we'd welcome discussions about if it you want to know our opinion [*] on how we're doing it will scale. What we're doing doesn't explode at 1M, there's headroom in it hence why we say "1M+". Again we're happy to talk about it, just ask your friendly arista person and if you don't know who to ask, ask me and i'll put you in touch with the right folks.
cheers,
lincoln. [*] ltd@arista.com
On 2016-04-28 11:06, Alain Hebert wrote:
Well,
Once you eliminate the ~160k superfluous prefixes (last time I checked)... This is a none issue.
Some work on some sort summary function would keep those devices alive... but we all know there is more money to be made the faster the device become obsolete :(
Can you explain how this works? How can a router determine which prefix is superfluous? How does it cope when a suppressed prefix is withdrawn or a more specific prefix is added? Is this just one of those 'it works some of the time' solutions or is this something that can be done safely with an appropriate algorithm? Thanks, Laszlo
Laszln, Thu, Apr 28, 2016 at 12:47:45PM +0000, Laszlo Hanyecz wrote:
On 2016-04-28 11:06, Alain Hebert wrote:
Well,
Once you eliminate the ~160k superfluous prefixes (last time I checked)... This is a none issue.
Some work on some sort summary function would keep those devices alive... but we all know there is more money to be made the faster the device become obsolete :(
Can you explain how this works? How can a router determine which prefix is superfluous? How does it cope when a suppressed prefix is withdrawn or a more specific prefix is added? Is this just one of those 'it works some of the time' solutions or is this something that can be done safely with an appropriate algorithm?
A fair chunk of the routing table is aggregable. If multiple aggregable prefixes share the same nexthop, the HW entries can be summarised accordingly, reducing the HW resource footprint. Should one of the smaller prefixes be withdrawn or best path change to another nexthop, the control plane needs to be smart enough to adapt and reprogram the HW accordingly. It is a fairly logical and reasonable algorithm to construct -- Patrick Cole <z@wwwires.com> Senior Network Specialist World Without Wires PO Box 869. Palm Beach, QLD, 4221 Ph: 0410 626 630
On Thu, Apr 28, 2016 at 1:33 AM, lincoln dale <ltd@interlink.com.au> wrote:
On Wed, Apr 27, 2016 at 4:41 PM, Peter Kranz <pkranz@unwiredltd.com> wrote:
Curious if you have any thoughts on the longevity of the 7500R and 7280R survival's with IPv4 full tables? How full are you seeing the TCAM getting today (I'm assuming they are doing some form of selective download)? And if we are currently adding 100k/routes a year, how much longer will it last?
[...]
One could ask Geoff Huston where he thinks combined IPv4+v6 will exceed 1M entries but I would expect it to be many years away based on http://bgp.potaroo.net/ and we'd welcome discussions about if it you want to know our opinion [*] on how we're doing it will scale. What we're doing doesn't explode at 1M, there's headroom in it hence why we say "1M+". Again we're happy to talk about it, just ask your friendly arista person and if you don't know who to ask, ask me and i'll put you in touch with the right folks.
Peter, I'd point you to https://labs.apnic.net/?p=767 for more historical detail and a table with some (recent) predictions. The summary is that the rate is mostly linear at around 10% per year and even 1MM routes lasts quite comfortably beyond 5 years at the current growth rate. I am not particularly worried about the table growth rate (or Moore's law) changing dramatically. With respect to the utilization of the hardware, our setup is basically the same as Lincoln's scenario #1 and so utilization looks about the same, on both platforms.
participants (7)
-
Alain Hebert -
Colton Conor -
Laszlo Hanyecz -
lincoln dale -
Patrick Cole -
Peter Kranz -
Ryan Woolley