BGP peering strategies for smaller routers
Hello, I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?). I am looking to connect to a second ip transit provider and I'm looking for any advice or strategies that would allow me to take advantage and make good forwarding decisions while not breaking the bank on bgp memory consumption. I simply don't understand how this would likely play out and what memory consumption mitigation steps may be necessary here. Im open to ideas... a pair of route reflectors? selective bgp download? static route filter maps? Thank you. Mike-
On Mon, May 2, 2016 at 3:07 PM, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hello,
I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?).
I am looking to connect to a second ip transit provider and I'm looking for any advice or strategies that would allow me to take advantage and make good forwarding decisions while not breaking the bank on bgp memory consumption. I simply don't understand how this would likely play out and what memory consumption mitigation steps may be necessary here. Im open to ideas... a pair of route reflectors? selective bgp download? static route filter maps?
Thank you.
Mike-
You have to keep in mind there are two pools of memory on the router. The RIB and the FIB. The RIB contains all the routes that the router could possibly load. This includes all your BGP routes, even the ones not selected as best, and all your IGP routes. Whereas the FIB would only have the best routes that the router is actively using to forward traffic. FIB = 'sh ip route 4.2.2.2' RIB = 'sh ip bgp 4.2.2.2' (or OSPF, or IS-IS, or RIP, etc, etc) Generally FIB capacity is given as a number of routes and RIB capacity is given as a memory amount. Since the router manufacturer doesn't know what protocols you'll be running or how many attributes you'll be storing they don't really have a good idea of how many routes you'll get in the memory the router has. Now that you have a full table on the router, you won't see much growth on the FIB. For the most part every route you'll learn is already learned. You'll have some growth because not every transit provider will advertise the exact same table. My router with 3 transits has 580k routes for instance. In short, you're fine. Read up on RIB and FIB though so you have a good handle on when you're about to start running into problems. JM
You have to keep in mind there are two pools of memory on the router.
There's actually three. 1. Prefix (path) via BGP: "show ip bgp <prefix>". BGP will select the 'best' BGP path (can be multiple if ECMP) and send that through to the RIB. 2. RIB. "show ip route <prefix>". routing table will show the path chosen - and if there are backup paths etc, but may be recursive, e.g. prefix a.b.c.d points at e.f.g.h which in turn points at i.j.k.l etc. 3. FIB. basically fully resolved prefixes. What you otherwise say is correct - you could have N transit providers at (1) providing lotsOfPaths x N providers which ultimately resolve to lotsOfRoutes with up to N next-hops. Much design effort goes into the routing stack to efficiently store lotsOfPaths. Can't speak for what an ASR1K does but suggest the OP talk to Cisco. cheers, lincoln.
On 2/May/16 21:07, Mike wrote:
Hello,
I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?).
Sounds like you have enough router resources to do your peering and take 2 full feeds. Mark.
Rib or Fib for the million - thats the question - but in any event the following will most likely work for you. BTW, full table is now over 600K in size. 1) Choose one Transit and take their full table. (pick whatever reasons cost savings, bigger pipe, coin flip, etc.) 2) With the second transit use a filter to drop all everything /22 or smaller. Now check your tables , see if you have enough room. 3) Next add your peers - no filtering and lpref those routes about the transits. 4) Ask both transits to send you a default route. If this doesn't fit, use some more policy filtering and while this is up and running begin the search for a router with larger tables to replace it...as the tables will soon grow larger. Thank You Bob Evans CTO
On 2/May/16 21:07, Mike wrote:
Hello,
I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?).
Sounds like you have enough router resources to do your peering and take 2 full feeds.
Mark.
Hello. When we was in a similar situation we opted for one transit provider to provide a default to us then we filtered on AS-HOPS so prefixes that was more than 3 hops away was denied. This way we got the ones that where closest to us and that where more likely to matter. Prefixes that’s more than 3 hops away on both links could probably just as well go on a default insteed. However it’s a rather crude way of fixing the issue. We just did it to have the router up while we got extra memory from it. (we had memory shortage after an update that we needed to apply to correct some bug I think. We couldn’t just rollback the update if my memory serves me correct.) //Gustav -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mike Sent: den 2 maj 2016 21:07 To: NANOG list <nanog@nanog.org> Subject: BGP peering strategies for smaller routers Hello, I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?). I am looking to connect to a second ip transit provider and I'm looking for any advice or strategies that would allow me to take advantage and make good forwarding decisions while not breaking the bank on bgp memory consumption. I simply don't understand how this would likely play out and what memory consumption mitigation steps may be necessary here. Im open to ideas... a pair of route reflectors? selective bgp download? static route filter maps? Thank you. Mike-
I have used variations Gustav's solution below to good effect as well, this also works with two smaller routers providing basic fail over and load balancing. I found its best to take Full + default from one provider and just default from the other. Set a higher local-pref on the default only provider than the full+default one, then filter the full+default routes by AS-path as desired. Incoming control via the normal prepending of outgoing advertisements. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Gustav Ulander Sent: Tuesday, 3 May 2016 8:30 AM To: Mike <mike-nanog@tiedyenetworks.com>; NANOG list <nanog@nanog.org> Subject: RE: BGP peering strategies for smaller routers Hello. When we was in a similar situation we opted for one transit provider to provide a default to us then we filtered on AS-HOPS so prefixes that was more than 3 hops away was denied. This way we got the ones that where closest to us and that where more likely to matter. Prefixes that’s more than 3 hops away on both links could probably just as well go on a default insteed. However it’s a rather crude way of fixing the issue. We just did it to have the router up while we got extra memory from it. (we had memory shortage after an update that we needed to apply to correct some bug I think. We couldn’t just rollback the update if my memory serves me correct.) //Gustav -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mike Sent: den 2 maj 2016 21:07 To: NANOG list <nanog@nanog.org> Subject: BGP peering strategies for smaller routers Hello, I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?). I am looking to connect to a second ip transit provider and I'm looking for any advice or strategies that would allow me to take advantage and make good forwarding decisions while not breaking the bank on bgp memory consumption. I simply don't understand how this would likely play out and what memory consumption mitigation steps may be necessary here. Im open to ideas... a pair of route reflectors? selective bgp download? static route filter maps? Thank you. Mike-
Careful with the ASR1000 and full tables at 4GB. http://www.gossamer-threads.com/lists/cisco/nsp/180710 I recommend adding some third party RAM to get 16GB. On Mon, May 2, 2016 at 12:07 PM, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hello,
I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?).
I am looking to connect to a second ip transit provider and I'm looking for any advice or strategies that would allow me to take advantage and make good forwarding decisions while not breaking the bank on bgp memory consumption. I simply don't understand how this would likely play out and what memory consumption mitigation steps may be necessary here. Im open to ideas... a pair of route reflectors? selective bgp download? static route filter maps?
Thank you.
Mike-
On 2/May/16 22:32, Richard Hicks wrote:
Careful with the ASR1000 and full tables at 4GB.
http://www.gossamer-threads.com/lists/cisco/nsp/180710
I recommend adding some third party RAM to get 16GB.
It will be fine with 4GB of RAM provided the OP does not enable software redundancy. Mark.
Mike, the ASR1k series has several ESP options (ESP5, 10, 20, 40, 100, 200). Each ESP comes with a fixed amount of forwarding tcam which holds the forwarding information base (FIB). The ESP5 has 5MB of tcam can hold ~500k routes. The ESP10 has 10MB of tcam, so theoretically should hold roughly double (1 million routes). The ESP20 and ESP40 have 40MB of tam; Cisco quotes these as 4 million routes. http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-agg... There are two route processor (RP) options for the ASR1k series. The RP1 and RP2. If you have 4GB, then you have an upgraded RP1. Which Cisco quotes at 1,000,000 IP4 routes *OR* 500k IP6 routes. Not to get too nitty gritty, but I would simplify these to say that there are 1,000,000 route slots; each IP4 route takes 1 slot and each IP6 route takes 2. http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-agg... As others have noted, the best routes from the BGP table and other routing tables are condensed into the overall routing information base (RIB) and stored in DRAM. The RIB is then condensed into the FIB and stored as TCAM. Adding additional BGP peers will take minimal amounts of FIB if you're receiving the same full feed from both. The effect on the RIB and overall router memory is usually not that great either (I think on the ASR1k platform it's maybe ~100MB per peer). If you have an ESP10+, you're fine for two or more full feeds. If you have an ESP5, you really don't have the hardware to hold a single full feed. --Blake Mike wrote on 5/2/2016 2:07 PM:
Hello,
I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?).
I am looking to connect to a second ip transit provider and I'm looking for any advice or strategies that would allow me to take advantage and make good forwarding decisions while not breaking the bank on bgp memory consumption. I simply don't understand how this would likely play out and what memory consumption mitigation steps may be necessary here. Im open to ideas... a pair of route reflectors? selective bgp download? static route filter maps?
Thank you.
Mike-
Mike, I just did this with a ASR1001. I had to upgrade it to 8gb of ram (I got the real Cisco stuff for ~ $500). Before the router would crash when loading the tables. Right now, I have full tables from two providers: router1#show ip bgp summary BGP router identifier 192.55.82.2, local AS number 4505 BGP table version is 11150622, main routing table version 11150622 582461 network entries using 144450328 bytes of memory 911730 path entries using 109407600 bytes of memory 148924/93298 BGP path/bestpath attribute entries using 36933152 bytes of memory 132977 BGP AS-PATH entries using 6043938 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 296835018 total bytes of memory BGP activity 962568/380103 prefixes, 5155645/4243915 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.55.82.3 4 4505 2532914 1634867 11150622 0 0 3w0d 330377 192.55.82.4 4 4505 672950 1634865 11150622 0 0 3w0d 1 209.117.103.33 4 2828 1837130 48052 11150557 0 0 2w1d 581351 router1#show ip cef summary IPv4 CEF is enabled for distributed and running VRF Default 582527 prefixes (582527/0 fwd/non-fwd) Table id 0x0 Database epoch: 2 (582527 entries at this epoch) -Eric -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mike Sent: Monday, May 2, 2016 3:07 PM To: NANOG list <nanog@nanog.org> Subject: BGP peering strategies for smaller routers Hello, I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?). I am looking to connect to a second ip transit provider and I'm looking for any advice or strategies that would allow me to take advantage and make good forwarding decisions while not breaking the bank on bgp memory consumption. I simply don't understand how this would likely play out and what memory consumption mitigation steps may be necessary here. Im open to ideas... a pair of route reflectors? selective bgp download? static route filter maps? Thank you. Mike-
On 05/02/2016 07:35 PM, Eric Sabotta wrote:
Mike,
I just did this with a ASR1001. I had to upgrade it to 8gb of ram (I got the real Cisco stuff for ~ $500). Before the router would crash when loading the tables.
Right now, I have full tables from two providers:
router1#show ip bgp summary BGP router identifier 192.55.82.2, local AS number 4505 BGP table version is 11150622, main routing table version 11150622 582461 network entries using 144450328 bytes of memory 911730 path entries using 109407600 bytes of memory 148924/93298 BGP path/bestpath attribute entries using 36933152 bytes of memory 132977 BGP AS-PATH entries using 6043938 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 296835018 total bytes of memory BGP activity 962568/380103 prefixes, 5155645/4243915 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.55.82.3 4 4505 2532914 1634867 11150622 0 0 3w0d 330377 192.55.82.4 4 4505 672950 1634865 11150622 0 0 3w0d 1 209.117.103.33 4 2828 1837130 48052 11150557 0 0 2w1d 581351
router1#show ip cef summary IPv4 CEF is enabled for distributed and running VRF Default 582527 prefixes (582527/0 fwd/non-fwd) Table id 0x0 Database epoch: 2 (582527 entries at this epoch)
-Eric
But if I'm reading the above right, it looks like bgp is eating ~300mb on your box. BGP using 296835018 total bytes of memory You would seem to have plenty of free ram. In my case, the ASR1002 doesn't have upgradable memory anyways so I'm stuck. Mike-
Mike wrote on 5/2/2016 9:43 PM:
On 05/02/2016 07:35 PM, Eric Sabotta wrote:
Mike,
I just did this with a ASR1001. I had to upgrade it to 8gb of ram (I got the real Cisco stuff for ~ $500). Before the router would crash when loading the tables.
Right now, I have full tables from two providers:
router1#show ip bgp summary BGP router identifier 192.55.82.2, local AS number 4505 BGP table version is 11150622, main routing table version 11150622 582461 network entries using 144450328 bytes of memory 911730 path entries using 109407600 bytes of memory 148924/93298 BGP path/bestpath attribute entries using 36933152 bytes of memory 132977 BGP AS-PATH entries using 6043938 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 296835018 total bytes of memory BGP activity 962568/380103 prefixes, 5155645/4243915 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.55.82.3 4 4505 2532914 1634867 11150622 0 0 3w0d 330377 192.55.82.4 4 4505 672950 1634865 11150622 0 0 3w0d 1 209.117.103.33 4 2828 1837130 48052 11150557 0 0 2w1d 581351
router1#show ip cef summary IPv4 CEF is enabled for distributed and running VRF Default 582527 prefixes (582527/0 fwd/non-fwd) Table id 0x0 Database epoch: 2 (582527 entries at this epoch)
-Eric
But if I'm reading the above right, it looks like bgp is eating ~300mb on your box.
BGP using 296835018 total bytes of memory
You would seem to have plenty of free ram. In my case, the ASR1002 doesn't have upgradable memory anyways so I'm stuck.
Mike-
Mike, I have a customer that has not had any operational issues taking two full IP4 feeds on an ASR1002 with RP1 @ 4GB of RAM. Again, I'd recommend an ESP10 to ensure you have enough TCAM to hold the FIB and to track netflow or other data that relies on the ESP. I can't comment on the RP2's memory usage or stability as I don't think I've ran into them. Here's the output to show you what to expect: #sh processes memory | inc BGP|Total:|PID Processor Pool Total: 1725514176 Used: 918478524 Free: 807035652 lsmpi_io Pool Total: 6295088 Used: 6294116 Free: 972 PID TTY Allocated Freed Holding Getbufs Retbufs Process 114 0 252 0 23264 4 4 BGP Scheduler 282 0 165948 327892 189212 0 0 BGP Task 294 0 0 0 17264 0 0 BGP HA SSO 317 0 610260780 0 220080 3387715 3387715 BGP I/O 355 0 0 17841472 23264 0 0 BGP Scanner 405 0 0 0 23316 0 0 BGP Consistency 422 0 0 0 23264 0 0 BGP Event 528 0 560439808 646907016 498109200 51 51 BGP Router 533 0 0 0 17264 0 0 BGP VA #sh bgp su BGP router identifier 1.2.3.4, local AS number ccccc BGP table version is 66249122, main routing table version 66249122 598594 network entries using 86197536 bytes of memory 1177526 path entries using 94202080 bytes of memory 208207/100401 BGP path/bestpath attribute entries using 28316152 bytes of memory 179429 BGP AS-PATH entries using 7090834 bytes of memory 4342 BGP community entries using 230370 bytes of memory 1 BGP extended community entries using 24 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 216036996 total bytes of memory BGP activity 2943907/2345309 prefixes, 10963622/9786096 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd a.a.a.a 4 aaa 5523968 51273 66249070 0 0 4w4d 578933 b.b.b.b 4 bbbbb 3731440 101400 66249070 0 0 4w4d 598586 #sh cef fib 599851 allocated IPv4 entries, 0 failed allocations 1 allocated IPv6 entry, 0 failed allocations #sh ip route summary IP routing table name is default (0x0) IP routing table maximum-paths is 32 Route Source Networks Subnets Replicates Overhead Memory (bytes) connected 0 60 0 3600 10800 static 20 79 0 6060 17820 bgp ccccc 180614 417599 0 35892780 107678340 External: 598213 Internal: 0 Local: 0 internal 6552 23993840 Total 187186 417738 0 35902440 131700800
On Mon, May 2, 2016 at 10:35 PM, Eric Sabotta <esabotta@whdh.com> wrote:
I just did this with a ASR1001. I had to upgrade it to 8gb of ram (I got the real Cisco stuff for ~ $500). Before the router would crash when loading the tables.
Hi Eric, Something very fishy there because:
router1#show ip bgp summary BGP using 296,835,018 total bytes of memory
Commas added for clarity. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
Hello all. Yes I can confirm that we also had the issue with the asr1001s. For us the router was fine until we upgraded it. When we rebooted it after the upgrade it ran out of memory when populating 2 full feeds. When we contacted TAC they confirmed that indeed it was a memory problem and that we would need to add more memory to the box. Perhaps 1002 isnt as thirsty? //Gustav -----Ursprungligt meddelande----- Från: NANOG [mailto:nanog-bounces@nanog.org] För William Herrin Skickat: den 3 maj 2016 18:13 Till: Eric Sabotta <esabotta@whdh.com> Kopia: NANOG list <nanog@nanog.org> Ämne: Re: BGP peering strategies for smaller routers On Mon, May 2, 2016 at 10:35 PM, Eric Sabotta <esabotta@whdh.com> wrote:
I just did this with a ASR1001. I had to upgrade it to 8gb of ram (I got the real Cisco stuff for ~ $500). Before the router would crash when loading the tables.
Hi Eric, Something very fishy there because:
router1#show ip bgp summary BGP using 296,835,018 total bytes of memory
Commas added for clarity. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Tue, May 3, 2016 at 3:50 PM, Gustav Ulander <gustav.ulander@telecomputing.se> wrote:
Yes I can confirm that we also had the issue with the asr1001s. For us the router was fine until we upgraded it. When we rebooted it after the upgrade it ran out of memory when populating 2 full feeds. When we contacted TAC they confirmed that indeed it was a memory problem and that we would need to add more memory to the box.
Hi Gustav, IMO, you should not accept that answer from the TAC. An IOS release that crashes with two 600k BGP feeds in 4 gigs of RAM is badly defective. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
William Herrin wrote:
IMO, you should not accept that answer from the TAC. An IOS release that crashes with two 600k BGP feeds in 4 gigs of RAM is badly defective.
I suspect the time the OP would spend raging down the phone would be better spent sourcing a third party memory upgrade to 8G or 16G. The upgrade would certainly be the cheaper option of the two, in addition to being the only option with a useful outcome. Nick
On 03 May 2016, at 22:31, William Herrin <bill@herrin.us> wrote:
On Tue, May 3, 2016 at 3:50 PM, Gustav Ulander <gustav.ulander@telecomputing.se> wrote:
Yes I can confirm that we also had the issue with the asr1001s. For us the router was fine until we upgraded it. When we rebooted it after the upgrade it ran out of memory when populating 2 full feeds. When we contacted TAC they confirmed that indeed it was a memory problem and that we would need to add more memory to the box.
Hi Gustav,
IMO, you should not accept that answer from the TAC. An IOS release that crashes with two 600k BGP feeds in 4 gigs of RAM is badly defective.
Not necessarily. In essence, your physical memory gets halved in two after router boots up, then it may be further halved if you’re using features like SSO. So, with 4GB RAM config and with SSO running, you may be left with around 600-650MB free after boot and with IOS-XE loaded, and then all the features kick in. Including your BGP feeds that need around 300MB of memory just to store the tables, then there’s CEF RAM representation, and so on. Here’s a good WP w/r to memory usage & architecture on ASR 1k: http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregatio... It actually contains the same recommendation given by TAC - with recent/current code if you want to run full tables with BGP, get 8GB of RAM on ASR 1k. In the 3.10-3.12S era I believe it was still possible to fit (without the SSO) full tables in RAM and be fine. As Nick just responded, it’s faster to source the RAM or modify the config to cut down on number of BGP prefixes rather than ping back and forth here discussing all the possibilities. -- Łukasz Bromirski CCIE R&S/SP #15929, CCDE #2012::17, PGP Key ID: 0xFD077F6A
On Tue, May 3, 2016 at 5:13 PM, Łukasz Bromirski <lukasz@bromirski.net> wrote:
On 03 May 2016, at 22:31, William Herrin <bill@herrin.us> wrote:
IMO, you should not accept that answer from the TAC. An IOS release that crashes with two 600k BGP feeds in 4 gigs of RAM is badly defective.
Not necessarily.
In essence, your physical memory gets halved in two after router boots up, then it may be further halved if you’re using features like SSO. So, with 4GB RAM config and with SSO running, you may be left with around 600-650MB free after boot and with IOS-XE loaded, and then all the features kick in. Including your BGP feeds that need around 300MB of memory just to store the tables, then there’s CEF RAM representation, and so on.
Here’s a good WP w/r to memory usage & architecture on ASR 1k: http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregatio...
Hi Łukasz, You make some great points and that's an excellent document.
As Nick just responded, it’s faster to source the RAM or modify the config to cut down on number of BGP prefixes rather than ping back and forth here discussing all the possibilities.
I respectfully disagree. Sourcing more ram won't fix the next bit of sloppiness with the software. Or the one after that. Once the manager of that team starts to accept poor code quality, the only thing with a chance of fixing it is strong customer push-back. And it is poor code quality. Even slicing and dicing the ram in odd ways, there's just no excuse for an order-of-magnitude increase in ram required to run the same algorithms on the same data. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
William Herrin wrote:
I respectfully disagree. Sourcing more ram won't fix the next bit of sloppiness with the software. Or the one after that. Once the manager of that team starts to accept poor code quality, the only thing with a chance of fixing it is strong customer push-back.
You could feasibly argue that the box was shipped with too little RAM when it was released in 2008/2009. Alternatively, you could argue that 4G was fine way back then and that it's hardly surprising that it's too little now because the DFZ has grown from 220k prefixes to 620k. Anyway, this is all completely academic because the box is end-of-life and the chances of getting this problem fixed are zero. So there is simply no point in berating some poor TAC representative about this problem, end of story. The replacement box (ASR1001-X) has 8G ram. That's fine for today. In 8 years, it probably won't be fine.
And it is poor code quality. Even slicing and dicing the ram in odd ways, there's just no excuse for an order-of-magnitude increase in ram required to run the same algorithms on the same data.
If RAM were expensive, your argument would make sense, but RAM is not expensive. I don't really think it's any more viable to complain about an 8yo end-of-life box being ill equipped to handle today's Internet than it is to complain about the fact that you can't run a desktop operating system on a computer with 8 megs of ram, like you used to be able to. Truth is, 8 megs of RAM then was more expensive than 8 gigs of RAM these days. Can we move on now? Nick
On Tue, May 3, 2016 at 6:18 PM, Nick Hilliard <nick@foobar.org> wrote:
William Herrin wrote:
And it is poor code quality. Even slicing and dicing the ram in odd ways, there's just no excuse for an order-of-magnitude increase in ram required to run the same algorithms on the same data.
If RAM were expensive, your argument would make sense, but RAM is not expensive.
Hi Nick, You missed the point. Sloppy memory management is a "canary in a coal mine." It's a user-visible symptom that reflects poor code quality underneath. Programmers who don't care how much ram they're consuming are the same fools who catch and then ignore exceptions, don't bother evaluating the big-oh running time of their algorithms (often have no idea what that is) and engage in a variety of other bad practices that you as the customer suffer for but never directly see. It's not the cost of the ram, it's the attitude that ram is cheap so I won't care. It's a bad attitude, a dangerous attitude when found in a computer programmer. One which consistently leads to failure. If you challenge poor code quality when you spot it, your vendor might correct course. If you let it slide then by the time the code base is damaged enough for the pointy-hairs to understand there's a problem on their own, your only real choice will be to switch to a different product or vendor.
Can we move on now?
Sure, why not. I've proselytized enough for one day. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
---------------------------------------------------------------------------------------------------------------------- Hi Nick,
You missed the point. Sloppy memory management is a "canary in a coal mine." It's a user-visible symptom that reflects poor code quality underneath. Programmers who >don't care how much ram they're consuming are the same fools who catch and then ignore exceptions, don't bother evaluating the big-oh running time of their algorithms >(often have no idea what that is) and engage in a variety of other bad practices that you as the customer suffer for but never directly see.
That Cisco URL covering ASR1K memory details did mention that due to 64 bit, everything does use more memory. http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregatio... My biggest beef is that right off the bat, IOSd process only gets half the physical RAM. I'm not sure of that reasoning. Maybe to support ISSU with SW redundancy? Would be nice to be able to disable or tune that. I'm not sure what else that memory would be reserved for. It doesn't seem right that 2 full feeds works fine on an ISR with 768MB RAM, yet doesn't work on a 1K with 4 gigs of RAM. Chuck
Chuck Church wrote on 5/4/2016 12:14 PM:
---------------------------------------------------------------------------------------------------------------------- Hi Nick,
You missed the point. Sloppy memory management is a "canary in a coal mine." It's a user-visible symptom that reflects poor code quality underneath. Programmers who >don't care how much ram they're consuming are the same fools who catch and then ignore exceptions, don't bother evaluating the big-oh running time of their algorithms >(often have no idea what that is) and engage in a variety of other bad practices that you as the customer suffer for but never directly see.
That Cisco URL covering ASR1K memory details did mention that due to 64 bit, everything does use more memory. http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregatio... My biggest beef is that right off the bat, IOSd process only gets half the physical RAM. I'm not sure of that reasoning. Maybe to support ISSU with SW redundancy? Would be nice to be able to disable or tune that. I'm not sure what else that memory would be reserved for. It doesn't seem right that 2 full feeds works fine on an ISR with 768MB RAM, yet doesn't work on a 1K with 4 gigs of RAM.
Chuck
Given that the IOSd process runs under 32bit Linux (on the RP1), the 2GB allocation is probably a reflection of the max that Cisco could allocate to a single process in user space. The other 2GB (in a 4GB system) gets used, just not by the IOSd process. On the 8GB and 16GB versions of the RP2, I'm not sure why you'd maintain the 50/50 split. Perhaps it's not quite as simple as the document lets on.
Łukasz Bromirski wrote on 5/3/2016 4:13 PM:
On 03 May 2016, at 22:31, William Herrin <bill@herrin.us> wrote:
On Tue, May 3, 2016 at 3:50 PM, Gustav Ulander <gustav.ulander@telecomputing.se> wrote:
Yes I can confirm that we also had the issue with the asr1001s. For us the router was fine until we upgraded it. When we rebooted it after the upgrade it ran out of memory when populating 2 full feeds. When we contacted TAC they confirmed that indeed it was a memory problem and that we would need to add more memory to the box. Hi Gustav,
IMO, you should not accept that answer from the TAC. An IOS release that crashes with two 600k BGP feeds in 4 gigs of RAM is badly defective. Not necessarily.
In essence, your physical memory gets halved in two after router boots up, then it may be further halved if you’re using features like SSO. So, with 4GB RAM config and with SSO running, you may be left with around 600-650MB free after boot and with IOS-XE loaded, and then all the features kick in. Including your BGP feeds that need around 300MB of memory just to store the tables, then there’s CEF RAM representation, and so on.
Here’s a good WP w/r to memory usage & architecture on ASR 1k: http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregatio...
It actually contains the same recommendation given by TAC - with recent/current code if you want to run full tables with BGP, get 8GB of RAM on ASR 1k. In the 3.10-3.12S era I believe it was still possible to fit (without the SSO) full tables in RAM and be fine.
As Nick just responded, it’s faster to source the RAM or modify the config to cut down on number of BGP prefixes rather than ping back and forth here discussing all the possibilities.
I feel like you're trying to fit some other (possible, but far fetched) scenario from where we started. Mike, the op, said he has a 1002 which has an RP1 configured with 4GB of RAM. First, this is not a 1001. Second, SSO is off by default and is unlikely to be configured on an ASR1002 (the op certainly never stated he had enabled it). I just checked a few ASR 1k RP1s I have access to and the 3.10 IOS image has similar memory usage to the 3.13 (the latest MD release for this platform). On the RP1 platform, with a default only BGP feed, the systems have ~ 1.3GB of proc mem available. With a single full IP4 BGP feed, the free proc mem goes down to ~850MB. With two full feeds, the proc mem goes down to ~ 800MB. RP memory goes from ~1.8GB used (default only), ~2.7GB used (1x full), ~2.8GB used (2x full). This still leaves over 1GB of free RP memory with two full BGP feeds. The amount of FIB is dependent on the ESP installed by the OP; Mike hasn't yet stated what ESP he has installed. So yes, the 1002 can, and will continue to, work with two full BGP feeds when fitted with an ESP10.
I know this thread has been primarily about memory to hold the routing tables, but how well does it do with the BGP convergence time?? which could be the other killer with multiple full route tables. Carlos Alcantar Race Communications / Race Team Member 1325 Howard Ave. #604, Burlingame, CA. 94010 Phone: +1 415 376 3314 / carlos@race.com / http://www.race.com ________________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Blake Hudson <blake@ispn.net> Sent: Tuesday, May 3, 2016 3:23:42 PM To: nanog@nanog.org Subject: Re: BGP peering strategies for smaller routers Łukasz Bromirski wrote on 5/3/2016 4:13 PM:
On 03 May 2016, at 22:31, William Herrin <bill@herrin.us> wrote:
On Tue, May 3, 2016 at 3:50 PM, Gustav Ulander <gustav.ulander@telecomputing.se> wrote:
Yes I can confirm that we also had the issue with the asr1001s. For us the router was fine until we upgraded it. When we rebooted it after the upgrade it ran out of memory when populating 2 full feeds. When we contacted TAC they confirmed that indeed it was a memory problem and that we would need to add more memory to the box. Hi Gustav,
IMO, you should not accept that answer from the TAC. An IOS release that crashes with two 600k BGP feeds in 4 gigs of RAM is badly defective. Not necessarily.
In essence, your physical memory gets halved in two after router boots up, then it may be further halved if you’re using features like SSO. So, with 4GB RAM config and with SSO running, you may be left with around 600-650MB free after boot and with IOS-XE loaded, and then all the features kick in. Including your BGP feeds that need around 300MB of memory just to store the tables, then there’s CEF RAM representation, and so on.
Here’s a good WP w/r to memory usage & architecture on ASR 1k: http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregatio...
It actually contains the same recommendation given by TAC - with recent/current code if you want to run full tables with BGP, get 8GB of RAM on ASR 1k. In the 3.10-3.12S era I believe it was still possible to fit (without the SSO) full tables in RAM and be fine.
As Nick just responded, it’s faster to source the RAM or modify the config to cut down on number of BGP prefixes rather than ping back and forth here discussing all the possibilities.
I feel like you're trying to fit some other (possible, but far fetched) scenario from where we started. Mike, the op, said he has a 1002 which has an RP1 configured with 4GB of RAM. First, this is not a 1001. Second, SSO is off by default and is unlikely to be configured on an ASR1002 (the op certainly never stated he had enabled it). I just checked a few ASR 1k RP1s I have access to and the 3.10 IOS image has similar memory usage to the 3.13 (the latest MD release for this platform). On the RP1 platform, with a default only BGP feed, the systems have ~ 1.3GB of proc mem available. With a single full IP4 BGP feed, the free proc mem goes down to ~850MB. With two full feeds, the proc mem goes down to ~ 800MB. RP memory goes from ~1.8GB used (default only), ~2.7GB used (1x full), ~2.8GB used (2x full). This still leaves over 1GB of free RP memory with two full BGP feeds. The amount of FIB is dependent on the ESP installed by the OP; Mike hasn't yet stated what ESP he has installed. So yes, the 1002 can, and will continue to, work with two full BGP feeds when fitted with an ESP10.
I turned up a full ip4 feed on an RP1 today. Took approximately 5 minutes to fill the rib and probably another 5 minutes to push to the fib. The CLI responsiveness was noticeably slowed during this process, but the router didn't drop traffic. I'm guessing a second feed would involve fewer rib and fib changes and would converge faster.
On May 3, 2016, at 6:02 PM, Carlos Alcantar <carlos@race.com> wrote:
I know this thread has been primarily about memory to hold the routing tables, but how well does it do with the BGP convergence time?? which could be the other killer with multiple full route tables.
Blake,
On 04 May 2016, at 00:23, Blake Hudson <blake@ispn.net> wrote:
Łukasz Bromirski wrote on 5/3/2016 4:13 PM:
On 03 May 2016, at 22:31, William Herrin <bill@herrin.us> wrote:
On Tue, May 3, 2016 at 3:50 PM, Gustav Ulander <gustav.ulander@telecomputing.se> wrote:
Yes I can confirm that we also had the issue with the asr1001s.
[...]
I feel like you're trying to fit some other (possible, but far fetched) scenario from where we started.
Yeah, sorry for that - saw 1001 in quote and kept that as original platform. For 1002 with SSO off you may be fine, sure. BTW, the versions you're quoting as working were also quoted by me as the ones that could have been OK even on the 1001 (I know, I know). -- Łukasz Bromirski CCIE R&S/SP #15929, CCDE #2012::17, PGP Key ID: 0xFD077F6A
Hello. Yea its a bit strange that we would need to add memory to the platform just because we upgraded the image. But since we were in kind of a tight spot we opted to upgrade it and then move away from the platform in that role. //Gustav -----Ursprungligt meddelande----- Från: William Herrin [mailto:bill@herrin.us] Skickat: den 3 maj 2016 22:32 Till: Gustav Ulander <gustav.ulander@telecomputing.se> Kopia: Eric Sabotta <esabotta@whdh.com>; NANOG list <nanog@nanog.org> Ämne: Re: BGP peering strategies for smaller routers On Tue, May 3, 2016 at 3:50 PM, Gustav Ulander <gustav.ulander@telecomputing.se> wrote:
Yes I can confirm that we also had the issue with the asr1001s. For us the router was fine until we upgraded it. When we rebooted it after the upgrade it ran out of memory when populating 2 full feeds. When we contacted TAC they confirmed that indeed it was a memory problem and that we would need to add more memory to the box.
Hi Gustav, IMO, you should not accept that answer from the TAC. An IOS release that crashes with two 600k BGP feeds in 4 gigs of RAM is badly defective. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Mon, May 2, 2016 at 3:07 PM, Mike <mike-nanog@tiedyenetworks.com> wrote:
I have an ASR1000 router with 4gb of ram. The specs say I can get '1 million routes' on it, but as far as I have been advised, a full table of internet routes numbers more than 530k by itself, so taking 2 full tables seems to be out of the question (?).
Hi Mike, There is no "routing table." Instead: Routing Information Bases (RIB) Forwarding Information Base (FIB) RIB = the routing data from your neighbors. Each protocol (e.g. BGP, OSPF, static, connected) has a RIB. If you have virtual routers (VRF) then each VRF has its own RIBs. For the BGP RIB, each prefix (route) from each neighbor will need its own slot in the RIB. FIB = the next hop table. Only the best next hop for each prefix is stored in the FIB. One FIB per protocol (IPv4, IPv6), unless you run VRFs, then one FIB each per VRF. All routers have both a FIB and RIBs. All routers keep the RIBs in DRAM. Big Iron like yours typically store the FIB in special hardware called Ternary Content Addressable Memory (TCAM). You're looking at two physically different kinds of memory in your router to store the RIB and the FIB. No matter how much DRAM you have, you only have so much TCAM in that routing engine. The TCAM is not upgradable. In fact, it's buried under a big heat sink. All the RIBs are processed down to a single FIB for each protocol/VRF. The best next hop is selected from all the possibilities in the RIBs and only that one gets stored in the FIB. The FIB in the TCAM is consulted every single time a packet is routed. The RIBs are only consulted when new information is received from a neighbor of when the FIB table needs to be rebuilt. The RIB is not consulted to route individual packets. Your "million routes" is a reflection of the TCAM on your routing engine. The FIB table size. 4G DRAM is enough for 10-15 million routes in the various RIBs. If you're running a single VRF, you have enough FIB headroom for the next two or three years, until the v4 and v6 BGP tables add up to around 900M prefixes. You already have too little FIB headroom to run two BGP-speaking VRFs. My piddly little 2811 carries a full IPv4 table, 580k routes. Its BGP RIB consumes all of 154 megabytes of DRAM. Unlike your router, the 2811 does not have a "hardware fast path." No TCAM. It stores the FIB in a radix tree in DRAM instead. As a result, it can only handle low data rates, in the 10s of megabits per second. I'm leaving out lots of details, but these are the most important with respect to your question. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
participants (15)
-
Blake Hudson -
Bob Evans -
Carlos Alcantar -
Chuck Church -
Eric Sabotta -
Gustav Ulander -
James Milko -
lincoln dale -
Mark Tinka -
Mike -
Nick Hilliard -
Richard Hicks -
Tony Wicks -
William Herrin -
Łukasz Bromirski