Looks like the 45k mark was reached: Folks with 7000's and SSE's should start monitoring their memory utilization via "show sse summary". Tony
Hmm, it's not news for us. 45K can hold core routing only as inter-back-bone router, not more. But why, why this crasy CISCO could not predict future when they designed 45K routers? It was not difficult for them develop this box to cary 64 or 128MB RAM.
Looks like the 45k mark was reached:
Folks with 7000's and SSE's should start monitoring their memory utilization via "show sse summary".
Tony
--- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
Hmm, it's not news for us. 45K can hold core routing only as inter-back-bone router, not more. But why, why this crasy CISCO could not predict future when they designed 45K routers? It was not difficult for them develop this box to cary 64 or 128MB RAM.
Looks like the 45k mark was reached:
Folks with 7000's and SSE's should start monitoring their memory utilization via "show sse summary".
There's a couple of comments here: First, 45k is not the limit. More like 60k. You'll pardon me for being cautious. The limitation is not DRAM. It's the 64k words of SRAM that the SSE uses for its high speed forwarding table. You don't want to pay for 64Mbytes of SRAM. ;-) When cisco's engineers designed the SSE, we knew very well what was happening. We expected to be given the opportunity to produce subsequent hardware which implemented the SSE in an ASIC. If, by that time, CIDR hadn't killed off the exponential growth, we would have expanded the address space. Unfortunately, cisco management decided that the SSE ASIC should not be implemented (a mistake which, to my knowledge, cisco has not corrected). Thus, the 7500 exists without an SSE. Tony
not to make this too cisco specific, but... the number of entries in the forwarding cache on the sse is generally more than the number of routes in the bgp rib (because of the way the cache handles more-specifics of over-lapping aggregates). so in addition to the raw number of routes in the rib, the efficiency (and scope) of aggregation are also important data points now a question. what does an sse do when its cache fills? it used to(*) bring down the whole sse, which doesn't really make much sense given that it's a cache and therefore it's normal operation for it to be incomplete. anyone know an answer to that one? /jws (*) -- "used to" implies that it happened before, which it did. but that was largely due to the ineffeciency of the data structures, and the problem was solved such that 2 years (and counting) was added to its life
Hmm, it's not news for us. 45K can hold core routing only as inter-back-bone router, not more.
But why, why this crasy CISCO could not predict future when they designed 45K routers? It was not difficult for them develop this box to cary 64 or 128MB RAM.
Looks like the 45k mark was reached:
Folks with 7000's and SSE's should start monitoring their memory utilization via "show sse summary".
There's a couple of comments here:
First, 45k is not the limit. More like 60k. You'll pardon me for being cautious.
The limitation is not DRAM. It's the 64k words of SRAM that the SSE uses for its high speed forwarding table. You don't want to pay for 64Mbytes of SRAM. ;-)
When cisco's engineers designed the SSE, we knew very well what was happening. We expected to be given the opportunity to produce subsequent hardware which implemented the SSE in an ASIC. If, by that time, CIDR hadn't killed off the exponential growth, we would have expanded the address space. Unfortunately, cisco management decided that the SSE ASIC should not be implemented (a mistake which, to my knowledge, cisco has not corrected). Thus, the 7500 exists without an SSE.
Tony
BTW, really all data incorporated from BGP protocl for 3 - 8 core routings do not need more than 32MB RAM; the fact CISCO need more than 32MB ram for core routing is due to inefficient data structures... Exactly it's because they (I think) have solved some other task (not minimize memory, but CPU and development time). What is really 100K routes? 100K * ( 8 bytes - address + maska 4 - pointer to interface 4 - pointer to ASPATH (ASPATHES are really shared by many different routes; You have not 1000,000 different AS PATHES at all) 8 - accounting 8 - some other, ) - it's about 32 bytes in this example. Of cource, it's not true - really we can expect about 64 bytes/route. 64 bytes * 100K = 6 Mb RAM. I can propose it's nessesary 128bytes - it'll be 12MB RAM. But anyway, if somebody designes specially INTERNET CORE router, he can hold any core routing tables in 16 MB RAM, not more. There is a lot of different ways to minimise memory usage. We are far from the theoretical memory leak; the more serious problem may be efficiensy etc..., but there is CISCO FUTIONS and other ideas to hold CPU usage in acceptable range. I am shure we'll be the witnesses of the such events as it have been last year when CISCO's engeneers have fixed serious BGP problem in 1 or 2 nights. Through this way (to solve hardware problems by software) have it's limits. --- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
BTW, really all data incorporated from BGP protocl for 3 - 8 core routings do not need more than 32MB RAM; the fact CISCO need more than 32MB ram for core routing is due to inefficient data structures... Exactly it's because they (I think) have solved some other task (not minimize memory, but CPU and development time).
What is really 100K routes?
<snip>
Of cource, it's not true - really we can expect about 64 bytes/route.
64 bytes * 100K = 6 Mb RAM. I can propose it's nessesary 128bytes - it'll be 12MB RAM.
Umm - also, you have memory used for the IP routing table. I don't know how big any of these data structures are - but I do know that it seems that the IP routing table representation uses even more ram than than the BGP routing table version. Also, what about overhead? OK, let's say you can share memory for common AS-PATHs. But how do you find them? How do you store them? Remember, you want to be able to add or delete 14k routes in a few seconds at most. Then, what about buffering? If a switch you're hooked to burps, you need to (want to) have enough memory to buffer that (though Ciscos like to do it on the cards, I think).
But anyway, if somebody designes specially INTERNET CORE router, he can hold any core routing tables in 16 MB RAM, not more. There is a lot of different ways to minimise memory usage.
A 2501 is/was within a few k of being able to hold a core routing table in 16mb (w/ the OS image running out of flash). But it lacks memory and cpu to then switch packets - and if someone adds a few K extra routes god help you.
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
tli or pferguso might be able to shed some light on the actual numbers, assuming he has the time and interest :) Avi
tli or pferguso might be able to shed some light on the actual numbers, assuming he has the time and interest :) cisco might consider those numbers a trade secret, so I'll decline. I'll simply point out that yours truly worked quite hard at conserving memory. As you correctly point out, there is indeed the inevitable time space tradeoff, and so some memory inefficiency is inevitable as there's only a finite amount of time (always insufficient, natch ;-). You might also recall that there's forwarding table, the cache, and then the SSE data structures. BGP is very efficient as a whole. The problem is that it's just a big messy problem. Tony
In cisco.external.nanog you write:
not to make this too cisco specific, but...
the number of entries in the forwarding cache on the sse is generally more than the number of routes in the bgp rib (because of the way the cache handles more-specifics of over-lapping aggregates). so in addition to the raw number of routes in the rib, the efficiency (and scope) of aggregation are also important data points
now a question. what does an sse do when its cache fills? it used to(*) bring down the whole sse, which doesn't really make much sense given that it's a cache and therefore it's normal operation for it to be incomplete. anyone know an answer to that one?
SSE memory usage is monitored and if it falls below a particular threshold, the ager will become more aggressive in aging entries.. --ravi
/jws
(*) -- "used to" implies that it happened before, which it did. but that was largely due to the ineffeciency of the data structures, and the problem was solved such that 2 years (and counting) was added to its life
Hmm, it's not news for us. 45K can hold core routing only as inter-back-bone router, not more.
But why, why this crasy CISCO could not predict future when they designed 45K routers? It was not difficult for them develop this box to cary 64 or 128MB RAM.
Looks like the 45k mark was reached:
Folks with 7000's and SSE's should start monitoring their memory utilization via "show sse summary".
There's a couple of comments here:
First, 45k is not the limit. More like 60k. You'll pardon me for being cautious.
The limitation is not DRAM. It's the 64k words of SRAM that the SSE uses for its high speed forwarding table. You don't want to pay for 64Mbytes of SRAM. ;-)
When cisco's engineers designed the SSE, we knew very well what was happening. We expected to be given the opportunity to produce subsequent hardware which implemented the SSE in an ASIC. If, by that time, CIDR hadn't killed off the exponential growth, we would have expanded the address space. Unfortunately, cisco management decided that the SSE ASIC should not be implemented (a mistake which, to my knowledge, cisco has not corrected). Thus, the 7500 exists without an SSE.
Tony
First, Happy New Year. 2'th, sorry for my mistake - I just meant 45K as Cisco 4500... Through it have not changed issue seriously -:) 3'nd.. Do you mean VIP2 idea is worst then SSE idea?
Hmm, it's not news for us. 45K can hold core routing only as inter-back-bone router, not more.
But why, why this crasy CISCO could not predict future when they designed 45K routers? It was not difficult for them develop this box to cary 64 or 128MB RAM.
> Looks like the 45k mark was reached: > > Folks with 7000's and SSE's should start monitoring their memory > utilization via "show sse summary".
There's a couple of comments here:
First, 45k is not the limit. More like 60k. You'll pardon me for being cautious.
The limitation is not DRAM. It's the 64k words of SRAM that the SSE uses for its high speed forwarding table. You don't want to pay for 64Mbytes of SRAM. ;-)
When cisco's engineers designed the SSE, we knew very well what was happening. We expected to be given the opportunity to produce subsequent hardware which implemented the SSE in an ASIC. If, by that time, CIDR hadn't killed off the exponential growth, we would have expanded the address space. Unfortunately, cisco management decided that the SSE ASIC should not be implemented (a mistake which, to my knowledge, cisco has not corrected). Thus, the 7500 exists without an SSE.
Tony
--- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
3'nd.. Do you mean VIP2 idea is worst then SSE idea? The VIP2 is a fine idea. However, it's again using conventional microprocessor technology for forwarding. And unfortunately, it needs to be fairly high end processor technology to sustain the rates. The problem is that the traffic demand far exceeds the growth rate of processor forwarding. So the problem is that as we approach higher speed (post-7500) interfaces, you're forced into more specialized hardware. At this point, not having an SSE ASIC becomes somewhat silly. The rest of the board is ASICs, which in volume end up cheaper than processors... Tony
3'nd.. Do you mean VIP2 idea is worst then SSE idea?
The VIP2 is a fine idea. However, it's again using conventional microprocessor technology for forwarding. And unfortunately, it needs to be fairly high end processor technology to sustain the rates. The problem is that the traffic demand far exceeds the growth rate of processor forwarding.
So the problem is that as we approach higher speed (post-7500) interfaces, you're forced into more specialized hardware. At this point, not having an SSE ASIC becomes somewhat silly. The rest of the board is ASICs, which in volume end up cheaper than processors...
Tony
! Sorry, my crazy Win95 (I HATE MS!) drop me from TELNET 3 times when ! I did attempted to answer this from the home -:). Really, there is one important question (not the blame about IOS's memory or so on) - does hardware vendors (just as CISCO etc) really predicts the future? When CISCO developed CS4500, they have decided 32MB RAM would be enougph just as MIPS processor they established. Just now they (seems) think it'll be enougph new MIPS and 128MB RAM. It's amazing but I can easy upgrade my 1,000$ IBM PC up to 256MB RAM, but I can't do it for my CISCO routers (about 10 - 30,000$) at all. It's easy to install 2 or 4 CPU into SUN ULTRA-2 or SGI SERVER computers, but it's impossible to do it with routers. And so on. What's about switching itself - I am not shure they (vendors) are on the wrong way. Seems CISCO have well defined plan - they are moving toward from 1 CPU doing everything (CS2500 and CS4700) to (1 CPU for routing, few CPU for the switching - VIP2) and to (1 CPU for the Routing, and many switches for the switching - MPOATM, TAG Switching, etc) - they are trieing to build mixed _routers + ATM_NETWORK_ backbone when routers would decide how to forward some virtual streams (TCP connections etc) and switches would forward the packets toward their destinations. And this mean we'll have more troubles with the routers, not switches. There is a lot of ways to increase switching capability (look on Stratocom, for example); but the routers itself are far beyong. We can easyly increase the CPU and Memory for the UNIX Servers; it's not difficult to build server with 512MB RAM, 4x300 MHZ CPU, 10 PCI or SBUS cards; but no one router can be expanded easyly. I wonder why it's so many discussion there about DRAM/SRAM/etc..., there exist fast and nonblocking switches just now; and moreover, you can install them in parallel because no one customer need 200Mbit data links itself /this means you can use 4x100 mbit links instead of 1x400Mbit withouth the lost of quality/. But what we'll do with the routers next 2 years? Just now existing CS7500 and CS4700 are 30% - 50% loaded by ROUTING (not switching but routing) and there is not ways to scale them easily. Or we'll go back to the UNIX boxes + GATED? Sometimes I guess this is true _:) --- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
my CISCO routers (about 10 - 30,000$) at all. It's easy to install 2 or 4 CPU into SUN ULTRA-2 or SGI SERVER computers, but it's impossible to do it with routers. And so on.
Well, it's also somewhat more complicated to distribute the tasks in a router to multiple CPUs. There are plenty of ways to break Bays by accidentally doing that distribution wrong.
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
Avi
Yes, no doubt. But I'd like to see router's vendors predicted future better than in the past. We all know few serious mistakes CISCO did - with the memory size in CS4500, with CS7000 and CS7010 routers; and I am afraid to fail into new trap in future...
my CISCO routers (about 10 - 30,000$) at all. It's easy to install 2 or 4 CPU into SUN ULTRA-2 or SGI SERVER computers, but it's impossible to do it with routers. And so on.
Well, it's also somewhat more complicated to distribute the tasks in a router to multiple CPUs.
There are plenty of ways to break Bays by accidentally doing that distribution wrong.
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
Avi
--- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
On Fri, 3 Jan 1997 alex@relcom.EU.net wrote: |} Yes, no doubt. But I'd like to see router's vendors predicted future |} better than in the past. We all know few serious mistakes CISCO did - |} with the memory size in CS4500, with CS7000 and CS7010 routers; and I |} am afraid to fail into new trap in future... Alex - I'm sure there were the visionaries at Cisco with their heads in the clouds who were telling the various marketing and engineering bodies to build a box like the 7500 or BFR some time ago. However, proving the market would be there and that there would actually be a customer base for such a product is a little more difficult. I believe, that only in the past 1-2yrs has ISP revenue actually accounted for a good portion of Cisco's overall revenue. (Perhaps someone from Cisco or someone with their financial statement can speak to this) Building boxes that require a huge amount of R&D and won't sell more than < 5,000 of isn't a very profitable business. Cisco should be thankful that the overall majority of their ISP user base is rather understanding when it comes to performance limitations, discovering and working around software issues, etc. etc. -jh-
In cisco.external.nanog you write:
Yes, no doubt. But I'd like to see router's vendors predicted future better than in the past. We all know few serious mistakes CISCO did - with the memory size in CS4500, with CS7000 and CS7010 routers; and I am afraid to fail into new trap in future...
Yep.. we are hearing you. Hopefully we will not fall into the same trap again. --ravi
my CISCO routers (about 10 - 30,000$) at all. It's easy to install 2 or 4 CPU into SUN ULTRA-2 or SGI SERVER computers, but it's impossible to do it with routers. And so on.
Well, it's also somewhat more complicated to distribute the tasks in a router to multiple CPUs.
There are plenty of ways to break Bays by accidentally doing that distribution wrong.
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
Avi
--- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
Really, there is one important question (not the blame about IOS's memory or so on) - does hardware vendors (just as CISCO etc) really predicts the future? There's a complicated answer. Yes, certain people at router vendors do understand and can predict what is necessary. However, this does not imply that the right thing happens. Please recall that most (all?) router manufacturers are driven by the enterprise market, not the ISP market. Memory constraints in the enterprise market are not an issue. Memory costs (and even the cost of the extra SIMM socket) _are_ an issue, as they affect the price that everyone on the street pays. This either cuts into manufacturers profit margins or into their sales. Thus, router manufacturers can indeed predict what's needed for the ISP market. But history shows that they consistently make a business decision to design hardware for the enterprise market. It's easy to install 2 or 4 CPU into SUN ULTRA-2 or SGI SERVER computers, but it's impossible to do it with routers. And so on. In fact, it's not impossible. That's _exactly_ what you're doing when you install a VIP. But what we'll do with the routers next 2 years? Just now existing CS7500 and CS4700 are 30% - 50% loaded by ROUTING (not switching but routing) and there is not ways to scale them easily. Again, getting the switching off of the CPU is key. Once that happens consistently, then there is much more CPU for routing computation. 30-50% for routing is fine once that happens. It just implies that we need to scale up the processor to match the growth in _routing_ traffic. Fortunately, that's _FAR_ lower than transit traffic, so I believe it's a tractable problem. Just gotta stay on the processor curve. Tony
In cisco.external.nanog you write:
3'nd.. Do you mean VIP2 idea is worst then SSE idea?
The VIP2 is a fine idea. However, it's again using conventional microprocessor technology for forwarding. And unfortunately, it needs to be fairly high end processor technology to sustain the rates. The problem is that the traffic demand far exceeds the growth rate of processor forwarding.
So the problem is that as we approach higher speed (post-7500) interfaces, you're forced into more specialized hardware. At this point, not having an SSE ASIC becomes somewhat silly. The rest of the board is ASICs, which in volume end up cheaper than processors...
Tony
! Sorry, my crazy Win95 (I HATE MS!) drop me from TELNET 3 times when ! I did attempted to answer this from the home -:). Really, there is one important question (not the blame about IOS's memory or so on) - does hardware vendors (just as CISCO etc) really predicts the future? When CISCO developed CS4500, they have decided 32MB RAM would be enougph just as MIPS processor they established. Just now they (seems) think it'll be enougph new MIPS and 128MB RAM. It's amazing but I can easy upgrade my 1,000$ IBM PC up to 256MB RAM, but I can't do it for my CISCO routers (about 10 - 30,000$) at all. It's easy to install 2 or 4 CPU into SUN ULTRA-2 or SGI SERVER computers, but it's impossible to do it with routers. And so on.
What's about switching itself - I am not shure they (vendors) are on the wrong way. Seems CISCO have well defined plan - they are moving toward from 1 CPU doing everything (CS2500 and CS4700) to (1 CPU for routing, few CPU for the switching - VIP2) and to (1 CPU for the Routing, and many switches for the switching - MPOATM, TAG Switching, etc) - they are trieing to build mixed _routers + ATM_NETWORK_ backbone when routers would decide how to forward some virtual streams (TCP connections etc) and switches would forward the packets toward their destinations.
And this mean we'll have more troubles with the routers, not switches. There is a lot of ways to increase switching capability (look on Stratocom, for example); but the routers itself are far beyong. We can easyly increase the CPU and Memory for the UNIX Servers; it's not difficult to build server with 512MB RAM, 4x300 MHZ CPU, 10 PCI or SBUS cards; but no one router can be expanded easyly. I wonder why it's so many discussion there about DRAM/SRAM/etc..., there exist fast and nonblocking switches just now; and moreover, you can install them in parallel because no one customer need 200Mbit data links itself /this means you can use 4x100 mbit links instead of 1x400Mbit withouth the lost of quality/. But what we'll do with the routers next 2 years? Just now existing CS7500 and CS4700 are 30% - 50% loaded by ROUTING (not switching but routing) and there is not ways to scale them easily.
As Tony said, the solution is to decouple the RP and allow it to just do routing.. Then we should be able to contain the route processing requirement within the CPU technology growth curve. If you are interested about the RSP/7500/VIP2s, you should talk to your account team.. some interesting things targeted exclusively for the ISPs are happening.. --ravi
Or we'll go back to the UNIX boxes + GATED? Sometimes I guess this is true _:)
--- Aleksei Roudnev, Network Operations Center, Relcom, Moscow (+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager) (+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
On Mon, 30 Dec 1996 16:26:23 -0800 (PST) Tony Li <tli@jnx.com> alleged:
Looks like the 45k mark was reached:
Folks with 7000's and SSE's should start monitoring their memory utilization via "show sse summary".
Nah, they should be trading them in for Pentium-Pro's and running BSD-Unix with gated :-) Mass Cisco Car boot sale -> Film at 23:00 :-) Regards, Neil. -- Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
participants (7)
-
alex＠relcom.EU.net -
Avi Freedman -
John W. Stewart III -
Jonathan Heiliger -
Neil J. McRae -
Ravi Chandra -
Tony Li