max-prefix and platform tcam limits: they are things
Submitted without comment: http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/ -Tk
I know that I should know better then comment on networks others then my own, ( and I know to never comment on my own publicly :) ) But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this? I won't even get into max-prefix and how we've managed this long with someone people still not setting them. -jim On Fri, Oct 5, 2012 at 7:31 PM, Anton Kapela <tkapela@gmail.com> wrote:
Submitted without comment: http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/
-Tk
On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said:
But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this?
If the device was only expecting 2K or so internal routes, getting hit with the 440K routes in the DFZ would be 210x....
Yes that math would work, but if your device can't handle 1x Internet routing and your running without some serious max-prefix/filters it says even more about your IP eng team then I'd be willing to comment on. -jim On Fri, Oct 5, 2012 at 9:17 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said:
But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this?
If the device was only expecting 2K or so internal routes, getting hit with the 440K routes in the DFZ would be 210x....
On Fri, 5 Oct 2012, jim deleskie wrote: Just ask yourself how many times you have seen a Godaddy IP/NOC person post anything to NANOG or to any other technical forum? -Hank
Yes that math would work, but if your device can't handle 1x Internet routing and your running without some serious max-prefix/filters it says even more about your IP eng team then I'd be willing to comment on.
-jim
On Fri, Oct 5, 2012 at 9:17 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said:
But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this?
If the device was only expecting 2K or so internal routes, getting hit with the 440K routes in the DFZ would be 210x....
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/5/2012 8:17 PM, Valdis.Kletnieks@vt.edu wrote:
On Fri, 05 Oct 2012 21:05:07 -0300, jim deleskie said:
But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this?
If the device was only expecting 2K or so internal routes, getting hit with the 440K routes in the DFZ would be 210x....
On outages GoDaddy provided a tiny bit more information. [quote] Obviously the explanation of the incident had to be consumed by the general public, however we encountered an unknown bug that was found which started the domino effect. Aside from this group, that level of detail wouldn't be understood by a majority of the recipients. With that said, please feel free to take this off list with Jason or Myself. Mike Dob Manager, Network Engineering [/quote] No information has been provided on what sort of "unknown bug" this was. A bug in code that GoDaddy wrote? A bug in their route servers or router OS, which others may also use and might want to be aware of? - -DMM -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQb32GAAoJECp6zT7OFmGa5wYIAIWp9vUwS/5zM73cAXlUrpwR 5U3XuUn3fasq8JyuNFhe99aDhkQY+i5tQEFhhhB60dVfWcyVGYsO1Ny0FMXupYfE Ely29vxutWHMDxX39XTvmmtNkeSsZ2cOtkqF14If+43/CccrDwDDiC06YoSyxb/x JEjWMhcthcw8rbndzF3P+bRCerdyxPpeQLzNy+l0/SbjobsLwzDA28CPW2kL82Bh 67dgqdXiMVFARC8rc91bYAoJ+NtkLs/GwYSbgXdNCk5dGrZvOk1rVWzaKxBrNV8T rldU43GRzeq2bJAKo0fx17/HE4j9qlfeBIW+bihwgkMpzw8p3kRS9S0WU4cGxGM= =1nls -----END PGP SIGNATURE-----
On Fri, 5 Oct 2012, jim deleskie wrote:
I know that I should know better then comment on networks others then my own, ( and I know to never comment on my own publicly :) )
But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this?
Is it plausible that Godaddy's internal network only normally has a few thousand BGP routes? 210 x a few thousand would run most modern gear out of FIB space. The "my DNS is broken, are we really being DDoS'd on udp/53 at the same time?" thing, I've seen, and I can imagine it being very confusing to someone seeing it for the first time. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On 10/5/12 5:05 PM, jim deleskie wrote:
I know that I should know better then comment on networks others then my own, ( and I know to never comment on my own publicly :) )
But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this?
it's pretty easy to inadvertently leak a copy of the internet from one vrf to another and effectively install two copies of the internet routes in your fib... There are plently of cases where you might to that or something similar on purpose, which is all good and well if you have 2million route fib capacity but less awesome if you have 512K route capacity linecards at this point. if you get those routes from a private peer on some non-internet-vrf well that might imply that your filter policy needs some tuning.
I won't even get into max-prefix and how we've managed this long with someone people still not setting them.
-jim On Fri, Oct 5, 2012 at 7:31 PM, Anton Kapela <tkapela@gmail.com> wrote:
Submitted without comment: http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/
-Tk
In case you missed it On Oct 5, 2012, at 7:05 PM, jim deleskie <deleskie@gmail.com> wrote:
I know that I should know better then comment on networks others then my own, ( and I know to never comment on my own publicly :) )
But here goes, 210x the size of normal really? 210% I'd have a hard time believing. Did anyone else anywhere see a route leak equal to larger then the entire Internet that day, anywhere else that could of caused this?
I won't even get into max-prefix and how we've managed this long with someone people still not setting them.
-jim On Fri, Oct 5, 2012 at 7:31 PM, Anton Kapela <tkapela@gmail.com> wrote:
Submitted without comment: http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/
-Tk
participants (8)
-
Anton Kapela -
David Miller -
Hank Nussbacher -
jim deleskie -
joel jaeggli -
Jon Lewis -
Lane Powers -
Valdis.Kletnieks＠vt.edu