Thinking Methodically about building a PoC
hi, I am asked to build a large lab/test it. I'm provided crazy scale numbers for lots of technologies (L*VPN, IPv*, IGP*, All Tunnels flavors...etc). It took me a lot of time to build this lab, because when I got the request/test plan handed over to me, I did not verify that these scaled numbers are even possible, not to mention the combination. I assumed some thought/research were done before. I'm trying to put together a list of the lessons learned, and the right way to do this for future reference, specially that this project was time critical and I got beaten hard because I did not deliver on time. So my question is, in your extensive experience, what is the right method/approach to this kind of task: 1) Get started immediately (MVP), things will break, tune it along the way. 2) Do some planning and research first. I'd appreciate any references to 'software engineering' or other industries/ Thanks
On Sun, Jun 12, 2016 at 9:49 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 13 Jun 2016, at 8:52, Kasper Adel wrote:
2) Do some planning and research first.
This.
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
We never design in a vacuum. There's always some target we're designing towards. Testing is no different. Think about what it is you'll need to support. Look at historical numbers related to those features/capabilities. Yes, as the stork market keeps reminding us, past performance is no guarantee of future results...but at the same time, those who don't learn from the past are doomed to re-implement it...poorly. So, when we test, we look at protocols we've already been running for years, and then we look at the growth curves we've seen in those protocols over the past X years, where X is approximately the estimated lifespan of the hardware in question. So, if the current router platform you're looking to replace has been in place in your network for 8 years, and you're testing the next generation for BGP route scaling, look at what the global BGP table size was 8 years ago, and look at where it is today; work out the percentage growth curve for it; then take the current BGP table size, apply the same compound growth percentage to it for the next 8 years, and you'll come up with a reasonable idea of the scale you'll need the box to handle over its lifetime. Test that; then, to give yourself a margin of error, double the number, and test again. That way you have a realistic idea of whether it can support your current growth rate, and whether it can support your growth if the growth rate is 1.4x what you expect. Do those calculations for each of the protocols under test, and you'll be able to come up with a reasonable testing profile that's supportable based on historical information, rather than flights of fancy. Hope that helps! Matt
Agreed, the margin of growth never seems to stay consistent or be what you predict. Instead of asking "What do you want?" (or taking their specs). I have lots of ideas of what I want but maybe not what I need. Ask these questions, they've gotten me better answers and have allowed me to do the hardware/software framework not the end user/client. "What are you trying to sell?" "What are you trying to do?" Brian ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Matthew Petach <mpetach@netflight.com> Sent: Sunday, June 12, 2016 11:20 PM To: Roland Dobbins Cc: NANOG list Subject: Re: Thinking Methodically about building a PoC On Sun, Jun 12, 2016 at 9:49 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 13 Jun 2016, at 8:52, Kasper Adel wrote:
2) Do some planning and research first.
This.
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
We never design in a vacuum. There's always some target we're designing towards. Testing is no different. Think about what it is you'll need to support. Look at historical numbers related to those features/capabilities. Yes, as the stork market keeps reminding us, past performance is no guarantee of future results...but at the same time, those who don't learn from the past are doomed to re-implement it...poorly. So, when we test, we look at protocols we've already been running for years, and then we look at the growth curves we've seen in those protocols over the past X years, where X is approximately the estimated lifespan of the hardware in question. So, if the current router platform you're looking to replace has been in place in your network for 8 years, and you're testing the next generation for BGP route scaling, look at what the global BGP table size was 8 years ago, and look at where it is today; work out the percentage growth curve for it; then take the current BGP table size, apply the same compound growth percentage to it for the next 8 years, and you'll come up with a reasonable idea of the scale you'll need the box to handle over its lifetime. Test that; then, to give yourself a margin of error, double the number, and test again. That way you have a realistic idea of whether it can support your current growth rate, and whether it can support your growth if the growth rate is 1.4x what you expect. Do those calculations for each of the protocols under test, and you'll be able to come up with a reasonable testing profile that's supportable based on historical information, rather than flights of fancy. Hope that helps! Matt
On Jun 13, 2016, at 12:49 AM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 13 Jun 2016, at 8:52, Kasper Adel wrote:
2) Do some planning and research first.
This.
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
I'll second that! How can you do it any other way and get any sort of reliable data...especially a POC. Seems like you would waste a lot of time just plodding forward without doing the research.
This may not be an answer very specific to your problem/question, but if you take a look at the following image, you will find a summary of what they called the engineering design methodology: http://www.cdn.sciencebuddies.org/Files/5083/9/2013-updated_engineering-meth... You can adapt it to your circumstances, for example: instead of defining a problem in step 1, you can define a product, and after knowing what is expected from that product, you can then move to background research, etc. Hope that helps. Rafael On Sun, Jun 12, 2016 at 8:52 PM, Kasper Adel <karim.adel@gmail.com> wrote:
hi,
I am asked to build a large lab/test it. I'm provided crazy scale numbers for lots of technologies (L*VPN, IPv*, IGP*, All Tunnels flavors...etc).
It took me a lot of time to build this lab, because when I got the request/test plan handed over to me, I did not verify that these scaled numbers are even possible, not to mention the combination. I assumed some thought/research were done before.
I'm trying to put together a list of the lessons learned, and the right way to do this for future reference, specially that this project was time critical and I got beaten hard because I did not deliver on time.
So my question is, in your extensive experience, what is the right method/approach to this kind of task:
1) Get started immediately (MVP), things will break, tune it along the way. 2) Do some planning and research first.
I'd appreciate any references to 'software engineering' or other industries/
Thanks
On Mon 2016-Jun-13 08:52:41 -0500, Possamai Rafael via NANOG <nanog@nanog.org> wrote:
This may not be an answer very specific to your problem/question, but if you take a look at the following image, you will find a summary of what they called the engineering design methodology:
http://www.cdn.sciencebuddies.org/Files/5083/9/2013-updated_engineering-meth...
Seriously thought initially that you were going to link to: http://i2.wp.com/tamingdata.com/wp-content/uploads/2010/07/tree-swing-projec...
You can adapt it to your circumstances, for example: instead of defining a problem in step 1, you can define a product, and after knowing what is expected from that product, you can then move to background research, etc.
Hope that helps.
Rafael
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
On Sun, Jun 12, 2016 at 8:52 PM, Kasper Adel <karim.adel@gmail.com> wrote:
hi,
I am asked to build a large lab/test it. I'm provided crazy scale numbers for lots of technologies (L*VPN, IPv*, IGP*, All Tunnels flavors...etc).
It took me a lot of time to build this lab, because when I got the request/test plan handed over to me, I did not verify that these scaled numbers are even possible, not to mention the combination. I assumed some thought/research were done before.
I'm trying to put together a list of the lessons learned, and the right way to do this for future reference, specially that this project was time critical and I got beaten hard because I did not deliver on time.
So my question is, in your extensive experience, what is the right method/approach to this kind of task:
1) Get started immediately (MVP), things will break, tune it along the way. 2) Do some planning and research first.
I'd appreciate any references to 'software engineering' or other industries/
Thanks
hahaha, that's a good one, remember seeing it a long time ago, i saved it now. On Mon, Jun 13, 2016 at 10:29 AM, Hugo Slabbert <hugo@slabnet.com> wrote:
On Mon 2016-Jun-13 08:52:41 -0500, Possamai Rafael via NANOG < nanog@nanog.org> wrote:
This may not be an answer very specific to your problem/question, but if
you take a look at the following image, you will find a summary of what they called the engineering design methodology:
http://www.cdn.sciencebuddies.org/Files/5083/9/2013-updated_engineering-meth...
Seriously thought initially that you were going to link to:
http://i2.wp.com/tamingdata.com/wp-content/uploads/2010/07/tree-swing-projec...
You can adapt it to your circumstances, for example: instead of defining a
problem in step 1, you can define a product, and after knowing what is expected from that product, you can then move to background research, etc.
Hope that helps.
Rafael
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
On Sun, Jun 12, 2016 at 8:52 PM, Kasper Adel <karim.adel@gmail.com> wrote:
hi,
I am asked to build a large lab/test it. I'm provided crazy scale numbers for lots of technologies (L*VPN, IPv*, IGP*, All Tunnels flavors...etc).
It took me a lot of time to build this lab, because when I got the request/test plan handed over to me, I did not verify that these scaled numbers are even possible, not to mention the combination. I assumed some thought/research were done before.
I'm trying to put together a list of the lessons learned, and the right way to do this for future reference, specially that this project was time critical and I got beaten hard because I did not deliver on time.
So my question is, in your extensive experience, what is the right method/approach to this kind of task:
1) Get started immediately (MVP), things will break, tune it along the way. 2) Do some planning and research first.
I'd appreciate any references to 'software engineering' or other industries/
Thanks
On Sun, Jun 12, 2016 at 9:52 PM, Kasper Adel <karim.adel@gmail.com> wrote:
1) Get started immediately (MVP), things will break, tune it along the way.
Howdy, The history of the Panama Canal (both the French failure and the deadly first two years of the U.S. effort) should offer insight (and appropriate anecdotes) as to why this is just about always the worst answer. The same history also shows how planning moves from the French effort (an impossible sea-level canal through a river valley that suffers seasonal floods) to the entirely different engineering which produced a gigantic man-made lake with flood control dams plus a gravity-driven lock system to move vessels from sea level to the lake level and back. They did it very wrong. Then they redid it very right salvaging only a little of the original effort. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
participants (8)
-
Brian R -
David Bass -
Hugo Slabbert -
Kasper Adel -
Matthew Petach -
Possamai Rafael -
Roland Dobbins -
William Herrin