Hello NANOG, We have been struggling with firmware bugs from a specific router vendor. I am looking to set up a test lab of our core network and a few remote site routers. Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or resources anyone has that might cover testing strategies and/or best practices. Thanks, Ken
In any lab,I find concurrent access to serial ports is still an essential diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but there are multiport devices from lots of vendors. These let you SSH into the server and then connect to any serial port, giving you separate serial port windows all on the same screen. I’ve become fond of the WiFi-capable multiport modules from get-console.com. The ability to record logs from these serial ports in real-time helps a lot for documenting regression tests. -mel beckman
On Sep 28, 2023, at 7:25 AM, Kenneth Vedder <kvedder@amplex.net> wrote:
Hello NANOG,
We have been struggling with firmware bugs from a specific router vendor. I am looking to set up a test lab of our core network and a few remote site routers. Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or resources anyone has that might cover testing strategies and/or best practices.
Thanks, Ken
If you're able to virtualize some of these appliances, almost any virtualization platform can be setup to connect nodes together. However I really enjoy using Eve-ng <https://www.eve-ng.net/>. It's a network virtualization web app that can be deployed on bare metal or in a VM. It lets you diagram, connect and configure setups and do testing with different network nodes. What is pretty nice with this is you can then just click on any of the connected links and wireshark pops up and you're viewing packet captures instantly. It kind of reminds me of Cisco's VIRL, but with maybe less overhead? It has enabled me to run and connect several vendor appliances and build up configuration prior to acquiring the physical hardware for deployment. - Nick Stevermer On Thu, Sep 28, 2023 at 10:46 AM Mel Beckman <mel@beckman.org> wrote:
In any lab,I find concurrent access to serial ports is still an essential diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but there are multiport devices from lots of vendors. These let you SSH into the server and then connect to any serial port, giving you separate serial port windows all on the same screen. I’ve become fond of the WiFi-capable multiport modules from get-console.com. The ability to record logs from these serial ports in real-time helps a lot for documenting regression tests.
-mel beckman
On Sep 28, 2023, at 7:25 AM, Kenneth Vedder <kvedder@amplex.net> wrote:
Hello NANOG,
We have been struggling with firmware bugs from a specific router vendor. I am looking to set up a test lab of our core network and a few remote site routers. Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or resources anyone has that might cover testing strategies and/or best practices.
Thanks, Ken
-- *Nick Stevermer* Network Engineer Technology Department Duluth Public Schools 713 Portia Johnson Dr. Duluth, MN 55811 Phone: 218-336-8754 ext 1227 -- Confidentiality Notice: This E-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply E-mail and destroy all copies of the original message.
Appliance virtualization is perfectly acceptable for a lot of things. But there are large sets of problems that you will never catch that way. To the OP : With respect to 'strategies' : 1. Test something to make sure it works. 2. Then test it to see where and how it breaks. Lots of people do #1. Not enough do #2. (Looking at you, equipment vendors. ) On Thu, Sep 28, 2023 at 12:04 PM Nickolas Stevermer via NANOG < nanog@nanog.org> wrote:
If you're able to virtualize some of these appliances, almost any virtualization platform can be setup to connect nodes together. However I really enjoy using Eve-ng <https://www.eve-ng.net/>. It's a network virtualization web app that can be deployed on bare metal or in a VM. It lets you diagram, connect and configure setups and do testing with different network nodes. What is pretty nice with this is you can then just click on any of the connected links and wireshark pops up and you're viewing packet captures instantly. It kind of reminds me of Cisco's VIRL, but with maybe less overhead? It has enabled me to run and connect several vendor appliances and build up configuration prior to acquiring the physical hardware for deployment.
- Nick Stevermer
On Thu, Sep 28, 2023 at 10:46 AM Mel Beckman <mel@beckman.org> wrote:
In any lab,I find concurrent access to serial ports is still an essential diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but there are multiport devices from lots of vendors. These let you SSH into the server and then connect to any serial port, giving you separate serial port windows all on the same screen. I’ve become fond of the WiFi-capable multiport modules from get-console.com. The ability to record logs from these serial ports in real-time helps a lot for documenting regression tests.
-mel beckman
On Sep 28, 2023, at 7:25 AM, Kenneth Vedder <kvedder@amplex.net> wrote:
Hello NANOG,
We have been struggling with firmware bugs from a specific router vendor. I am looking to set up a test lab of our core network and a few remote site routers. Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or resources anyone has that might cover testing strategies and/or best practices.
Thanks, Ken
-- *Nick Stevermer* Network Engineer Technology Department Duluth Public Schools 713 Portia Johnson Dr. Duluth, MN 55811 Phone: 218-336-8754 ext 1227
Confidentiality Notice: This E-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply E-mail and destroy all copies of the original message.
I agree with others here... Physical lab - gotta have console server for the most control - perle console server is good, and also good ole fashion cisco terminal server (2509/2511 or 2600 with asynch module) Virtual labs are great for testing features and functionality - Juniper vLabs - Cisco DevNet sandbox - Cisco CML (i think fka VIRL) - EVE-NG - GNS3 I use these virtual environments a lot and do videos about them on my youtube channel, where I try to cover some SP-related topics. Hope it helps https://jlabs.juniper.net/vlabs/ https://developer.cisco.com/site/sandbox/ https://www.youtube.com/@aarontechtalk https://www.youtube.com/playlist?list=PL2ZMKm7ZEEWI8YyRWm9fnYNtRaV-fi-7x https://www.youtube.com/playlist?list=PL2ZMKm7ZEEWLMVxuZqeXzciRu59C02NAc -Aaron On 9/28/2023 9:14 AM, Kenneth Vedder wrote:
Hello NANOG,
We have been struggling with firmware bugs from a specific router vendor. I am looking to set up a test lab of our core network and a few remote site routers. Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or resources anyone has that might cover testing strategies and/or best practices.
Thanks, Ken
-- -Aaron
++ all that was said thus far. Physical equipment with console access is the best way to test software/firmware issues. As for virtualization, it's great for expanding your topology quickly. Use a virtual bridge in GNS3 or EVE-NG and you can make your smaller footprint physical lab into a larger topology with ease -- especially around cabling. It also allows you to do packet generation & link simulation (packet loss, jitter) much easier. You can even couple it with T-Rex. - Mark
I love the built-in Wireshark capability in EVE-NG. BTW, EVE-NG Community is free. You just have to get images for anything you want to emulate. Virtual images for various vendor products are sometimes freely available, with trail licenses. For instance Junipers vMX was freely available for a while with a 60 day license. …also vSRX, vQFX, and the new vJunos-switch (I think vEX). Aaron
On Sep 28, 2023, at 3:16 PM, Mark Prosser <mark@zealnetworks.ca> wrote:
++ all that was said thus far. Physical equipment with console access is the best way to test software/firmware issues. As for virtualization, it's great for expanding your topology quickly.
Use a virtual bridge in GNS3 or EVE-NG and you can make your smaller footprint physical lab into a larger topology with ease -- especially around cabling. It also allows you to do packet generation & link simulation (packet loss, jitter) much easier. You can even couple it with T-Rex.
- Mark
On Thu, Sep 28, 2023 at 10:14:24AM -0400, Kenneth Vedder wrote:
Hello NANOG,
We have been struggling with firmware bugs from a specific router vendor. I am looking to set up a test lab of our core network and a few remote site routers. Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a little OSPF sprinkled in. I'd be grateful for any tips or resources anyone has that might cover testing strategies and/or best practices.
if you are just testing control plane features the virtualized stuff works well. i personally use vrnetlab. if you are trying to test hw features and use actual hardware: remote controlled power optical cross connect boxes(calient/telescent) if you are testing optical stuff with distance limitation something like a timbercon spool ixia/spirent -b
participants (8)
-
Aaron Gould -
Aaron1
-
Brad Dreisbach -
Kenneth Vedder -
Mark Prosser -
Mel Beckman -
Nickolas Stevermer -
Tom Beecher