I have together with some other people, collected a wish list for OOB support, mainly aimed for core routers. This is to replace the legacy serial port usually present on core routing equipment and to move/collapse all its functionality to an ethernet only port. Some equipment already have an mgmt ethernet port, but usually this can't do "everything", meaning today one has to have OOB ethernet *and* OOB serial which just brings more pain than before. I would like to post it here to solicit feedback on it. Feel free to use it to tell your vendor account teams you want this if you feel it useful. I've already sent it to one vendor. http://swm.pp.se/oob.txt Priorities: [P1] -> must have, otherwise not useful [P2] -> would be very useful, to most operators [P3] -> nice to have, useful to some
From the OOB ethernet port it should be possible to:
[P1]: Powercycle the RP, switchfabrics and linecards (hard, as in they might be totally dead and I want to cut power to it via the back plane. Also useful for FPGA upgrades). [P1]: Connect to manage the RP(s) and linecards (equivalent of todays "connect" on GSR and ASR9k or connecting to RP serial port). [P2]: It should be possible to connect to the OOB from the RP as well (to diagnose OOB connectivity problems). [P2]: Upload software to the RP or otherwise make information available to the RP (for later re-install/turboboot for example). RP should have access to local storage on the OOB device to transfer configuration or software from the OOB device to the RP). [P2]: Read logs and other state of the components in the chassis (displays and LEDs) plus what kind of card is in each slot. [P1]: The OOB port should support (configurable), telnet, ssh and optionally [P3] https login (with a java applet or equivalent to give CLI access in the browser) with ACLs to limit wherefrom things can be done. OOB should support ssh key based logins to admin account. [P1]: The IP address of the OOB port should be set via DHCP/DHCPv6/SLAAC and should have both IPv4 and IPv6 support. If not both, then IPv6 only. [P1]: It should be possible to transfer data using tftp, ftp and scp (ftp client on the OOB device, scp being used to transfer data *to* the device (OOB being scp server). [P2]: OOB device should have tacacs and radius and [P1] local user/password database support for authentication. [P3] OOB should support ssh-key based authentication. [P3] Chassis should have a character display or LEDs with configurable blink pattern from OOB, to aid remote hands identification. [P3] OOB should have two USB ports, one to use to insert storage to transfer files to/from device. The other should be USB port that presents itself as ether USB serial port, or USB ethernet port, where the OOB device would have built in DHCP/DHCPv6 server to give IPv4v6 access to a laptop connected to the OOB so the onsite engineer can then use ssh/telnet to administrate the OOB. Optionally this port could be ethernet port (compare todays CON and AUX ports). [P2] OOB should have procedure to factory default its configuration, perhaps physical button that can be pressed and held for duration of time. The fact that this is done should be logged to the RP. [P3] OOB should have possibility to show power supply and environmental state. [P3] The factory default configuration should not include an empty or obvious login password. The factory-default login password should be the MAC address (without punctuation) of the OOB ethernet interface, which should be printed on the chassis next to the OOBE port. -- Mikael Abrahamsson email: swmike@swm.pp.se
On (2013-01-09 15:37 +0100), Mikael Abrahamsson wrote:
equipment already have an mgmt ethernet port, but usually this can't do "everything", meaning today one has to have OOB ethernet *and* OOB serial which just brings more pain than before.
The key difference is, that those are not OOB at all, they are on-band as they fate-share the control-plane. Having real OOB port is demand everyone should put in their RFQ/RFP.
Fully support. In essence, all CSCO needs to do, is bring CMP back that they had in Nexus7k/SUP1 and SUP2T but removed again in Nexus7k/SUP2 citing thermal, pintcount and lack of customer adoption. Other vendors need to check out CMP and copy it. I emailed freescale, since they are the ones who can solve the thermal and pincount problem by implementing mgmt board directly. They replied 'something similar will be implemented in the next generation of our multicore processors' (big kudos to large semi to replies quickly and cluefully to non-customer queries) Once there is hardware for this, getting vendors to implement software should be easy peasy. -- ++ytti
On Wed, Jan 9, 2013 at 9:37 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
I have together with some other people, collected a wish list for OOB support, mainly aimed for core routers.
Hi Mikael, I generally agree but have several quibbles:
[P1]: The IP address of the OOB port should be set via DHCP/DHCPv6/SLAAC and should have both IPv4 and IPv6 support. If not both, then IPv6 only.
(a) This is a P2 not a P1. Asking the OOB to be critically dependent on an external network element is dubious to begin with but even if desired it's usable without. About the only time you'd strictly *need* dynamic configuration in an OOB is when directly connecting it to a commodity Internet link. If you're willing to give your poorly secured and rarely updated OOB a public IP address, you're a braver man than I am. If you are that "brave" then you'll need a more robust set of dynamic configuration tools than just the ones you've listed and you'll also need a dynamic dns client or some other mechanism for the the OOB to let you know what addresses it ended up on. (b) IPv6-only in an OOB won't be broadly acceptable for at least another 5 years if then. You'd be foolish not to include IPv6 support in a greenfield design -- the writing is on the wall -- but there are today very few scenarios in which an IPv4 only OOB would not be usable.
[P1]: It should be possible to transfer data using tftp, ftp and scp (ftp client on the OOB device, scp being used to transfer data *to* the device (OOB being scp server).
For security and performance reasons, FTP has no place in a modern network. If you're still using it anywhere, you're borrowing grief. Replace with an http/https client. TFTP has such a strong legacy of use on routers that its presence remains just barely tolerable. For now. Have a look at how HP iLO3 makes use of http to implement virtual media. You can upload an ISO image to a web server somewhere and then instruct ilo to mount the URL as a virtual dvdrom. Best of all, if your management session disconnects, the virtual media remains mounted via the web server. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Wed, Jan 9, 2013 at 11:18 AM, William Herrin <bill@herrin.us> wrote:
About the only time you'd strictly *need* dynamic configuration in an OOB is when directly connecting it to a commodity Internet link. If you're willing to give your poorly secured and rarely updated OOB a public IP address, you're a braver man than I am. If you are that "brave" then you'll need a more robust set of dynamic configuration tools than just the ones you've listed and you'll also need a dynamic dns client or some other mechanism for the the OOB to let you know what addresses it ended up on.
it's possible that he's thinking of a world where your dhcp is not 'dynamic' but a management system which can keep all the other bits of information updated (and easily updatable!) for the remote nodes: ip address def-gw dns servers for instance.
On Wed, Jan 9, 2013 at 11:21 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 9, 2013 at 11:18 AM, William Herrin <bill@herrin.us> wrote:
About the only time you'd strictly *need* dynamic configuration in an OOB is when directly connecting it to a commodity Internet link. If you're willing to give your poorly secured and rarely updated OOB a public IP address, you're a braver man than I am. If you are that "brave" then you'll need a more robust set of dynamic configuration tools than just the ones you've listed and you'll also need a dynamic dns client or some other mechanism for the the OOB to let you know what addresses it ended up on.
it's possible that he's thinking of a world where your dhcp is not 'dynamic' but a management system which can keep all the other bits of information updated (and easily updatable!) for the remote nodes: ip address def-gw dns servers
for instance.
Sure, but in that scenario you don't *need* a dhcp system, it's merely a "nice to have." Hence a P2 not a P1. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Wed, 9 Jan 2013, Christopher Morrow wrote:
On Wed, Jan 9, 2013 at 11:18 AM, William Herrin <bill@herrin.us> wrote:
About the only time you'd strictly *need* dynamic configuration in an OOB is when directly connecting it to a commodity Internet link. If you're willing to give your poorly secured and rarely updated OOB a public IP address, you're a braver man than I am. If you are that "brave" then you'll need a more robust set of dynamic configuration tools than just the ones you've listed and you'll also need a dynamic dns client or some other mechanism for the the OOB to let you know what addresses it ended up on.
it's possible that he's thinking of a world where your dhcp is not 'dynamic' but a management system which can keep all the other bits of information updated (and easily updatable!) for the remote nodes:
Well, I was actually thinking more about initial factory default configuration. After I can reach the device, I would like to be able to set a static address. I'll consider adding this to the document. My grief with this is that if we're going to go into that kind of level, we need a RFC style document with a lot of detail, and that wasn't what I was initially aiming for. I wanted more to spark the discussion and see what came out of it. If there indeed is a lot of interest in this, I'd gladly like to try to create a more detailed document. I would be very happy if multiple vendors could standardise on a functionality and software though, perhaps even with API. Don't know which standards body would be right for this though. -- Mikael Abrahamsson email: swmike@swm.pp.se
On (2013-01-09 11:18 -0500), William Herrin wrote:
(a) This is a P2 not a P1. Asking the OOB to be critically dependent on an external network element is dubious to begin with but even if desired it's usable without.
Agreed that P2 suffices. Usage scenario is installing fresh router. You order router from vendor to remote location, notsosmarthands plug it to wires, boom you configure it remotely.
About the only time you'd strictly *need* dynamic configuration in an OOB is when directly connecting it to a commodity Internet link. If you're willing to give your poorly secured and rarely updated OOB a public IP address, you're a braver man than I am. If you are that
This is not absolute truth, but depends on what hat you wear. If you are DC guy, you have handful of POPs, arranging proper OOB network there is a breeze. If you are incumbent, you can't buy anything externally, as everyone buys from you, so you need to build separate network just for OOB. All other service providers may have hundreds of pops, you're not going to build non-revenue generating network to reach all those hundreds of pops, just to build OOB. You get cheapest connection you can get there, maybe competitor ADSL, cable model, 3G, public WLAN, ISDN what ever is available which is not fate-sharing with your network. Then plug in say cisco CPE to the OOB port, which offers address via DHCP and connect over IPSEC DMVPN to your own network. 0 touch installation of new router. Some might be ghetto and omit the CPE and use IPSEC from the management plane to openswan linux.
(b) IPv6-only in an OOB won't be broadly acceptable for at least another 5 years if then. You'd be foolish not to include IPv6 support in a greenfield design -- the writing is on the wall -- but there are today very few scenarios in which an IPv4 only OOB would not be usable.
Agreed. IPv4 would be priority for most.
For security and performance reasons, FTP has no place in a modern network. If you're still using it anywhere, you're borrowing grief. Replace with an http/https client.
http(s), scp would be my picks. Hell with FTP.
TFTP has such a strong legacy of use on routers that its presence remains just barely tolerable. For now.
There is no standard way to send arbitrary size files over TFTP, not worth the pain. -- ++ytti
On Jan 9, 2013, at 11:18 AM, William Herrin <bill@herrin.us> wrote:
[P1]: It should be possible to transfer data using tftp, ftp and scp (ftp client on the OOB device, scp being used to transfer data *to* the device (OOB being scp server).
For security and performance reasons, FTP has no place in a modern network. If you're still using it anywhere, you're borrowing grief. Replace with an http/https client.
TFTP has such a strong legacy of use on routers that its presence remains just barely tolerable. For now.
We have encountered cases where a vendor TFTP implementation + latency from the ROMMON can take a few hours to load images. I'm for ditching TFTP and replacing it with HTTP. This forces them to put in a TCP stack, and hopefully something that can window-scale and deal with the latency vs 'wait for block', ok, req next block.. The testers involved in their labs are never loading an image from 1600km away so don't get to enjoy this 'fun'. - Jared
On 10/01/2013 13:51, Jared Mauch wrote:
We have encountered cases where a vendor TFTP implementation + latency from the ROMMON can take a few hours to load images. I'm for ditching TFTP and replacing it with HTTP. This forces them to put in a TCP stack, and hopefully something that can window-scale and deal with the latency vs 'wait for block', ok, req next block..
The testers involved in their labs are never loading an image from 1600km away so don't get to enjoy this 'fun'.
From a hotel bedroom. At 03:00 in the morning.
Re: other comments: - tftp: I've run into enough problems with stupid tftp incompatibilities that I'd be really happy never having to use it again in my life. - netflow: seriously, this is not an appropriate sort of port of exporting netflow. this is a "your RP is toast" recovery mechanism, at which point netflow is probably long gone. - rs232: please no. it's 2013. I don't want or need a protocol which was designed for access speeds appropriate to the 1980s. - USB: no. can you route USB? No. DNW. - original list: sounds great, except that I want ipv4 and ipv6 given equal priority for mgmt access. Nick
On Thu, Jan 10, 2013 at 9:10 AM, Nick Hilliard <nick@foobar.org> wrote:
- netflow: seriously, this is not an appropriate sort of port of exporting netflow. this is a "your RP is toast" recovery mechanism, at which point netflow is probably long gone.
it's possible that roland was saying that the oob network should collect flow records and export them to 'something' so you'd have an idea about what traffic was on the network... I can see some value in that. I don't think roland was really saying that normal netflow from a device in production pushing a few hundred gbps of traffic would be appropriate to ship out the OOB network... or I hope that wasn't his point. I don't think oob networks need to be sized for that. I do think that having a reliable OOB Ethernet would be nice, having it not be part of the forwarding plane (and not reachable from the forwarding plane) of the device in the field would also be nice. iLO/DRAC are good analogies...
- rs232: please no. it's 2013. I don't want or need a protocol which was designed for access speeds appropriate to the 1980s.
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement. -chris
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
TDM is disappearing quickly in at least some parts of the world. We may not be quite there yet, but I think it's entirely reasonable to start asking for Ethernet console in procurement documents. Steinar Haug, Nethelp consulting, sthaug@nethelp.no
On Thu, Jan 10, 2013 at 9:44 AM, <sthaug@nethelp.no> wrote:
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
TDM is disappearing quickly in at least some parts of the world. We may not be quite there yet, but I think it's entirely reasonable to start asking for Ethernet console in procurement documents.
don't disagree... I was saying that the cost of higher speed transport in some regions is 'very high', as compared to dialup, and that the networking in question here is purely overhead costs, so keeping the cost down is important.
On Thu, 10 Jan 2013, Christopher Morrow wrote:
- rs232: please no. it's 2013. I don't want or need a protocol which was designed for access speeds appropriate to the 1980s.
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
I don't understand this argument. Are you connecting your CON directly to something that transports it out-of-the-area? Modem? If you have a consolerouter there with T1 interface as link to outside world, what's wrong with having ethernet port from that T1 router to the ethernet OOB port on the router needing OOB access, instead of having RS232 port on them. It's cheaper and easier to cable ethernet compared to RS232. RS232 has much shorter cable length compared to ethernet (9600 reaches 20 meters or so). -- Mikael Abrahamsson email: swmike@swm.pp.se
On Jan 10, 2013, at 9:51 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
On Thu, 10 Jan 2013, Christopher Morrow wrote:
- rs232: please no. it's 2013. I don't want or need a protocol which was designed for access speeds appropriate to the 1980s.
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
I don't understand this argument.
Are you connecting your CON directly to something that transports it out-of-the-area? Modem?
Yes, we have done this in a site with one device.
If you have a consolerouter there with T1 interface as link to outside world, what's wrong with having ethernet port from that T1 router to the ethernet OOB port on the router needing OOB access, instead of having RS232 port on them. It's cheaper and easier to cable ethernet compared to RS232. RS232 has much shorter cable length compared to ethernet (9600 reaches 20 meters or so).
I certainly want to use something more modern, having run Xmodem to load images into devices or net-booted systems with very large images in the past… I've seen all sorts of creative ways to do this (e.g.: DSL for OOB, 3G, private VPLS network via outside carrier). It is a challenge in the modern network space. Plus I have to figure that 9600 modems are going to be harder to find as time goes by.. at some point folks will stop making them. - Jared
On 01/10/2013 07:02 AM, Jared Mauch wrote:
On Jan 10, 2013, at 9:51 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
I certainly want to use something more modern, having run Xmodem to load images into devices or net-booted systems with very large images in the past…
I've seen all sorts of creative ways to do this (e.g.: DSL for OOB, 3G, private VPLS network via outside carrier). It is a challenge in the modern network space. Plus I have to figure that 9600 modems are going to be harder to find as time goes by.. at some point folks will stop making them.
Isn't the biggest issue here resilience? If you have ethernet/IP as your OOB mechanism, how sure can you be that it's really "OOB"? This is, I'm assuming the fallback for when things are really, really hosed. What would happen if you needed to physically get hands into many, many pops? Mike
I have a Cyclades acs-48 console server. Direct power and Ethernet drop from the ceiling with a public ip. In my subnet, but not through my routers/switches or pdus. Completely out of band, except for relying on colo power/net, which if that's not up then oob is worthless to me anyway. I have every device hooked to this. Pdus, routers, switches, vm, storage servers. That allows me to get console and power cycle every device. What more would I want? Dialup means I need to be in a place I can hook up a modem. Not too many of those. If I make a configuration mistake, need to reboot a box etc, I want to be able to access my kit from anywhere with ip connectivity. If power or network in the colo is down, then oob does me no good, and I have a dr site for that scenario. That dr site also monitors production and emails my sms address. Michael Thomas <mike@mtcc.com> wrote:
On Jan 10, 2013, at 9:51 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
I certainly want to use something more modern, having run Xmodem to load images into devices or net-booted systems with very large images in the past…
I've seen all sorts of creative ways to do this (e.g.: DSL for OOB, 3G, private VPLS network via outside carrier). It is a challenge in
On 01/10/2013 07:02 AM, Jared Mauch wrote: the modern network space. Plus I have to figure that 9600 modems are going to be harder to find as time goes by.. at some point folks will stop making them.
Isn't the biggest issue here resilience? If you have ethernet/IP as your OOB mechanism, how sure can you be that it's really "OOB"? This is, I'm assuming the fallback for when things are really, really hosed. What would happen if you needed to physically get hands into many, many pops?
Mike
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
On (2013-01-10 11:52 -0600), Charles N Wyble wrote:
I have every device hooked to this. Pdus, routers, switches, vm, storage servers. That allows me to get console and power cycle every device.
What more would I want? Dialup means I need to be in a place I can hook up a modem. Not too many of those. If I make a configuration mistake, need to reboot a box etc, I want to be able to access my kit from anywhere with ip connectivity.
If you fuck up your JunOS/IOS install and box does not have working image anymore, you need to go on-site. Otherwise you're pretty much there. But you've paid good money for the setup, especially the powercycle is expensive and introduces another place where power feed can break down. Cyclades is very good RS232 console server, supports multiplexing and maybe even persistent logging of console messages (to read what router puked out, before it hard crashed). Having ILO/DRAC/vPro style port (CMP in cisco) in your router, you'd get all this and more, cheaper. -- ++ytti
On Thu, Jan 10, 2013 at 9:51 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
On Thu, 10 Jan 2013, Christopher Morrow wrote:
- rs232: please no. it's 2013. I don't want or need a protocol which was designed for access speeds appropriate to the 1980s.
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
I don't understand this argument.
Are you connecting your CON directly to something that transports it out-of-the-area? Modem?
sure
If you have a consolerouter there with T1 interface as link to outside
i may not have a T1, because a T1 is ~2k/month or more in some places. I may have dialup to a 'console server' that services the items in the pop/location. I do hope to improve that solution with some networked thing, so I do want ethernet... I'm just saying that today it's not cost effective everywhere. You seem to agree with this, in previous posts at least.
world, what's wrong with having ethernet port from that T1 router to the ethernet OOB port on the router needing OOB access, instead of having RS232 port on them. It's cheaper and easier to cable ethernet compared to RS232. RS232 has much shorter cable length compared to ethernet (9600 reaches 20 meters or so).
odd, I could swear I've used 9600 baud over a couple hundred feet, though that's less of an issues, really.
On Jan 10, 2013, at 9:35 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
- rs232: please no. it's 2013. I don't want or need a protocol which was designed for access speeds appropriate to the 1980s.
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
I think it does beg a few questions though: Some of the POTS carriers are trying to jettison their equipment before the end of this decade. In the absence of a modem + console server, I think that IP transport is going to become increasingly important for this function, but honestly - the vendors aren't mature in this space for core equipment. Without the ability to access the removable media in the 2010 timeframe at boot time is a major oversight. There is no consistent learning or 'continual improvement' in this space. I tried to give some focus to this about a decade ago for one vendor and it led to interesting discussions at first, but it is often so low in acquisition priorities it doesn't show up. Anyone dealing with modern servers will know of the experience with the few seconds to sync up to the VGA signal and how that can allow you to miss the "Press DEL/F1/F2/F8/F12" messages. The modernization of equipment in this space has led to side-effects. I'm … (wanted to say fearful, but…) concerned with what they will concoct given their independent thought at times. Now that being said, the idea of an industry document may be something we can collaborate on as a group to list what doesn't work and why. (e.g.: I think Roland is confusing ROMMON w/ management ethers.. these can be the same physical port, but not always). - Jared
On (2013-01-10 09:54 -0500), Jared Mauch wrote:
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
Some of the POTS carriers are trying to jettison their equipment before the end of this decade. In the absence of a modem + console server,
If modem to RS232 is what OP meant. Then obviously he can do this with OOB ETH also. Just buy modem with ethernet port. I'd need this in hundreds of pops, I'm not going to build second non-revenue generating network just to get OOB. I'm going to each pop check what I can get, which does not use my network. Sometimes it's ADSL, cablemodem, ISDN, 3G, WLAN maybe even PSTN. Today: Router_RS232 - ConsoleServer - Cisco CPE (DMVPN+IPSEC) ---random_access Tomorrow Router OOB - Cisco CPE(DMVPN+IPSEC) ---random_access I'm not changing my access at all. I'm removing devices, I'm gaining ability to send images. I'm gaining ability to fix box when control-plane is fucked up. What ever access you have, it won't stop working. And people who cry 'oh my analog PSTN modem never breaks, cisco ISR will'. Availability of OOB network is irrelevant, you maybe need it long-term 5min per device per year. So as long as it is up then, you're golden. Rest of the year, have your nagios SSH into the OOB ETH every 5min, and raise alarm if shit is broken, then fix the broken shit. -- ++ytti
On (2013-01-10 09:35 -0500), Christopher Morrow wrote:
I don't think you can get ethernet and transport out-of-the-area in some places at a reasonable cost, so having serial-console I think is still a requirement.
I don't understand this point. Where does your RS232 port go? It goes to Console server in POP, which is ethernet connected? At least this is how vast majority to do it, maybe you have CON2AUX between neighbouring devices, then you could have OOB ETH to ETH between neighbouring devices. Console server costs more than ethernet switch, so it's actually cheaper to do it right. -- ++ytti
On Jan 10, 2013, at 9:35 AM, Christopher Morrow wrote:
I don't think roland was really saying that normal netflow from a device in production pushing a few hundred gbps of traffic would be appropriate to ship out the OOB network... or I hope that wasn't his point. I don't think oob networks need to be sized for that.
Actually, that is what I'm saying - and with sampled flow telemetry, this isn't a huge bps issue. With flow telemetry, we're typically looking at 0.5% - 1.5% of the aggregate utilized bandwidth of the exporting device, multiplied by the sampling ratio (e.g., .01, .0001, whatever) - so, they isn't a huge amount of traffic, even for very high-speed network. Modern networks should be designed with OOB/DCN which are sufficiently sized to handle flow telemetry. Exporting it in-band means that one is blind during network partition events, when visibility is key. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On 1/10/13, Nick Hilliard <nick@foobar.org> wrote:
On 10/01/2013 13:51, Jared Mauch wrote:
- rs232: please no. it's 2013. I don't want or need a protocol which was designed for access speeds appropriate to the 1980s. [snip] Maybe stop with rs232 versus Ethernet, and implement _both_ as separate OOB, instead of "one or the other".
The year on the calendar has little to do with the usefulness of rs232, there has been no thorough replacement for every situation. Rs232 also a simpler technology; as in, no likely way to accidentally flow production traffic over rs232, in case of security breach or anything breaking SSH/Telnet processes, crypto mechanisms, or SSH/Telnet login access; where IP/Ethernet management network is normally served by the same software processes as the Inband management; Rs232 is what you'll most likely need for recovery, for most network devices it works from device bootup and provides useful info before the OS is booted. No IP address had to be assigned that each managed device has to remember; no config information to be lost or mucked up by the unit when it dumps core, with corrupt Nvram; usually the IP and routing configuration information for management interfaces resides in the same place on the device as IP configuration information for inband interfaces on network devices. More likely to not break at the same time. Where your network device's config got blown up, or it will no longer boot, because of a corrupt image. Whereas, with RS232 you can potentially reconfig from scratch; if necessary, combine with remote power cycle capability, and a console server that can handle sending Breaks and Xmodem uploads, and it provides more robust control of the hardware. You will need some RS232 somewhere in those worst-case situations; even, as the case may be -- you resort to the OOB management method of a human bringing a computer to the network device, and plugging in a RS232 cable (Which is still OOB management, it's just not networked OOB management). Many devices log debugging output to console, and a good console switch can log this. So there are some reasons to ensure access to /both/ access avenues on the OOB management network; RS232 and Ethernet/IP not mutually exclusive. -- -JH
On 12/01/2013 18:54, Jimmy Hess wrote:
The year on the calendar has little to do with the usefulness of rs232, there has been no thorough replacement for every situation.
Tell that to Juniper who appear to think that running an RE console at 9600 baud is actually OK in a emergency situation in a production environment. Do the people who design monumental stupidity of this scale into systems ever get to hear about real life customer requirements? Every single UART in existence today supports 115200 or greater, more likely 230400. What is the problem here?
Rs232 also a simpler technology
No it really isn't - from an operational point of view. Ok, the basic signalling is far simpler, but why do we have straight-thru cables, cross-over cables, null modem cables, with db9, db15, db25, RJ11, at least 3-4 common varieties of RJ45, RS232 over USB (entirely USB driver dependent), along with 7/8 bits, N/O/E parity and 0/1/2 stop bit settings. And who in the name of gibbering lunacy thought that 2 stop bits on an RSP440/ASR9000 was a smart idea? Is there anything else in the known universe which uses 2 stop bits?? I mean seriously Cisco, WTF?? Seriously, WTF. I don't want to have to deal with emergencies which have me fumbling around in my toolbox at 02:00 to try to guess which particular pinout is used by some godforsaken piece of kit in the corner which happens to have died. Or to have to deal with some dumbass kernel panic on my mac because the manufacturer's PL2303 driver is fighting with the cheap knockoff clone converter. Or try to reverse engineer the fact that some brain damaged vendor needs a female DTE or male DCE cable. Or deal with line noise on a cable which is slightly loose, and which caused bootup to fail silently. These are all real-life problems which have happened, btw. Some regularly. I am tired of dealing with this stupidity. I don't want a toolbox where I feel that it's necessary to pack in 2xDB9 + 2xDB25 + RJ45 + pinout converter + USB-serial converter and then have to remember the settings for the lot of them in such a way that I don't need to carry around a manual for dealing with this crap at 3 in the morning. Kill it with fire. I want OOB with ethernet, MDIX, 100base-TX or 1000base-TX, with DHCP client support. With a cherry. Nick
On Sat, Jan 12, 2013 at 6:26 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Sat, Jan 12, 2013 at 3:26 PM, Nick Hilliard <nick@foobar.org> wrote:
I want OOB with ethernet, MDIX, 100base-TX or 1000base-TX, with DHCP client support. With a cherry.
and auto configuration that works? :) reliably? with your switch/router upstream? :)
Thank goodness ethernet never has problems with negotiation going awry, and coming up with mismatched duplexes, and vendors never had to implement "no negotiation-auto" in their configs because you couldn't count on everyone's implementations working together just absolutely perfectly the first time on bootup. Yes, it sure is a good thing ethernet never has issues like that which would cripple your ability to get a box up and running at 2am. Matt PS--while we're at it, can I have a pony?
On Sat, 12 Jan 2013, Matthew Petach wrote:
Thank goodness ethernet never has problems with negotiation going awry, and coming up with mismatched duplexes, and vendors never had to implement "no negotiation-auto" in their configs because you couldn't count on everyone's implementations working together just absolutely perfectly the first time on bootup. Yes, it sure is a good thing ethernet never has issues like that which would cripple your ability to get a box up and running at 2am.
Has this happened to you with equipment designed and manufactured the past 5 years? For me this was a problem with equipment released around 2000, since 2005-2007 or so I haven't seen a single problem that I can recall. I blame part of this problem on Cisco who was especially bad at handling autoneg. I remember in 1998 when we couldn't even get link up between a 100 meg LE interface on a Sun and a (I believe) 3500XL. We had to use a hub in between to get link at all. Even worse, different port blocks on the 3500XL behaved differently. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Sat, 12 Jan 2013, Matthew Petach wrote:
Thank goodness ethernet never has problems with negotiation going awry, and coming up with mismatched duplexes, and vendors never had to implement "no negotiation-auto" in their configs because you couldn't count on everyone's implementations working together just absolutely perfectly the first time on bootup. Yes, it sure is a good thing ethernet never has issues like that which would cripple your ability to get a box up and running at 2am. Deployments I've worked on are order of 10^6 managed ports at a time 10^5ish oob ports, auto-negotiation on copper is not a problem that
On 1/13/13 12:12 AM, Mikael Abrahamsson wrote: figures in rollouts anymore and hasn't for more than half a decade.
Has this happened to you with equipment designed and manufactured the past 5 years?
For me this was a problem with equipment released around 2000, since 2005-2007 or so I haven't seen a single problem that I can recall.
I blame part of this problem on Cisco who was especially bad at handling autoneg. I remember in 1998 when we couldn't even get link up between a 100 meg LE interface on a Sun and a (I believe) 3500XL. We had to use a hub in between to get link at all. Even worse, different port blocks on the 3500XL behaved differently.
From: Mikael Abrahamsson [mailto:swmike@swm.pp.se] On Sat, 12 Jan 2013, Matthew Petach wrote:
Thank goodness ethernet never has problems with negotiation going awry, and coming up with mismatched duplexes, and vendors never had to implement "no negotiation-auto" in their configs because you couldn't count on everyone's implementations working together just absolutely perfectly the first time on bootup. Yes, it sure is a good thing ethernet never has issues like that which would cripple your ability to get a box up and running at 2am.
Has this happened to you with equipment designed and manufactured the past 5 years?
This happened to me just last month. Jamie
On 13/01/2013 07:42, Matthew Petach wrote:
PS--while we're at it, can I have a pony?
The day that we see good quality trouble-free OOB on all networking kit that everyone is happy about will be the day that vendors shower us with ponies for all. I'm quite sure of it. Nick
On Wed, 9 Jan 2013, Mikael Abrahamsson wrote:
I would like to post it here to solicit feedback on it. Feel free to use it to tell your vendor account teams you want this if you feel it useful. I've already sent it to one vendor.
Ethernet/Serial/USB management is useful, but I would not be in favor of eliminating the serial port entirely, because having an OOB serial console connection has saved me more than a few times. Plus, having working connectivity from something like a terminal server or a headless Linux box has proven its value countless times in the past. I would also add the following to your list: If $vendor's device provides an IP-based management interface over an ethernet port, it should provide both a web-based interface and a CLI. The web interface should be as platform-agnostic as possible (not restrict people to specific platforms and browsers), and be as gentle as possible in requiring things like Java. Being forced to deal with Java runtime version dependency hell in a critical situation would not fill my heart with joy. jms
I think this list goes too far, and has a decent chance of introducing other fun failure modes as a result. The goal of OOB is generally to gain control of a "misbehaving" device. Now, misbehaving can take many forms, from the device actually being ok and all of it's circuits going down (fiber cut isolating it), to the device being very much not ok with a bad linecard trying to lock up the bus, core dumps, etc. I'm going to pick on one specific example: In a message written on Wed, Jan 09, 2013 at 03:37:16PM +0100, Mikael Abrahamsson wrote:
[P1]: Powercycle the RP, switchfabrics and linecards (hard, as in they might be totally dead and I want to cut power to it via the back plane. Also useful for FPGA upgrades).
Most Cisco high end devices can do this today from the CLI (test mbus power off on a GSR, for example). Let's consider what it would take to move that functionality from the live software to some sort of "etherent oob" as proposed... The first big step is that some sort of "computer" to operate the ethernet oob is required. I think where you're going with this is some sort of small SoC type thing connected to the mangement buss of the device, not unlike an IPMI device on a server. Using IPMI devices on servers as an example this is now another device that must be secured, upgraded, and has the potential for bugs of its own. Since only a small fraction of high end users will use the OOB at all (inband is fine for many, many networks), there will not be a lot of testing of this code, or demand for features/fixes. So while I agree with the list of features in large part, I'm not sure I agree with the concept of having some sort of ethernet interface that allows all of this out of band. I think it will add cost, complexity, and a lot of new failure modes. The reality is the current situation on high end gear, a serial console plus ethernet "management" port is pretty close to a good situation, and could be a really good situation with a few minor modifications. My list would be much simpler as a result: 1) I would like to see serial consoles replaced with USB consoles. They would still appear to be serial devices to most equipment, but would enable much faster speeds making working on the console a much more reasonable option. For bonus points, an implementation that presents 2-4 serial "terminals" over the same USB cable would allow multiple people to log into the device without the need for any network connectivity. This would also allow USB hubs to be used to connect multiple devices in a colo, rather than the serial terminal servers needed today. 2) I would like to see "manangement" ethernets that live in their own walled off world out of the box. Yes, I know with most boxes you can put them in a VRF or similar, but that should be the default. I should be able to put an IP and default route on a management ethernet and still have a 100% empty (main) routing table. This would allow the management port to be homed to a separate network simply and easily. 3) I would like to see "legacy protocols" dumped. TFTP, bye bye. FTP, bye bye. rcp, bye bye. HTTP, HTTPS, and SCP should be supported for all operations at all levels of the OS. In this ideal world, the deployment model is simple. A small OOB device would be deployed (think like a Cisco 1900, or Juniper SRX 220), connected to a separate network (DSL, cable modem, cell modem, ethernet to some other provider, or gasp, even an old school analog modem). Each large router would get an ethernet port and usb console to that device. SSH to the right port would get the USB console, ideally with the 2-4 consoles exposed where hitting the same port just cycles through them. At that point all of the functionality described in the original post should be available in the normal CLI on the device. File transfer operations should be able to specify the management port "copy [mangement]http://1.2.3.4/newimage.code flash:" to use that interface/routing table. I also think on most boxes this would require no hardware changes. The high end boxes have Ethernet, they have USB...it's just updating the software to make them act in a much more useful way, rather than the half brain-dead ways they act now... -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On (2013-01-09 09:12 -0800), Leo Bicknell wrote:
So while I agree with the list of features in large part, I'm not sure I agree with the concept of having some sort of ethernet interface that allows all of this out of band. I think it will add cost, complexity, and a lot of new failure modes.
It already exists, CMP is its name, and it is great. Server people woke up to this over decade ago, so should networking people. Failure modes are somewhat uninteresting, as long as it does not fate-share with control-plane. Having RS232 or USB console on forwarding-plane is not OOB. And even OOB version of these is of limited value, you can't send images over them, you can't multiplex over them and RS232 OOB 'server' costs more than switch. So you get less and you pay more. HW + SW wise it's extremely simple contraption, all the code and HW needed is proven. RS232 on-band management is case of 'well it's always done like this, so it must be the right way to do it' -- ++ytti
On Jan 9, 2013, at 12:34 PM, Saku Ytti <saku@ytti.fi> wrote:
Having RS232 or USB console on forwarding-plane is not OOB. And even OOB version of these is of limited value, you can't send images over them, you can't multiplex over them and RS232 OOB 'server' costs more than switch. So you get less and you pay more. HW + SW wise it's extremely simple contraption, all the code and HW needed is proven.
I am very much against USB consoles. there can be a whole plethora of issues involved from OS-level to the device-level. When I'm on the console, things have already gone bad. I don't need to find out if the vendor has the right 'entitlement' established for me to download and load the driver or anything else.. It *needs* to work, I can't wait for the device on the other end to negotiate with the host system, etc.. I understand why people want it, but USB as it exists today isn't the way. (I can screw down a rs232 connector and it can be secure, I can't attach USB with the same certainty). - Jared
On (2013-01-10 08:57 -0500), Jared Mauch wrote:
I am very much against USB consoles. there can be a whole plethora of issues involved from OS-level to the device-level. When I'm on the console, things have already gone bad. I don't need to find out if the vendor has the right 'entitlement' established for me to download and load the driver or anything else..
I'm certainly not rooting for USB console, I don't want to fix broken solution with another broken solution. I'm all for Ethernet OOB (true OOB, not fate-sharing control-plane), exactly like CMP in Cisco. -- ++ytti
I absolutely agree that USB is a bad way to go with this, as well as web management. I have no interest in trying to use some terrible web app to bring a network back up when simple 300 baud would suffice. I've got no problem with telnet/ssh, although I hate the idea of needing to know an ip address to emergency jack in to a device instead of just a bit rate, but please no web app. -Blake On Thu, Jan 10, 2013 at 8:00 AM, Saku Ytti <saku@ytti.fi> wrote:
On (2013-01-10 08:57 -0500), Jared Mauch wrote:
I am very much against USB consoles. there can be a whole plethora of issues involved from OS-level to the device-level. When I'm on the console, things have already gone bad. I don't need to find out if the vendor has the right 'entitlement' established for me to download and load the driver or anything else..
I'm certainly not rooting for USB console, I don't want to fix broken solution with another broken solution. I'm all for Ethernet OOB (true OOB, not fate-sharing control-plane), exactly like CMP in Cisco.
-- ++ytti
On Wed, 9 Jan 2013, Leo Bicknell wrote:
of the device, not unlike an IPMI device on a server. Using IPMI
IPMI is exactly what we're going for.
In this ideal world, the deployment model is simple. A small OOB device would be deployed (think like a Cisco 1900, or Juniper SRX 220), connected to a separate network (DSL, cable modem, cell modem, ethernet to some other provider, or gasp, even an old school analog modem). Each large router would get an ethernet port and usb console to that device. SSH to the right port would get the USB console, ideally with the 2-4 consoles exposed where hitting the same port just cycles through them.
This is added cost and complexity. Sometimes there is only 1-2 devices in the pop, and now there is need to install a serial console router with DC (limits options) just to connect to the serial console, which might not work anyway because the control plane might be so screwed up that it actually needs power cycling. So I want to retire serial ports in the front to be needed for normal operation. Look at the XR devices from Cisco for instance. For "normal maintenance" you pretty much require both serial console (to do rommon stuff one would imagine shouldn't be needed) and also mgmt ethernet (to use tftp for downloading software when you need to turbo-boot because the system is now screwed up because the XR developer ("install") team messed up the SMUs *again*). For instance, if you have single RP the upgrade instructions for 4.2.1 lists going into rommon and doing "boot -s", *or* power cycling the box, after FPGA upgrade. -- Mikael Abrahamsson email: swmike@swm.pp.se
In a message written on Wed, Jan 09, 2013 at 06:39:28PM +0100, Mikael Abrahamsson wrote:
IPMI is exactly what we're going for.
For Vendors that use a "PC" motherboard, IPMI would probably not be difficult at all! :) I think IPMI is a pretty terrible solution though, so if that's your target I do think it's a step backwards. Most IPMI cards are prime examples of my worries, Linux images years out of date, riddled with security holes and universally not trusted. You're going to need a "firewall" in front of any such solution to deploy it, so you can't really eliminate the extra box I proposed just change its nature. I also still think there's a lot of potential here to take gigantic steps backwards. Replacing a serial console with a Java applet in a browser (a la most IPMI devices) would be a huge step backwards. Today it's trival to script console access, in a Java applet world, not so much. Having a IPMI like device with dedicated ethernet and connection to the management bus would allow it to have a web interface to do things like power cycle individual line cards and may be a win, but I would posit these things are to work around horribly broken upgrade procedures that vendors have not given enough thought. They could be solved with more intelligent software in the ROM and on the main box without needing any add on device.
So I want to retire serial ports in the front to be needed for normal operation. Look at the XR devices from Cisco for instance. For "normal maintenance" you pretty much require both serial console (to do rommon stuff one would imagine shouldn't be needed) and also mgmt ethernet (to use tftp for downloading software when you need to turbo-boot because the system is now screwed up because the XR developer ("install") team messed up the SMUs *again*).
Your vendor is going to hire those same developers to write the code for your OOB device. The solution here is not bad developers writing and deploying even more code, it's to demand your vendors uplevel their developers and software. Ever have these problems on Vendor J? No, the upgrade process there is smooth as silk. Not to say that vendor is perfect, they just have different warts. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On (2013-01-09 10:18 -0800), Leo Bicknell wrote:
I also still think there's a lot of potential here to take gigantic steps backwards. Replacing a serial console with a Java applet in a browser (a la most IPMI devices) would be a huge step backwards. Today it's trival to script console access, in a Java applet world, not so much.
P1 requirement was ssh.
Ever have these problems on Vendor J? No, the upgrade process there is smooth as silk. Not to say that vendor is perfect, they just have different warts.
I'm getting maybe bit too far from topic. We have different opinion of smooth or silk. I hate how in J once you do the upgrade, current config is stored with it, so when you finally boot, you're using that configuration. So this means, you can't install new image to all boxes at once you decide new standard release and then reload then when you get maint window, without having any extra work during expensive maint hours. Also I've seen very poor hit-miss ratio with ISSU, so I can't use it at all. -- ++ytti
On Jan 9, 2013, at 1:18 PM, Leo Bicknell <bicknell@ufp.org> wrote:
In a message written on Wed, Jan 09, 2013 at 06:39:28PM +0100, Mikael Abrahamsson wrote:
IPMI is exactly what we're going for.
For Vendors that use a "PC" motherboard, IPMI would probably not be difficult at all! :)
I think IPMI is a pretty terrible solution though, so if that's your target I do think it's a step backwards. Most IPMI cards are prime examples of my worries, Linux images years out of date, riddled with security holes and universally not trusted. You're going to need a "firewall" in front of any such solution to deploy it, so you can't really eliminate the extra box I proposed just change its nature.
https://www.schneier.com/blog/archives/2013/01/the_eavesdroppi.html --Steve Bellovin, https://www.cs.columbia.edu/~smb
On 1/9/2013 9:12 AM, Leo Bicknell wrote:
I think this list goes too far, and has a decent chance of introducing other fun failure modes as a result. The goal of OOB is generally to gain control of a "misbehaving" device. Now, misbehaving can take many forms, from the device actually being ok and all of it's circuits going down (fiber cut isolating it), to the device being very much not ok with a bad linecard trying to lock up the bus, core dumps, etc.
I'm going to pick on one specific example:
In a message written on Wed, Jan 09, 2013 at 03:37:16PM +0100, Mikael Abrahamsson wrote:
[P1]: Powercycle the RP, switchfabrics and linecards (hard, as in they might be totally dead and I want to cut power to it via the back plane. Also useful for FPGA upgrades). Most Cisco high end devices can do this today from the CLI (test mbus power off on a GSR, for example). Let's consider what it would take to move that functionality from the live software to some sort of "etherent oob" as proposed...
Install an Embedded Peer (a Bus Level Peering Card like a SlotServer or the Fuji Module and provide console level access through the peer. Then the Peer itself becomes the controller interface. Todd
The first big step is that some sort of "computer" to operate the ethernet oob is required. I think where you're going with this is some sort of small SoC type thing connected to the mangement buss of the device, not unlike an IPMI device on a server. Using IPMI devices on servers as an example this is now another device that must be secured, upgraded, and has the potential for bugs of its own. Since only a small fraction of high end users will use the OOB at all (inband is fine for many, many networks), there will not be a lot of testing of this code, or demand for features/fixes.
So while I agree with the list of features in large part, I'm not sure I agree with the concept of having some sort of ethernet interface that allows all of this out of band. I think it will add cost, complexity, and a lot of new failure modes.
The reality is the current situation on high end gear, a serial console plus ethernet "management" port is pretty close to a good situation, and could be a really good situation with a few minor modifications. My list would be much simpler as a result:
1) I would like to see serial consoles replaced with USB consoles. They would still appear to be serial devices to most equipment, but would enable much faster speeds making working on the console a much more reasonable option. For bonus points, an implementation that presents 2-4 serial "terminals" over the same USB cable would allow multiple people to log into the device without the need for any network connectivity.
This would also allow USB hubs to be used to connect multiple devices in a colo, rather than the serial terminal servers needed today.
2) I would like to see "manangement" ethernets that live in their own walled off world out of the box. Yes, I know with most boxes you can put them in a VRF or similar, but that should be the default. I should be able to put an IP and default route on a management ethernet and still have a 100% empty (main) routing table. This would allow the management port to be homed to a separate network simply and easily.
3) I would like to see "legacy protocols" dumped. TFTP, bye bye. FTP, bye bye. rcp, bye bye. HTTP, HTTPS, and SCP should be supported for all operations at all levels of the OS.
In this ideal world, the deployment model is simple. A small OOB device would be deployed (think like a Cisco 1900, or Juniper SRX 220), connected to a separate network (DSL, cable modem, cell modem, ethernet to some other provider, or gasp, even an old school analog modem). Each large router would get an ethernet port and usb console to that device. SSH to the right port would get the USB console, ideally with the 2-4 consoles exposed where hitting the same port just cycles through them.
At that point all of the functionality described in the original post should be available in the normal CLI on the device. File transfer operations should be able to specify the management port "copy [mangement]http://1.2.3.4/newimage.code flash:" to use that interface/routing table.
I also think on most boxes this would require no hardware changes. The high end boxes have Ethernet, they have USB...it's just updating the software to make them act in a much more useful way, rather than the half brain-dead ways they act now...
-- Regards TSG "Ex-Cruce-Leo" //Confidential Mailing - Please destroy this if you are not the intended recipient.
On Jan 9, 2013, at 9:37 AM, Mikael Abrahamsson wrote:
Flow telemetry export - many of these so-called 'management' ports can't be used to export flow, oddly enough. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On (2013-01-09 23:17 +0000), Dobbins, Roland wrote:
Flow telemetry export - many of these so-called 'management' ports can't be used to export flow, oddly enough.
That is task for on-band interfaces, which attach to your forwarding-logic. OOB is separate component, really only relying on same powersupply, it's key value that it is not fate-sharing any more than absolutely must with host. To export flow, you need port to be connected to your forwarding hardware, not control-plane and certainly not OOB management-plane. The port sole purpose is disaster recovery, it's like NG RS232 port, it would not be used as your day to day port to configure the box. Cheack Cisco's "CMP" this is what we need. -- ++ytti
On Jan 10, 2013, at 2:15 AM, Saku Ytti wrote:
That is task for on-band interfaces, which attach to your forwarding-logic.
No it isn't, any more than SNMP is a task for those interfaces.
To export flow, you need port to be connected to your forwarding hardware, not control-plane and certainly not OOB management-plane.
Again, the analogy is with SNMP. There's no requirement to be part of the data-plane, it's quite possible to get the flow telemetry to the management processor, same as with SNMP.
Cheack Cisco's "CMP" this is what we need.
I'm quite familiar with it, thanks. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Thu, 10 Jan 2013, Dobbins, Roland wrote:
No it isn't, any more than SNMP is a task for those interfaces.
Well, then what you're looking for is not what we're looking for (?). You seem to want the type of classic mgmt ethernet currently residing on high end router platforms (on the RP) and not a ILO/CMP type interface that we're looking for. I definitely do not want SNMP and netflow on my disaster recovery OOB network. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Jan 10, 2013, at 6:15 AM, Mikael Abrahamsson wrote:
I definitely do not want SNMP and netflow on my disaster recovery OOB network.
Of course you do - else you're deaf, dumb, and blind at precisely the time you most need complete network visibility, i.e., during a disruptive event of some sort. The ability to type commands via ssh and/or console ports isn't very helpful if one lacks enough context to know what to type, heh. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Thu, 10 Jan 2013, Dobbins, Roland wrote:
Of course you do - else you're deaf, dumb, and blind at precisely the time you most need complete network visibility, i.e., during a disruptive event of some sort.
You and me seem to talk about different types of disasters. In my type of disaster, SNMP and netflow doesn't work because the RP is out of commission or seriously malfunctioning.
The ability to type commands via ssh and/or console ports isn't very helpful if one lacks enough context to know what to type, heh.
I don't know what to respond to this because I don't understand what you're getting at. -- Mikael Abrahamsson email: swmike@swm.pp.se
On (2013-01-10 10:48 +0000), Dobbins, Roland wrote:
No it isn't, any more than SNMP is a task for those interfaces.
Sending flowrecords to your slow ppc CPU just to allow export in non-HW interface is silly, when HW can export it directly, without ever hitting your control-plane. Polling SNMP is low volume, so you easily allow RP to poll for them from HW. But implementing this also in HW would be interesting for low-interval SNMP polling, then it also would stop working in your non-HW interfaces.
Again, the analogy is with SNMP. There's no requirement to be part of the data-plane, it's quite possible to get the flow telemetry to the management processor, same as with SNMP.
Sure, if performance is not important and if you're too poor to buy interface in your router. -- ++ytti
My main requirements would be: 1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?) 2. Something that is standard across everything, and can be aggregated easily onto a "console server" or the like I don't really see what is wrong with with keeping the serial port as the standard. Things like servers and RAID cards and such are coming with "BIOS"es that are graphical and even require a mouse to use. What use is that when I need to get into the BIOS from a remote site that is completely down? Likewise OS vendors are increasingly dropping support for installing OSes via serial port (RHEL, VMWare, etc.) At leaset with RHEL, you can make your own boot image that gets rid of the asinine splash screen (which is the only thing that causes the requirement for a full VGA console) It might be nice to have a "management-only" port of some sort to do more advanced things that serial cannot do, but the serial port is ubiquitous already, and I don't see any reason to remove it as the very low-level access method. thanks, -Randy ----- Original Message -----
I have together with some other people, collected a wish list for OOB support, mainly aimed for core routers. This is to replace the legacy serial port usually present on core routing equipment and to move/collapse all its functionality to an ethernet only port. Some equipment already have an mgmt ethernet port, but usually this can't do "everything", meaning today one has to have OOB ethernet *and* OOB serial which just brings more pain than before.
I would like to post it here to solicit feedback on it. Feel free to use it to tell your vendor account teams you want this if you feel it useful. I've already sent it to one vendor.
Priorities:
[P1] -> must have, otherwise not useful [P2] -> would be very useful, to most operators [P3] -> nice to have, useful to some
From the OOB ethernet port it should be possible to:
[P1]: Powercycle the RP, switchfabrics and linecards (hard, as in they might be totally dead and I want to cut power to it via the back plane. Also useful for FPGA upgrades).
[P1]: Connect to manage the RP(s) and linecards (equivalent of todays "connect" on GSR and ASR9k or connecting to RP serial port).
[P2]: It should be possible to connect to the OOB from the RP as well (to diagnose OOB connectivity problems).
[P2]: Upload software to the RP or otherwise make information available to the RP (for later re-install/turboboot for example). RP should have access to local storage on the OOB device to transfer configuration or software from the OOB device to the RP).
[P2]: Read logs and other state of the components in the chassis (displays and LEDs) plus what kind of card is in each slot.
[P1]: The OOB port should support (configurable), telnet, ssh and optionally [P3] https login (with a java applet or equivalent to give CLI access in the browser) with ACLs to limit wherefrom things can be done. OOB should support ssh key based logins to admin account.
[P1]: The IP address of the OOB port should be set via DHCP/DHCPv6/SLAAC and should have both IPv4 and IPv6 support. If not both, then IPv6 only.
[P1]: It should be possible to transfer data using tftp, ftp and scp (ftp client on the OOB device, scp being used to transfer data *to* the device (OOB being scp server).
[P2]: OOB device should have tacacs and radius and [P1] local user/password database support for authentication. [P3] OOB should support ssh-key based authentication.
[P3] Chassis should have a character display or LEDs with configurable blink pattern from OOB, to aid remote hands identification.
[P3] OOB should have two USB ports, one to use to insert storage to transfer files to/from device. The other should be USB port that presents itself as ether USB serial port, or USB ethernet port, where the OOB device would have built in DHCP/DHCPv6 server to give IPv4v6 access to a laptop connected to the OOB so the onsite engineer can then use ssh/telnet to administrate the OOB. Optionally this port could be ethernet port (compare todays CON and AUX ports).
[P2] OOB should have procedure to factory default its configuration, perhaps physical button that can be pressed and held for duration of time. The fact that this is done should be logged to the RP.
[P3] OOB should have possibility to show power supply and environmental state.
[P3] The factory default configuration should not include an empty or obvious login password. The factory-default login password should be the MAC address (without punctuation) of the OOB ethernet interface, which should be printed on the chassis next to the OOBE port.
-- Mikael Abrahamsson email: swmike@swm.pp.se
Once upon a time, Randy Carpenter <rcarpen@network1.net> said:
Likewise OS vendors are increasingly dropping support for installing OSes via serial port (RHEL, VMWare, etc.)
At leaset with RHEL, you can make your own boot image that gets rid of the asinine splash screen (which is the only thing that causes the requirement for a full VGA console)
RHEL installs with a serial console just fine. You also don't have to "make your own boot image" to get a non-graphical boot. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
----- Original Message -----
Once upon a time, Randy Carpenter <rcarpen@network1.net> said:
Likewise OS vendors are increasingly dropping support for installing OSes via serial port (RHEL, VMWare, etc.)
At leaset with RHEL, you can make your own boot image that gets rid of the asinine splash screen (which is the only thing that causes the requirement for a full VGA console)
RHEL installs with a serial console just fine. You also don't have to "make your own boot image" to get a non-graphical boot.
Probably a bit off topic for this thread, but... If I boot the default install disc/image on any of my servers (mostly Supermicro), it hangs at a blank screen when isolinux loads. If you get rid of the splash screen, it works fine. This has been an issue since RHEL4, I think. Maybe other server manufacturers handle the video a little differently, and are able to get past the splash screen. -Randy
Uplogix has a pretty rad solution..
From my Galaxy Note II, please excuse any mistakes.
-------- Original message -------- From: Randy Carpenter <rcarpen@network1.net> Date: 01/09/2013 7:07 PM (GMT-08:00) To: Mikael Abrahamsson <swmike@swm.pp.se> Cc: nanog@nanog.org Subject: Re: OOB core router connectivity wish list My main requirements would be: 1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?) 2. Something that is standard across everything, and can be aggregated easily onto a "console server" or the like I don't really see what is wrong with with keeping the serial port as the standard. Things like servers and RAID cards and such are coming with "BIOS"es that are graphical and even require a mouse to use. What use is that when I need to get into the BIOS from a remote site that is completely down? Likewise OS vendors are increasingly dropping support for installing OSes via serial port (RHEL, VMWare, etc.) At leaset with RHEL, you can make your own boot image that gets rid of the asinine splash screen (which is the only thing that causes the requirement for a full VGA console) It might be nice to have a "management-only" port of some sort to do more advanced things that serial cannot do, but the serial port is ubiquitous already, and I don't see any reason to remove it as the very low-level access method. thanks, -Randy ----- Original Message -----
I have together with some other people, collected a wish list for OOB support, mainly aimed for core routers. This is to replace the legacy serial port usually present on core routing equipment and to move/collapse all its functionality to an ethernet only port. Some equipment already have an mgmt ethernet port, but usually this can't do "everything", meaning today one has to have OOB ethernet *and* OOB serial which just brings more pain than before.
I would like to post it here to solicit feedback on it. Feel free to use it to tell your vendor account teams you want this if you feel it useful. I've already sent it to one vendor.
Priorities:
[P1] -> must have, otherwise not useful [P2] -> would be very useful, to most operators [P3] -> nice to have, useful to some
From the OOB ethernet port it should be possible to:
[P1]: Powercycle the RP, switchfabrics and linecards (hard, as in they might be totally dead and I want to cut power to it via the back plane. Also useful for FPGA upgrades).
[P1]: Connect to manage the RP(s) and linecards (equivalent of todays "connect" on GSR and ASR9k or connecting to RP serial port).
[P2]: It should be possible to connect to the OOB from the RP as well (to diagnose OOB connectivity problems).
[P2]: Upload software to the RP or otherwise make information available to the RP (for later re-install/turboboot for example). RP should have access to local storage on the OOB device to transfer configuration or software from the OOB device to the RP).
[P2]: Read logs and other state of the components in the chassis (displays and LEDs) plus what kind of card is in each slot.
[P1]: The OOB port should support (configurable), telnet, ssh and optionally [P3] https login (with a java applet or equivalent to give CLI access in the browser) with ACLs to limit wherefrom things can be done. OOB should support ssh key based logins to admin account.
[P1]: The IP address of the OOB port should be set via DHCP/DHCPv6/SLAAC and should have both IPv4 and IPv6 support. If not both, then IPv6 only.
[P1]: It should be possible to transfer data using tftp, ftp and scp (ftp client on the OOB device, scp being used to transfer data *to* the device (OOB being scp server).
[P2]: OOB device should have tacacs and radius and [P1] local user/password database support for authentication. [P3] OOB should support ssh-key based authentication.
[P3] Chassis should have a character display or LEDs with configurable blink pattern from OOB, to aid remote hands identification.
[P3] OOB should have two USB ports, one to use to insert storage to transfer files to/from device. The other should be USB port that presents itself as ether USB serial port, or USB ethernet port, where the OOB device would have built in DHCP/DHCPv6 server to give IPv4v6 access to a laptop connected to the OOB so the onsite engineer can then use ssh/telnet to administrate the OOB. Optionally this port could be ethernet port (compare todays CON and AUX ports).
[P2] OOB should have procedure to factory default its configuration, perhaps physical button that can be pressed and held for duration of time. The fact that this is done should be logged to the RP.
[P3] OOB should have possibility to show power supply and environmental state.
[P3] The factory default configuration should not include an empty or obvious login password. The factory-default login password should be the MAC address (without punctuation) of the OOB ethernet interface, which should be printed on the chassis next to the OOBE port.
-- Mikael Abrahamsson email: swmike@swm.pp.se
On Wed, 9 Jan 2013, Randy Carpenter wrote:
My main requirements would be:
1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)
I don't understand this at all. Why can't an OOB network be ethernet based towards the equipment needing management?
2. Something that is standard across everything, and can be aggregated easily onto a "console server" or the like
Yes, ethernet is the proposed management standard interface.
I don't really see what is wrong with with keeping the serial port as the standard.
Because it's slow and can't be multiplexed, and it's expensive, only does short distance (20 meters or so), uses different cabling, requires separate planning etc. There are lot of reasons to drop serial port support.
Things like servers and RAID cards and such are coming with "BIOS"es that are graphical and even require a mouse to use. What use is that when I need to get into the BIOS from a remote site that is completely down?
My email stated https was way down on the priorities, and ssh and telnet was the prime way of managing the OOB part.
It might be nice to have a "management-only" port of some sort to do more advanced things that serial cannot do, but the serial port is ubiquitous already, and I don't see any reason to remove it as the very low-level access method.
An ethernet port is generally a lot cheaper compared to a serial port. Your OOB network would consist of a switch or router with ethernet towards the equipment needing management. -- Mikael Abrahamsson email: swmike@swm.pp.se
----- Original Message -----
On Wed, 9 Jan 2013, Randy Carpenter wrote:
My main requirements would be:
1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)
I don't understand this at all. Why can't an OOB network be ethernet based towards the equipment needing management?
How do I connect to it from many miles away when the network is down? I have connected to a misbehaving border device at a remote network via dial-up before, and was able to get it back up and running. I would not have been able to do that if the only options were ethernet or ethernet.
2. Something that is standard across everything, and can be aggregated easily onto a "console server" or the like
Yes, ethernet is the proposed management standard interface.
I don't really see what is wrong with with keeping the serial port as the standard.
Because it's slow and can't be multiplexed,
agreed. Although, I generally don't care if it is slow, as the only need is to get the real network connected features back up and running.
and it's expensive, only does short distance (20 meters or so),
I completely disagree. The ability for serial to go over POTS makes it ridiculously cheap compared to building a reliable ethernet connection over hundreds or thousands of miles.
uses different cabling, requires separate planning etc. There are lot of reasons to drop serial port support.
The separate part is what makes it useful. The only reason you should need to access the serial port is because the network is not functioning. If you move the last resort access to be network, how do you access it when you have network issues?
An ethernet port is generally a lot cheaper compared to a serial port. Your OOB network would consist of a switch or router with ethernet towards the equipment needing management.
But having a console->serial is significantly less complex than console->IP_Stack->ethernet. So many more things to go wrong. I've never had a device that had a faulty serial port. I have seen numerous faulty or misbehaving network ports. I like the current trend of vendors like Juniper. Dedicated management ethernet, *and* serial console port. Best of both worlds. -Randy
I completely disagree. The ability for serial to go over POTS makes it ridiculously cheap compared to building a reliable ethernet connection over hundreds or thousands of miles.
This is identical to ethernet. You need external device then, dial-up modem or CPE, no difference.
The separate part is what makes it useful. The only reason you should need to access the serial port is because the network is not functioning. If you move the last resort access to be network, how do you access it when you have network issues?
Or because device is not functioning, and if OS is broken, so is your 'OOB'. Or maybe you fucked up upgrade and corrupted image? No way to recover over RS232 (RS232 image upload not supported anymore, even if it would be, Juniper is hard set to 9600bps, even if it would support 115200 it would take 12h to upload image, faster to fly with usb key).
But having a console->serial is significantly less complex than console->IP_Stack->ethernet. So many more things to go wrong. I've never had a device that had a faulty serial port. I have seen numerous faulty or misbehaving network ports.
Only thing matters is that it fails at different time to production network.
I like the current trend of vendors like Juniper. Dedicated management ethernet, *and* serial console port. Best of both worlds.
This is fully on-band port ethernet-port, relies 100% on the host OS being up and running. Replace this port with true OOB ethernet, keep RS232 for people who can't migrate day1. You never introduce new thing and kill old thing day1, this is not how things work. You add new thing, then after good amount of time phase-out old thing. -- ++ytti
On Thu, 10 Jan 2013, Randy Carpenter wrote:
How do I connect to it from many miles away when the network is down? I have connected to a misbehaving border device at a remote network via dial-up before, and was able to get it back up and running. I would not have been able to do that if the only options were ethernet or ethernet.
You have the same kind of "console router", but instead of having lots of serial ports, you have ethernet ports on it. Or you use your DWDM system OOB channel (if available). There are a lot of options.
agreed. Although, I generally don't care if it is slow, as the only need is to get the real network connected features back up and running.
With modern routers the software is a big image so if you need to re-install, serial is dysfunctional (if it even works, on XR devices it doesn't).
I completely disagree. The ability for serial to go over POTS makes it ridiculously cheap compared to building a reliable ethernet connection over hundreds or thousands of miles.
RS232 doesn't go hundreds of thousands of miles. A modem connected to a POTS port does this though.
The separate part is what makes it useful. The only reason you should need to access the serial port is because the network is not functioning. If you move the last resort access to be network, how do you access it when you have network issues?
You seem to be totally misreading everything I'm saying. Did you even read my initial email? This proposed port is OOB, it's completely separate, it's like having a built in serial console router into the chassis with connections to line cards and other devices within the chassis.
I like the current trend of vendors like Juniper. Dedicated management ethernet, *and* serial console port. Best of both worlds.
No it isn't, because if you want to be able to cut power to the device to power cycle it, you need yet another device to do that that sits inline with the power feed and if it's DC then I would imagine there aren't that many options (=expensive). -- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, Jan 10, 2013 at 1:24 AM, Randy Carpenter <rcarpen@network1.net> wrote:
On Wed, 9 Jan 2013, Randy Carpenter wrote:
My main requirements would be:
1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)
I don't understand this at all. Why can't an OOB network be ethernet based towards the equipment needing management?
How do I connect to it from many miles away when the network is down? I have connected to a misbehaving border device at a remote network via dial-up before, and was able to get it back up and running. I would not have been able to do that if the only options were ethernet or ethernet.
Dial up with PPP and then cross the ethernet? Drop off a cellular modem with IP service instead of a dialup modem? Perhaps you haven't noticed but IP over circuit-switched voice lines is giving way to voice over IP packet switched systems. That POTS line the dialup modem needs doesn't have a lot of future left.
But having a console->serial is significantly less complex than console->IP_Stack->ethernet. So many more things to go wrong. I've never had a device that had a faulty serial port. I have seen numerous faulty or misbehaving network ports.
I've had faulty serial consoles more than once but that's beside the point. Yes, ethernet-based OOB is more complex than a simple serial console. It's also a lot more effective. At this point the server vendors have gotten it down to a science where it's just as reliable and not especially expensive. Time I'd say for the big iron router vendors to follow suit. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On 1/10/2013 11:18 AM, William Herrin wrote:
On Thu, Jan 10, 2013 at 1:24 AM, Randy Carpenter <rcarpen@network1.net> wrote:
On Wed, 9 Jan 2013, Randy Carpenter wrote:
My main requirements would be:
1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)
I don't understand this at all. Why can't an OOB network be ethernet based towards the equipment needing management?
How do I connect to it from many miles away when the network is down? I have connected to a misbehaving border device at a remote network via dial-up before, and was able to get it back up and running. I would not have been able to do that if the only options were ethernet or ethernet.
Dial up with PPP and then cross the ethernet? Drop off a cellular modem with IP service instead of a dialup modem? Perhaps you haven't noticed but IP over circuit-switched voice lines is giving way to voice over IP packet switched systems. That POTS line the dialup modem needs doesn't have a lot of future left.
Nothing beats POTS in a broad power outage scenario. Numerous power outages have taken down mobile service completely while the POTS lines stayed up as it carries its own power by design. -- Randy
On (2013-01-10 11:41 -0500), Randy Whitney wrote:
Nothing beats POTS in a broad power outage scenario. Numerous power outages have taken down mobile service completely while the POTS lines stayed up as it carries its own power by design.
Is your RS232 Modem POTS powered? If POP is powerless, where will be POTS powered RS232 Modem connect to? -- ++ytti
On Jan 10, 2013, at 11:52 AM, Saku Ytti <saku@ytti.fi> wrote:
On (2013-01-10 11:41 -0500), Randy Whitney wrote:
Nothing beats POTS in a broad power outage scenario. Numerous power outages have taken down mobile service completely while the POTS lines stayed up as it carries its own power by design.
Is your RS232 Modem POTS powered?
If POP is powerless, where will be POTS powered RS232 Modem connect to?
Not sure about you, but I've used the ability for a POTS line to either ring or give me a modem tone to determine the power status at the site. - Jared
On (2013-01-10 12:08 -0500), Jared Mauch wrote:
Not sure about you, but I've used the ability for a POTS line to either ring or give me a modem tone to determine the power status at the site.
So the modem is not PSTN powered, so if it responds, pop must be powered? Wouldn't any old CPE on any access have same benefit, except you could ping it. However this has again nothing to do with the RS232 onband/eth oob on the router, you can still have your modem just fine and run the ETH OOB over it. Keeping any value you today extract from PSTN. Personally, I'd really love to see dying gasp over SNMP trap for powerloss. I was really happy when I saw ME3400 and ME3400E difference list 'dying gasp', but turns out it's some EOAM stuff which I didn't bother figuring out how to get all the way to NMS. Dying gasp trap to NMS would be neat way to see immediately in monitoring that box is down, due to losing electricity, can exclude many possible fault reasons right there. -- ++ytti
On Thu, Jan 10, 2013 at 12:08 PM, Jared Mauch <jared@puck.nether.net> wrote:
Not sure about you, but I've used the ability for a POTS line to either ring or give me a modem tone to determine the power status at the site.
- Jared
When I worked in the BBN NOC, we used the customers fax line to determine if the site still had power :) Too many times the cleaners would blow fuses when using the vacuum on the same circuit as the router. -Steve
On Thu, Jan 10, 2013 at 11:41 AM, Randy Whitney <randy.whitney@verizon.com> wrote:
Nothing beats POTS in a broad power outage scenario. Numerous power outages have taken down mobile service completely while the POTS lines stayed up as it carries its own power by design.
Carries it from somewhere that has to remain powered which typically isn't a building with an automatic generator any more. Access to the POTS lines of yesteryear is dwindling and not all that slowly. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
----- Original Message -----
From: "William Herrin" <bill@herrin.us>
On Thu, Jan 10, 2013 at 11:41 AM, Randy Whitney <randy.whitney@verizon.com> wrote:
Nothing beats POTS in a broad power outage scenario. Numerous power outages have taken down mobile service completely while the POTS lines stayed up as it carries its own power by design.
Carries it from somewhere that has to remain powered which typically isn't a building with an automatic generator any more. Access to the POTS lines of yesteryear is dwindling and not all that slowly.
Oh, I dunno, Bill. Sure there are lots more RSUs than there used to be, but at least it's not all *that* hard to tell if you're connected to one. Much easier than, say, finding out if both sides of your loop have been groomed into the same cable. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
On Fri, Jan 11, 2013 at 1:26 PM, Jay Ashworth <jra@baylink.com> wrote:
Oh, I dunno, Bill. Sure there are lots more RSUs than there used to be, but at least it's not all *that* hard to tell if you're connected to one.
Much easier than, say, finding out if both sides of your loop have been groomed into the same cable.
In the same sense that the number of real numbers is a larger infinity than the number of integers. Best of luck with either mission. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
----- Original Message -----
From: "William Herrin" <bill@herrin.us>
On Fri, Jan 11, 2013 at 1:26 PM, Jay Ashworth <jra@baylink.com> wrote:
Oh, I dunno, Bill. Sure there are lots more RSUs than there used to be, but at least it's not all *that* hard to tell if you're connected to one.
Much easier than, say, finding out if both sides of your loop have been groomed into the same cable.
In the same sense that the number of real numbers is a larger infinity than the number of integers. Best of luck with either mission.
You are suggesting that it is *at all* difficult for a technically competent end-user to determine whether a given new POTS line will go to a CO or to an RSU? Really? Do we work in *that different* corners of the world? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
On Fri, Jan 11, 2013 at 4:43 PM, Jay Ashworth <jra@baylink.com> wrote:
You are suggesting that it is *at all* difficult for a technically competent end-user to determine whether a given new POTS line will go to a CO or to an RSU?
Well, let me treat this as an opportunity to learn. How does one arrange for a POTS line ordered from the telco to travel its own dedicated copper pair all the way back to the central office building if the the tech tells you he only built it from one of the local holes in the ground? Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
I work for a rural Telecom in northwest US. Typically when I hear statements like that, it's that the tech built (strung aerially, trenched through ground, or through buried conduit) from a pedestal or other copper splice point to the customer premise. I would only expect this to go to the nearest remote terminal, or central office if there is no remote terminal. In a lot of (rural) cases, there is no direct copper between most houses and the central office, instead they have to (in most cases, depending on what copper cabling is available you are only able to reach one remote) cable you to the closest remote that has equipment, where you are aggregated and back-hauled (typically via fiber, but sometimes by T1) to the central office. If someone wanted completely physical diversity, up to the point of the CO, you would have to ask (likely a few times, and possibly being escalated to an engineering department of sorts) if your new POTS line can be homed to a different remote, or directly to the CO, ideally on a different physical cable route, assuming your goal is backhoe diversity. For a business line, they may be willing to work with you on diversity requirements. About the only way to guess if you're connected to a RSU or directly to the CO, you would have to know where the CO is, guess the approximate copper distance to it (which may involve guessing the approximate path the cable goes) and then hook up some equipment to your POTS line that measures and estimates the distance of that copper pair. Then you can guess where you might be connected to. ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Jay Ashworth" <jra@baylink.com> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, January 11, 2013 2:30:48 PM Subject: Re: OOB core router connectivity wish list On Fri, Jan 11, 2013 at 4:43 PM, Jay Ashworth <jra@baylink.com> wrote:
You are suggesting that it is *at all* difficult for a technically competent end-user to determine whether a given new POTS line will go to a CO or to an RSU?
Well, let me treat this as an opportunity to learn. How does one arrange for a POTS line ordered from the telco to travel its own dedicated copper pair all the way back to the central office building if the the tech tells you he only built it from one of the local holes in the ground? Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
The issue wasn't diversity, it was "is my POTS on Central Battery"; sorry for the comparative red herring. - jra Walter Keen <walter.keen@rainierconnect.net> wrote:
I work for a rural Telecom in northwest US.
Typically when I hear statements like that, it's that the tech built (strung aerially, trenched through ground, or through buried conduit) from a pedestal or other copper splice point to the customer premise.
I would only expect this to go to the nearest remote terminal, or central office if there is no remote terminal. In a lot of (rural) cases, there is no direct copper between most houses and the central office, instead they have to (in most cases, depending on what copper cabling is available you are only able to reach one remote) cable you to the closest remote that has equipment, where you are aggregated and back-hauled (typically via fiber, but sometimes by T1) to the central office.
If someone wanted completely physical diversity, up to the point of the CO, you would have to ask (likely a few times, and possibly being escalated to an engineering department of sorts) if your new POTS line can be homed to a different remote, or directly to the CO, ideally on a different physical cable route, assuming your goal is backhoe diversity.
For a business line, they may be willing to work with you on diversity requirements.
About the only way to guess if you're connected to a RSU or directly to the CO, you would have to know where the CO is, guess the approximate copper distance to it (which may involve guessing the approximate path the cable goes) and then hook up some equipment to your POTS line that measures and estimates the distance of that copper pair. Then you can guess where you might be connected to.
----- Original Message -----
From: "William Herrin" <bill@herrin.us> To: "Jay Ashworth" <jra@baylink.com> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, January 11, 2013 2:30:48 PM Subject: Re: OOB core router connectivity wish list
On Fri, Jan 11, 2013 at 4:43 PM, Jay Ashworth <jra@baylink.com> wrote:
You are suggesting that it is *at all* difficult for a technically competent end-user to determine whether a given new POTS line will go to a CO or to an RSU?
Well, let me treat this as an opportunity to learn. How does one arrange for a POTS line ordered from the telco to travel its own dedicated copper pair all the way back to the central office building if the the tech tells you he only built it from one of the local holes in the ground?
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
In the US, any incumbent phone carrier (ILEC), is required to have POTS lines on a power infrastructure capable of sustaining at least an 8 hour interruption in commercial power, whether it's in a remote or central office. Most companies use batteries at remotes (and put portable generators out when needed) and have permanent generators at central offices I know this is not the exact wording, but in the US at least, it's required by the FCC. I can't remember if competitive local exchange carriers (CLEC) have the same requirements. Your local carrier may or may not be in compliance with having (battery/generator) there to sustain 8 hours of operation, and I doubt they would tell you details of their power systems. ----- Original Message ----- From: "Jay Ashworth" <jra@baylink.com> To: "Walter Keen" <walter.keen@rainierconnect.net>, "William Herrin" <bill@herrin.us> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, January 11, 2013 4:09:25 PM Subject: Re: OOB core router connectivity wish list The issue wasn't diversity, it was "is my POTS on Central Battery"; sorry for the comparative red herring. - jra Walter Keen <walter.keen@rainierconnect.net> wrote: I work for a rural Telecom in northwest US. Typically when I hear statements like that, it's that the tech built (strung aerially, trenched through ground, or through buried conduit) from a pedestal or other copper splice point to the customer premise. I would only expect this to go to the nearest remote terminal, or central office if there is no rem ote terminal. In a lot of (rural) cases, there is no direct copper between most houses and the central office, instead they have to (in most cases, depending on what copper cabling is available you are only able to reach one remote) cable you to the closest remote that has equipment, where you are aggregated and back-hauled (typically via fiber, but sometimes by T1) to the central office. If someone wanted completely physical diversity, up to the point of the CO, you would have to ask (likely a few times, and possibly being escalated to an engineering department of sorts) if your new POTS line can be homed to a different remote, or directly to the CO, ideally on a different physical cable route, assuming your goal is backhoe diversity. For a business line, they may be willing to work with you on diversity requirements. About the only way to guess if you're connected to a RSU or directly to the CO, you would have to know where the CO is, guess the approximate copper distance to it (which may involve guessing the approximate path the cable goes) and then hook up some equipment to your POTS line that measures and estimates the distance of that copper pair. Then you can guess where you might be connected to. ----- Original Message ----- From: "William Herrin" <bill@herrin.us> < b>To: "Jay Ashworth" <jra@baylink.com> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, January 11, 2013 2:30:48 PM Subject: Re: OOB core router connectivity wish list On Fri, Jan 11, 2013 at 4:43 PM, Jay Ashworth <jra@baylink.com> wrote:
You are suggesting that it is *at all* difficult for a technically competent end-user to determine whether a given new POTS line will go to a CO or to an RSU?
Well, let me treat this as an opportunity to learn. How does one arrange for a POTS line ordered from the telco to travel its own dedicated copper pair all the way back to the central office building if the the tech tells you he only built it from one of the local holes in the ground? Regards, Bill Herrin -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
Sure. I assume it on real wire centers, I don't on RSUs or carrier. Luckily it's easy to tell which is which, in most cases. Walter Keen <walter.keen@rainierconnect.net> wrote:
In the US, any incumbent phone carrier (ILEC), is required to have POTS lines on a power infrastructure capable of sustaining at least an 8 hour interruption in commercial power, whether it's in a remote or central office. Most companies use batteries at remotes (and put portable generators out when needed) and have permanent generators at central offices
I know this is not the exact wording, but in the US at least, it's required by the FCC. I can't remember if competitive local exchange carriers (CLEC) have the same requirements.
Your local carrier may or may not be in compliance with having (battery/generator) there to sustain 8 hours of operation, and I doubt they would tell you details of their power systems.
----- Original Message -----
From: "Jay Ashworth" <jra@baylink.com> To: "Walter Keen" <walter.keen@rainierconnect.net>, "William Herrin" <bill@herrin.us> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, January 11, 2013 4:09:25 PM Subject: Re: OOB core router connectivity wish list
The issue wasn't diversity, it was "is my POTS on Central Battery"; sorry for the comparative red herring. - jra
Walter Keen <walter.keen@rainierconnect.net> wrote:
I work for a rural Telecom in northwest US.
Typically when I hear statements like that, it's that the tech built (strung aerially, trenched through ground, or through buried conduit) from a pedestal or other copper splice point to the customer premise.
I would only expect this to go to the nearest remote terminal, or central office if there is no rem ote terminal. In a lot of (rural) cases, there is no direct copper between most houses and the central office, instead they have to (in most cases, depending on what copper cabling is available you are only able to reach one remote) cable you to the closest remote that has equipment, where you are aggregated and back-hauled (typically via fiber, but sometimes by T1) to the central office.
If someone wanted completely physical diversity, up to the point of the CO, you would have to ask (likely a few times, and possibly being escalated to an engineering department of sorts) if your new POTS line can be homed to a different remote, or directly to the CO, ideally on a different physical cable route, assuming your goal is backhoe diversity.
For a business line, they may be willing to work with you on diversity requirements.
About the only way to guess if you're connected to a RSU or directly to the CO, you would have to know where the CO is, guess the approximate copper distance to it (which may involve guessing the approximate path the cable goes) and then hook up some equipment to your POTS line that measures and estimates the distance of that copper pair. Then you can guess where you might be connected to.
----- Original Message -----
From: "William Herrin" <bill@herrin.us> < b>To: "Jay Ashworth" <jra@baylink.com> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, January 11, 2013 2:30:48 PM Subject: Re: OOB core router connectivity wish list
On Fri, Jan 11, 2013 at 4:43 PM, Jay Ashworth <jra@baylink.com> wrote:
You are suggesting that it is *at all* difficult for a technically competent end-user to determine whether a given new POTS line will go to a CO or to an RSU?
Well, let me treat this as an opportunity to learn. How does one arrange for a POTS line ordered from the telco to travel its own dedicated copper pair all the way back to the central office building if the the tech tells you he only built it from one of the local holes in the ground?
Regards, Bill Herrin
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
On Fri, Jan 11, 2013 at 7:09 PM, Jay Ashworth <jra@baylink.com> wrote:
The issue wasn't diversity, it was "is my POTS on Central Battery"; sorry for the comparative red herring.
The issue was: is my POTS going to survive an extended regional power outage that my cellular/DSL/cable modem doesn't, making it a superior OOB channel to something purely IP based during difficult conditions. A central office line will be backed by the CO's generator. An RSU line won't be, so it may give out during the outage. Could be a couple hours later but it'll still run out of power. And you don't have a whole lot of choice about which one you get. That's one reliability measure. Another reliability measure is testability: can you easily monitor whether your POTS-based OOB is operational or do you discover that Bob in accounting failed to pay the bill only when you actually need it? What about your IP-based OOB? Same? Different? Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
A POTS circuit necessarily terminates on a piece of gear with a specific CLLI, generally discernable at order time. What that gear will be, and if it's in a CO with a "real" battery plant is also known in advance. And, to tie it back on topic, the odds of a core router being in a place where its serving switch is /not/ a real CO are, I speculate, comfortably below 10%. - jra William Herrin <bill@herrin.us> wrote:
On Fri, Jan 11, 2013 at 4:43 PM, Jay Ashworth <jra@baylink.com> wrote:
You are suggesting that it is *at all* difficult for a technically competent end-user to determine whether a given new POTS line will go to a CO or to an RSU?
Well, let me treat this as an opportunity to learn. How does one arrange for a POTS line ordered from the telco to travel its own dedicated copper pair all the way back to the central office building if the the tech tells you he only built it from one of the local holes in the ground?
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
On Thu, Jan 10, 2013 at 11:41 AM, Randy Whitney <randy.whitney@verizon.com>wrote
Nothing beats POTS in a broad power outage scenario. Numerous power outages have taken down mobile service completely while the POTS lines stayed up as it carries its own power by design. -- Randy
It's been a while since I've tried, but it used to be an absolute nightmare to get POTS service in many colos. Has that changed? -Steve
Why is Satellite not a good OOB option?
From my Galaxy Note II, please excuse any mistakes.
-------- Original message -------- From: William Herrin <bill@herrin.us> Date: 01/10/2013 8:20 AM (GMT-08:00) To: Randy Carpenter <rcarpen@network1.net> Cc: nanog@nanog.org Subject: Re: OOB core router connectivity wish list On Thu, Jan 10, 2013 at 1:24 AM, Randy Carpenter <rcarpen@network1.net> wrote:
On Wed, 9 Jan 2013, Randy Carpenter wrote:
My main requirements would be:
1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)
I don't understand this at all. Why can't an OOB network be ethernet based towards the equipment needing management?
How do I connect to it from many miles away when the network is down? I have connected to a misbehaving border device at a remote network via dial-up before, and was able to get it back up and running. I would not have been able to do that if the only options were ethernet or ethernet.
Dial up with PPP and then cross the ethernet? Drop off a cellular modem with IP service instead of a dialup modem? Perhaps you haven't noticed but IP over circuit-switched voice lines is giving way to voice over IP packet switched systems. That POTS line the dialup modem needs doesn't have a lot of future left.
But having a console->serial is significantly less complex than console->IP_Stack->ethernet. So many more things to go wrong. I've never had a device that had a faulty serial port. I have seen numerous faulty or misbehaving network ports.
I've had faulty serial consoles more than once but that's beside the point. Yes, ethernet-based OOB is more complex than a simple serial console. It's also a lot more effective. At this point the server vendors have gotten it down to a science where it's just as reliable and not especially expensive. Time I'd say for the big iron router vendors to follow suit. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Antenna is pretty small now. Can back haul all alarms privately, single hop back to the teleport. Very low power consumption, and very decent throughput (we can run 100mbps+ these days, which is pricey).
From my Galaxy Note II, please excuse any mistakes.
-------- Original message -------- From: Christopher Morrow <morrowc.lists@gmail.com> Date: 01/10/2013 9:24 AM (GMT-08:00) To: Warren Bailey <wbailey@satelliteintelligencegroup.com> Cc: bill@herrin.us,rcarpen@network1.net,nanog@nanog.org Subject: Re: OOB core router connectivity wish list On Thu, Jan 10, 2013 at 12:16 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
Why is Satellite not a good OOB option?
inside iron boxes satellite signal is 'hard'. getting a roof mounted antenna is extra cost/complexity. or so some thinking goes.
On Thu, Jan 10, 2013 at 12:16 PM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
Why is Satellite not a good OOB option?
Sometimes it is, and a larger colo could probably make another few nickles selling connections to an OOB access network which included, as one of the ways in, a satellite link. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Also getting POTS line in your pop sometimes get tricky. 2G/3G modems with cheap plans cost like 10$/month (dunno about US though), thats almost same as POTS line. On 10/01/13 20:18, William Herrin wrote:
Dial up with PPP and then cross the ethernet? Drop off a cellular modem with IP service instead of a dialup modem? Perhaps you haven't noticed but IP over circuit-switched voice lines is giving way to voice over IP packet switched systems. That POTS line the dialup modem needs doesn't have a lot of future left.
On 1/11/13 02:44 , Nikolay Shopik wrote:
Also getting POTS line in your pop sometimes get tricky. 2G/3G modems with cheap plans cost like 10$/month (dunno about US though), thats almost same as POTS line.
They don't generally have public IPs (that can be arranged). verizon 4G cards have ipv6 now but cradlepoint routers for example don't support that. I had reverse tunnel from one of our DC's over a 3/4g usb dongle that had a measured availability of less than 50% which oddly I didn't consider acceptable.
On 10/01/13 20:18, William Herrin wrote:
Dial up with PPP and then cross the ethernet? Drop off a cellular modem with IP service instead of a dialup modem? Perhaps you haven't noticed but IP over circuit-switched voice lines is giving way to voice over IP packet switched systems. That POTS line the dialup modem needs doesn't have a lot of future left.
On 12.01.2013 3:44, Joel jaeggli wrote:
On 1/11/13 02:44 , Nikolay Shopik wrote:
Also getting POTS line in your pop sometimes get tricky. 2G/3G modems with cheap plans cost like 10$/month (dunno about US though), thats almost same as POTS line.
They don't generally have public IPs (that can be arranged). verizon 4G cards have ipv6 now but cradlepoint routers for example don't support that.
Sure, I forgetting this, on 2G/3G modems this just cost additional 4$/month. Otherwise reverse tunnel as you saying. One of 4G operators here is just giving away internet free for now but only 64Kbits, you only need modem. But you know coverage outside city not that great (almost non-existent)
I had reverse tunnel from one of our DC's over a 3/4g usb dongle that had a measured availability of less than 50% which oddly I didn't consider acceptable.
How is that possible?
On 10/01/13 20:18, William Herrin wrote:
Dial up with PPP and then cross the ethernet? Drop off a cellular modem with IP service instead of a dialup modem? Perhaps you haven't noticed but IP over circuit-switched voice lines is giving way to voice over IP packet switched systems. That POTS line the dialup modem needs doesn't have a lot of future left.
On Jan 12, 2013, at 2:10 AM, Nikolay Shopik <shopik@inblock.ru> wrote:
I had reverse tunnel from one of our DC's over a 3/4g usb dongle that had a measured availability of less than 50% which oddly I didn't consider acceptable.
How is that possible?
Nothing stops you from having the device auto-VPN back into some network element you operate. … Assuming it's not behind whatever network element just failed ;-) - jared
On (2013-01-09 22:05 -0500), Randy Carpenter wrote:
1. Something that is *not* network (ethernet or otherwise) (isn't that the point of OOB?)
No. This is not what OOB means. Out-of-band means not fate-sharing your production network. OOB networks are networks, running ethernet, frame-relay, ATM or something else. Just that they are broken at different time to production network, so if you fuck up your production network, you can still access your device via OOB network. The serial port you have in your router is not OOB port, it's fate-sharing the control-plane, if your OS breaks down, your serial port. Think IPMI, if you break your linux installation, you can't login to linux to reload the linux, you need IPMI/DRAC/vPro for that, which is normal ethernet.
2. Something that is standard across everything, and can be aggregated easily onto a "console server" or the like
'console server' costs more than ethernet switch. So you get less and you pay more. You can't you use RS232 to send images.
I don't really see what is wrong with with keeping the serial port as the standard.
Then you don't have to use one, you're welcome to use existing solutions, this is not robbing that that on-band RS232 from you. It's adding something new. Cisco devices which have CMP still have legacy on-band RS232, because not all people will realise immediately why and even if they do, not all people can migrate it day1 in non-greenfield. -- ++ytti
participants (29)
-
Adam Vitkovsky -
Blake Dunlap -
Charles N Wyble -
Chris Adams -
Christopher Morrow -
Dobbins, Roland -
Jamie Bowden -
Jared Mauch -
Jay Ashworth -
Jimmy Hess -
joel jaeggli -
Joel jaeggli
-
Justin M. Streiner -
Leo Bicknell -
Matthew Petach -
Michael Thomas -
Mikael Abrahamsson -
Nick Hilliard -
Nikolay Shopik -
Randy Carpenter -
Randy Whitney -
Saku Ytti -
Steve Meuse -
Steven Bellovin -
sthaug＠nethelp.no -
tglassey -
Walter Keen -
Warren Bailey -
William Herrin