I've been reading the "${VENDOR}'s support has really gotten worse lately" threads for pretty much every vendor for the past twenty years. That's not to say they've all been wrong. But it reminds me of those quotes you'll see about how "these kids today are awful and society is going to pot" and then the big reveal is that it was written in the 1950s, or 1920s, or just before the peak of Rome, or something like that. The general tendency for people to view the past as the good ol' days. My most memorable Cisco TAC disaster story. Taking away "configure" from TAC wouldn't have saved us. The guy simply reloaded the switch without asking. The core switch for a building with hundreds of end users. In the middle of the day. The building with most of the C-level execs. Our management was pi-i-i-issed. That got escalated pretty high, pretty quickly. And quick policy change that we did not give TAC keyboard control. This was about ten years ago. On Tue, Mar 12, 2024 at 7:47 AM Lyden, John C <lyden@rowan.edu> wrote:
when a TAC engineer wanted to bounce our Voice VLAN SVI in the middle of an *airport* production day. I about turned over my desk trying to wrest the remote control session back from him before he hit enter on the shut. Since then, I have had to go through a not insignificant evaluation period of TAC engineers before I let them take control of a remote session, and it is now simply pure instinct to log SSH sessions.
Picture it, Cisco TAC, on a troubleshooting call, runs 'no ip routing' and hits enter before our engineer could scream "NO" at 11:30AM on a core L3 on a college campus.
RCA afterwards:
1. "Always log all terminals (we prefer SecureCRT) from Windows bastion host to OneDrive or Google Drive" 2. New CiscoTAC TACACS login created allowing Enable but Denying "configure" as a command. When you troubleshoot, you log in as CiscoTAC.
The CiscoTAC tacacs profile description in Clearpass makes it clear why it's there. I left the curse words out.
-J
John C. Lyden Associate Director, Network Operations Division of Information Resources & Technology Rowan University