Strange Cisco 6503 problem
Hi all, I experienced a strange problem with one of our Cisco 6503 routers - right after the terminal PC connected to the router via console is rebooted the router reboots itself. Even when there is no Eth connection to the PC the situations is the same - reboot follows. I tried to check the messages the router sends: *: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 31 seconds [1/0] :: No EOBC input, dump debuginfo, interval 10, times 3 : %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 60 seconds [1/0] : %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 90 seconds [1/0] : %FIB-3-FIBDISABLE: Fatal error, slot/cpu 1/0: IPC Failure: timeout : %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 120 seconds [1/0] : %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 150 seconds [1/0]* All the possible reasons according to Cisco that may caused that case will be checked soon because the router is in operational mode although some of them looks strange: /*CPU_MONITOR-3-TIMED_OUT or CPU_MONITOR-6-NOT_HEARD Problem The switch reports these error messages: CPU_MONITOR-3-TIMED_OUT: CPU monitor messages have failed, resetting system CPU_MONITOR-6-NOT_HEARD: CPU monitor messages have not been heard for [dec] seconds Description These messages indicate that CPU monitor messages have not been heard for a significant amount of time. A time-out most probably occurs, which resets the system. [dec] is the number of seconds. The problem possibly occurs because of these reasons: * Badly seated line card or module * Bad ASIC or bad backplane * Software bugs * Parity error * High traffic in the Ethernet out of band channel (EOBC) channel The EOBC channel is a half duplex channel that services many other functions, which includes Simple Network Management Protocol (SNMP) traffic and packets that are destined to the switch. If the EOBC channel is full of messages because of a storm of SNMP traffic, then the channel is subjected to collisions. When this happens, EOBC is possibly not able to carry IPC messages. This makes the switch display the error message. Workaround Reseat the line card or module. If a maintenance window can be scheduled, reset the switch in order to clear any transient issues. ######################################### Error Message %ERR_DET-5-ERR_DET_NO_EOBC_INPUT: No EOBC input, dump debuginfo, interval %u, times %u Explanation No Ethernet Out-of-Band Channel (EOBC) input was received. Debugging information will be dumped. Recommended Action No action is required.*/ I'm curious if some of you faced such a problem - reboot of the router caused by the console connection. Here is some brief info about the router: *Supervisor:* Mod Ports Card Type Model --- ----- -------------------------------------- ------------------ ----------- 1 9 Supervisor Engine 32 8GE (Active) WS-SUP32-GE-3B *Processor:* cisco WS-C6503-E (R7000) processor *IOS:* IOS (tm) s3223_rp Software (s3223_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF8, RELEASE SOFTWARE (fc2) Thank you in advance for all your replays. Best~
Dean Belev wrote:
I'm curious if some of you faced such a problem - reboot of the router caused by the console connection.
I once managed to send a BREAK signal to a 3640 by plugging in a console cable. At the time, it was a pretty key router in the network and sat at the rommon> prompt :) I had that down to static somewhere, as it's the only explanation I could find. Peter
----- Original Message ---- From: Peter Hicks <peter.hicks@poggs.co.uk> To: Dean Belev <dbelev@gmail.com>
I'm curious if some of you faced such a problem - reboot of the router caused by the console connection.
I once managed to send a BREAK signal to a 3640 by plugging in a console cable. At the time, it was a pretty key router in the network and sat at the rommon> prompt :)
I had that down to static somewhere, as it's the only explanation I could find.
Certain serial speed mismatches get interpreted as BREAK - I routinely use "space bar at 1200" to password crack routers where they are expecting 9600. David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com
On Jan 28, 2010, at 6:15 PM, Peter Hicks wrote:
Dean Belev wrote:
I'm curious if some of you faced such a problem - reboot of the router caused by the console connection.
I once managed to send a BREAK signal to a 3640 by plugging in a console cable. At the time, it was a pretty key router in the network and sat at the rommon> prompt :)
I had that down to static somewhere, as it's the only explanation I could find.
Actually, it's not at all surprising, but it depends on the UART or equivalent. A serial line has two states, "mark" -- a 1-bit -- and "space" (guess). Normally, the line is at "mark", which corresponds to a voltage of -3V:-25V at the receiver. Space is +3V:+25V; -3V:+3V is undefined. (http://www.lammertbies.nl/comm/info/RS-232_specs.html is pretty good, and as far as I remember quite accurate, though it's ~20 years since I used a breakout box.) Now -- a break signal is normally a "long space", a 0 signal that lasts too long, often about .25 seconds. It originally got the name because it looked like a break in the teletype line; teletypes used a current loop standard (don't ask). More precisely -- an asynchronous byte is followed by a "stop bit", which isn't so much a bit as a time interval -- one bit-time -- during which the signal must be in the mark state. If you're sending at the wrong speed or send break -- something that's holding the line at space for long enough that it will run into the stop bits at any speed -- the UART will detect the problem; this is sometimes known as a "framing error". So -- when you disconnect the cable, the voltage at the pin goes to 0. How should that be interpreted? If the board has a pull-up resistor to a +5V line, it will appear as a space signal; if it doesn't, it's up to the UART or equivalent, since it's undefined by the spec. --Steve Bellovin, http://www.cs.columbia.edu/~smb
On Thu, 2010-01-28 at 18:36 -0500, Steven Bellovin wrote:
Actually, it's not at all surprising, but it depends on the UART or equivalent.
and the dynamic characteristics of the power rails, to a certain extent. Sun kit is quite sensitive to this sort of thing. Zonker has a good guide to what does what and what borks in his conserver pages: http://www.conserver.com/consoles/ as well as a bucketload of pinout info for console ports and console servers in general. The whole site is good reference for younger techies born in the USB age ;) Gord -- I'm giving up the sigs - I'm on patches and gum.
Please make sure you config register is set to x2102. You shouldn't see any issues if you the correct config register. Regards Abdul -----Original Message----- From: Peter Hicks [mailto:peter.hicks@poggs.co.uk] Sent: Thu 1/28/2010 3:15 PM To: Dean Belev Cc: nanog@nanog.org Subject: Re: Strange Cisco 6503 problem Dean Belev wrote:
I'm curious if some of you faced such a problem - reboot of the router caused by the console connection.
I once managed to send a BREAK signal to a 3640 by plugging in a console cable. At the time, it was a pretty key router in the network and sat at the rommon> prompt :) I had that down to static somewhere, as it's the only explanation I could find. Peter
participants (6)
-
Abdulkadir Egal (aegal) -
David Barak -
Dean Belev -
gordon b slater -
Peter Hicks -
Steven Bellovin