This last Saturday (29 Mar 2003), about 4pm Eastern time my router -- for lack of a better term -- wigged out. I was able to ping to & through it, however any attempt to get a TCP connection (specifically ssh and http) was almost immediately terminated. I think DNS was working fine, which would hint that UDP was getting through as well, but I won't swear to that in court. After convincing someone to drive to its location and do a power cycle, it rebooted happily and has run fine since. My mrtg graphs show that the CPU was pegged at 100% during the time it was acting up; memory was fine; traffic was (not surprisingly) very low -- and no spike prior to the CPU getting pegged. I've been running this version of IOS since it was released as a response to the flaw found in SNMP.... and the router has been rock solid! CPU is normally 15-20% with occasional spikes, but never for long. Memory erodes slowly, but never dropping below 20MB. Has anyone seen anything like this before? Basically, I'm wondering whether this may be an IOS bug or whether I may have hardware on its way out or whether this was some kind of new crafty DoS attack. TIA! Mark J. Scheller (scheller@u1.net)
We had what I would say is exactly the same problem last Thursday around 3:00am. The traffic lights on the router were pegged solid as usual, so it appeard to be up and running, but not really passing any useful traffic. Telnetting to it was pretty much useless, although it did glimmer to work for a minute but not enough to get in and see what was going on. It did not reload itself. We power cycled it, and it was fine. Running c7200-jk9o3s-mz.122-8.T5.bin Dan. "Mark J. Scheller" wrote:
This last Saturday (29 Mar 2003), about 4pm Eastern time my router -- for lack of a better term -- wigged out. I was able to ping to & through it, however any attempt to get a TCP connection (specifically ssh and http) was almost immediately terminated. I think DNS was working fine, which would hint that UDP was getting through as well, but I won't swear to that in court.
After convincing someone to drive to its location and do a power cycle, it rebooted happily and has run fine since. My mrtg graphs show that the CPU was pegged at 100% during the time it was acting up; memory was fine; traffic was (not surprisingly) very low -- and no spike prior to the CPU getting pegged.
I've been running this version of IOS since it was released as a response to the flaw found in SNMP.... and the router has been rock solid! CPU is normally 15-20% with occasional spikes, but never for long. Memory erodes slowly, but never dropping below 20MB.
Has anyone seen anything like this before? Basically, I'm wondering whether this may be an IOS bug or whether I may have hardware on its way out or whether this was some kind of new crafty DoS attack.
TIA!
Mark J. Scheller (scheller@u1.net)
Wow thought I was alone in the world on that one. I dont run a web server on my VXR but telnet and ssh did indeed go away. this was after about 250 days of uptime. I had been very happy with this version of IOS. I was able to access the router OOB on the console port so it wasnt too urgent, and much like you guys a reboot fixed everything. I can swear in a court of law that everything else seemed to work fine (Save the normal cef bugs and general other IOS Roulette thingys) c7200-ik2s-mz.121-5.T10.bin -Scotty ----- Original Message ----- From: "Dan Armstrong" <dan@beanfield.com> To: "Mark J. Scheller" <scheller@u1.net>; <nanog@merit.edu> Sent: Tuesday, April 01, 2003 2:05 PM Subject: Re: Router too busy???
We had what I would say is exactly the same problem last Thursday around
The traffic lights on the router were pegged solid as usual, so it appeard to be up and running, but not really passing any useful traffic. Telnetting to it was pretty much useless, although it did glimmer to work for a minute but not enough to get in and see what was going on. It did not reload itself. We power cycled it, and it was fine.
Running c7200-jk9o3s-mz.122-8.T5.bin
Dan.
"Mark J. Scheller" wrote:
This last Saturday (29 Mar 2003), about 4pm Eastern time my router -- for lack of a better term -- wigged out. I was able to ping to & through it, however any attempt to get a TCP connection (specifically ssh and http) was almost immediately terminated. I think DNS was working fine, which would hint
UDP was getting through as well, but I won't swear to that in court.
After convincing someone to drive to its location and do a power cycle, it rebooted happily and has run fine since. My mrtg graphs show that the CPU was pegged at 100% during the time it was acting up; memory was fine;
(not surprisingly) very low -- and no spike prior to the CPU getting
3:00am. that traffic was pegged.
I've been running this version of IOS since it was released as a
response to
the flaw found in SNMP.... and the router has been rock solid! CPU is normally 15-20% with occasional spikes, but never for long. Memory erodes slowly, but never dropping below 20MB.
Has anyone seen anything like this before? Basically, I'm wondering whether this may be an IOS bug or whether I may have hardware on its way out or whether this was some kind of new crafty DoS attack.
TIA!
Mark J. Scheller (scheller@u1.net)
What was the process that was eating the CPU ? ---Mike At 07:29 PM 4/1/2003 -0500, k. scott bethke wrote:
Wow thought I was alone in the world on that one. I dont run a web server on my VXR but telnet and ssh did indeed go away. this was after about 250 days of uptime. I had been very happy with this version of IOS.
I was able to access the router OOB on the console port so it wasnt too urgent, and much like you guys a reboot fixed everything. I can swear in a court of law that everything else seemed to work fine (Save the normal cef bugs and general other IOS Roulette thingys)
c7200-ik2s-mz.121-5.T10.bin
-Scotty
----- Original Message ----- From: "Dan Armstrong" <dan@beanfield.com> To: "Mark J. Scheller" <scheller@u1.net>; <nanog@merit.edu> Sent: Tuesday, April 01, 2003 2:05 PM Subject: Re: Router too busy???
We had what I would say is exactly the same problem last Thursday around
The traffic lights on the router were pegged solid as usual, so it appeard to be up and running, but not really passing any useful traffic. Telnetting to it was pretty much useless, although it did glimmer to work for a minute but not enough to get in and see what was going on. It did not reload itself. We power cycled it, and it was fine.
Running c7200-jk9o3s-mz.122-8.T5.bin
Dan.
"Mark J. Scheller" wrote:
This last Saturday (29 Mar 2003), about 4pm Eastern time my router -- for lack of a better term -- wigged out. I was able to ping to & through it, however any attempt to get a TCP connection (specifically ssh and http) was almost immediately terminated. I think DNS was working fine, which would hint
UDP was getting through as well, but I won't swear to that in court.
After convincing someone to drive to its location and do a power cycle, it rebooted happily and has run fine since. My mrtg graphs show that the CPU was pegged at 100% during the time it was acting up; memory was fine;
(not surprisingly) very low -- and no spike prior to the CPU getting
3:00am. that traffic was pegged.
I've been running this version of IOS since it was released as a
response to
the flaw found in SNMP.... and the router has been rock solid! CPU is normally 15-20% with occasional spikes, but never for long. Memory erodes slowly, but never dropping below 20MB.
Has anyone seen anything like this before? Basically, I'm wondering whether this may be an IOS bug or whether I may have hardware on its way out or whether this was some kind of new crafty DoS attack.
TIA!
Mark J. Scheller (scheller@u1.net)
-------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike
Thus spake "Mark J. Scheller" <scheller@u1.net>
After convincing someone to drive to its location and do a power cycle, it rebooted happily and has run fine since. My mrtg graphs show that the CPU was pegged at 100% during the time it was acting up; memory was fine; traffic was (not surprisingly) very low -- and no spike prior to the CPU getting pegged. ... Has anyone seen anything like this before? Basically, I'm wondering whether this may be an IOS bug or whether I may have hardware on its way out or whether this was some kind of new crafty DoS attack.
In my experience, this is most often caused by overzealous NMS types "accidentally" downloading the routing table every few minutes. DoS attacks against routers are thankfully pretty rare, but it's possible. Since you didn't list the IOS version you're running, I can't comment on the odds of this being a bug. S Stephen Sprunk "God does not play dice." --Albert Einstein CCIE #3723 "God is an inveterate gambler, and He throws the K5SSS dice at every possible opportunity." --Stephen Hawking
participants (5)
-
Dan Armstrong -
k. scott bethke -
Mark J. Scheller -
Mike Tancsa -
Stephen Sprunk