Hey Everyone - We have two 7507 routers configured with dual RSP4s w/256MB RAM, VIP2-50s with 128/8MB RAM, Gig, POSIP OC3 and Fast Ethernet interfaces. These routers have run flawlessly for over two years now. But about two weeks ago, all of a sudden we started having serious crashing problems with these two routers. The routers will lose bgp connectivity (one at a time) to our upstreams (configured on each router). First, we would see a keepalive not sent message, then a bgp hold timer expire, then the bgp peering session would go down. OSPF would start crashing, then we would see the memory error messages, then all interfaces would blink off-line. (Note - we are running the max memory we can on both the RSPs and the VIPs). Within 1 minute, the exact same thing would happen to the other router. Often times we would have to reboot the router to get it to come back online. We would see the following errors and have to reboot multiple times to get the router back: %SYS-2-MALLOCFAIL: Memory allocation of 704 bytes failed from 0x60329F00, alignment 0 Pool: Processor Free: 92744 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 6 -Traceback= 6038049C 60382200 60329F08 6038DEDC %TCP-6-NOBUFF: TTY0, no buffer available -Process= "BGP Router", ipl= 0, pid= 132 %% Low on memory; try again later GigabitEthernet1/1/0: keepalive not sent We are running the latest S train IOS patched for the IPV4 issue - however downgrading to the code we had run for the previous year did not solve the problem, nor did replacing the RSPs, VIPs and interfaces with new cards. In addition, we have complied with the Cisco recommendations for mitigating the effects of the Nachi Worm. http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186... We also shut down one of the routers totally and the other router still experienced the same issue. None of these updates or fixes have solved the problem. I am thinking it may have something to do with all the virus stuff running around (same thing was crashing my Lucent TNT's), but I cannot seem to get an answer from Cisco, nor can I find anyone seeing the same issues. Hopefully someone here can shed some light on this problem. Thanks in Advance Richard I fly because it releases my mind from the tyranny of petty things . .
On 9/10/03 10:58 PM, "Richard J.Sears" <rsears@adnc.com> wrote:
%SYS-2-MALLOCFAIL: Memory allocation of 704 bytes failed from 0x60329F00, alignment 0 Pool: Processor Free: 92744 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 6 -Traceback= 6038049C 60382200 60329F08 6038DEDC
%TCP-6-NOBUFF: TTY0, no buffer available -Process= "BGP Router", ipl= 0, pid= 132
%% Low on memory; try again later
Did you enable CEF? Are you dropping 92 byte ICMP packets where needed? -- Robert Blayzor, BOFH INOC, LLC rblayzor@inoc.net PGP: http://www.inoc.net/~dev/ Key fingerprint = A445 7D1E 3D4F A4EF 6875 21BB 1BAA 10FE 5748 CFE9 "I don't need parents. All I need is a recording that says, 'Go play outside!" - Calvin and Hobbes
Hi Robert, Thanks for the info. We are running dCEF...routers show about 4% CPU load and the following memory: BR02#sh mem Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 613AE340 247798976 106515996 141282980 140653360 134546752 Fast 6138E340 131080 37240 93840 93840 93788 Also, we are not blocking 92 byte ICMP due to the traceroute problems on customers networks... Thanks On Wed, 10 Sep 2003 23:17:01 -0400 Robert Blayzor <rblayzor@inoc.net> wrote:
On 9/10/03 10:58 PM, "Richard J.Sears" <rsears@adnc.com> wrote:
%SYS-2-MALLOCFAIL: Memory allocation of 704 bytes failed from 0x60329F00, alignment 0 Pool: Processor Free: 92744 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 6 -Traceback= 6038049C 60382200 60329F08 6038DEDC
%TCP-6-NOBUFF: TTY0, no buffer available -Process= "BGP Router", ipl= 0, pid= 132
%% Low on memory; try again later
Did you enable CEF? Are you dropping 92 byte ICMP packets where needed?
-- Robert Blayzor, BOFH INOC, LLC rblayzor@inoc.net PGP: http://www.inoc.net/~dev/ Key fingerprint = A445 7D1E 3D4F A4EF 6875 21BB 1BAA 10FE 5748 CFE9
"I don't need parents. All I need is a recording that says, 'Go play outside!" - Calvin and Hobbes
****************************************** Richard J. Sears Vice President American Digital Network ---------------------------------------------------- rsears@adnc.com http://www.adnc.com ---------------------------------------------------- 858.576.4272 - Phone 858.427.2401 - Fax ---------------------------------------------------- I fly because it releases my mind from the tyranny of petty things . . "Work like you don't need the money, love like you've never been hurt and dance like you do when nobody's watching."
I am not sure if this post belongs here, so I apologize if it does not. I have been experiencing some weirdness while traveling and wondered if the group has any insight into what seems to be a pretty ugly situation. I am traveling and have my lap top with me. I am staying in a hotel that offers broadband support. There are 2 of us (with 2 lap tops) sharing a room. I acquire an internet connection and sign up for the service, so get an IP address. In my case that IP address is 12.44.189.24. I disconnect my cable and pass it to my roommate. He plugs in and acquires IP address 12.44.189.47. He does the email thing for a while and then passes the cable back to me. Imagine my surprise when the network routes packets destined for his IP address (from his email server no less) to my computer. My firewall (Zone alarm) detects these incoming packets and blocks them since they are unsolicited. In further analysis of the logs, I see that there are a large number of IP addresses that are packet destinations and routed to my computer Zone Alarm detects them and blocks them. According to Zone Alarm I am getting packets for destination IP addresses as follows:12.44.189.244. 12.44.189.178 12.44.189.181 12.189.44.244 and some others too. They are all port 80 requests, identified by Zone Alarm as TCP (flags:S). This seems strange to me since they are arriving at an IP address that is different from mine. How can this happen? Is there the potential for a problem (I am thinking particularly about future guests who may not have the degree of protection (limited though it is) that Zone Alarm is affording me.)? This then got me thinking about corporate security. If I have taken my laptop and put it on an external network (e.g. the hotel network) what protections can I realistically expect, and what should my corporate IT department do to make sure my compute hasn't contracted something nasty while it was away from home. I could see that the kind of network behavior that I observed could infect a less well protected computer and thus cause me to bring an infection back to my office where it can attack from behind the corporate shields and firewalls. Any comments would be very welcome. Regards Chris Bird
Whoa stop press! You connected a computer to a public IP and zone alarm starts buzzing away.. FBI! Depends on how the hotel system works, it may be broadcasting or doing some other IP weirdness, either way its not surprising. But there is no security threat from some left over packets from old TCP/IP sessions... as for the question on corporate security, I would hope that any connection to the Internet be it a corporate LAN or a travelling user on a remote network is done from a computer which has been adequately setup to be protected from the latest vulnerabilities and is locked down as much as possible, goes without saying! This is one such way as you mention of how office networks with their fancy one stop, protects all ills firewalls are still succumb to viruses and other nasties. I'd assume your IT department enforces policies on regularly installing OS patches and updating local virus scanners as part of its security policy... right? Steve On Wed, 10 Sep 2003, Christopher Bird wrote:
I am not sure if this post belongs here, so I apologize if it does not. I have been experiencing some weirdness while traveling and wondered if the group has any insight into what seems to be a pretty ugly situation.
I am traveling and have my lap top with me. I am staying in a hotel that offers broadband support. There are 2 of us (with 2 lap tops) sharing a room. I acquire an internet connection and sign up for the service, so get an IP address. In my case that IP address is 12.44.189.24.
I disconnect my cable and pass it to my roommate. He plugs in and acquires IP address 12.44.189.47. He does the email thing for a while and then passes the cable back to me. Imagine my surprise when the network routes packets destined for his IP address (from his email server no less) to my computer. My firewall (Zone alarm) detects these incoming packets and blocks them since they are unsolicited.
In further analysis of the logs, I see that there are a large number of IP addresses that are packet destinations and routed to my computer Zone Alarm detects them and blocks them. According to Zone Alarm I am getting packets for destination IP addresses as follows:12.44.189.244. 12.44.189.178 12.44.189.181 12.189.44.244 and some others too. They are all port 80 requests, identified by Zone Alarm as TCP (flags:S).
This seems strange to me since they are arriving at an IP address that is different from mine.
How can this happen? Is there the potential for a problem (I am thinking particularly about future guests who may not have the degree of protection (limited though it is) that Zone Alarm is affording me.)?
This then got me thinking about corporate security. If I have taken my laptop and put it on an external network (e.g. the hotel network) what protections can I realistically expect, and what should my corporate IT department do to make sure my compute hasn't contracted something nasty while it was away from home. I could see that the kind of network behavior that I observed could infect a less well protected computer and thus cause me to bring an infection back to my office where it can attack from behind the corporate shields and firewalls.
Any comments would be very welcome.
Regards
Chris Bird
Christopher Bird wrote:
This seems strange to me since they are arriving at an IP address that is different from mine.
That's the function of a hub, and the reason why you don't ever want to send out sensitive information in plaintext. Your neighbor in the next room over could run a packet sniffer and capture your logins.
Mike Lewinski wrote:
Christopher Bird wrote:
This seems strange to me since they are arriving at an IP address that is different from mine.
That's the function of a hub, and the reason why you don't ever want to send out sensitive information in plaintext. Your neighbor in the next room over could run a packet sniffer and capture your logins.
Yes and no. On a broadcast network, like Ethernet, you expect to see traffic for other hosts. The issue here is that it looks like the traffic for these other hosts was getting delivered to Christopher's link-layer address. If he kicks his NIC into promiscuous mode and sees all of this traffic with a sniffer, then this is expected. That's the scenario you highlight. If, OTOH, he's sitting there and traffic for other IP addresses is getting delivered to his NIC with the NIC's MAC address in there, something strange is going on. Unless he has his NIC switched to promiscuous mode and the OS's IP stack is misbehaving badly, Zone Alarm should not see the traffic on the LAN that does not have his MAC address on it. How would a switch/router be deciding that these other IP addresses should go to his PC's NIC (MAC address)? -- Crist J. Clark crist.clark@globalstar.com Globalstar Communications (408) 933-4387
Hi, we've seen this.. yuo need to make sure you filter the nachi worm 92 byte icmp echo's on your interfaces and it will be fine. The problem seems to be input buffers which use all the memory up for some reason. Steve On Wed, 10 Sep 2003, Richard J.Sears wrote:
Hey Everyone -
We have two 7507 routers configured with dual RSP4s w/256MB RAM, VIP2-50s with 128/8MB RAM, Gig, POSIP OC3 and Fast Ethernet interfaces.
These routers have run flawlessly for over two years now. But about two weeks ago, all of a sudden we started having serious crashing problems with these two routers. The routers will lose bgp connectivity (one at a time) to our upstreams (configured on each router). First, we would see a keepalive not sent message, then a bgp hold timer expire, then the bgp peering session would go down. OSPF would start crashing, then we would see the memory error messages, then all interfaces would blink off-line. (Note - we are running the max memory we can on both the RSPs and the VIPs).
Within 1 minute, the exact same thing would happen to the other router. Often times we would have to reboot the router to get it to come back online. We would see the following errors and have to reboot multiple times to get the router back:
%SYS-2-MALLOCFAIL: Memory allocation of 704 bytes failed from 0x60329F00, alignment 0 Pool: Processor Free: 92744 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 6 -Traceback= 6038049C 60382200 60329F08 6038DEDC
%TCP-6-NOBUFF: TTY0, no buffer available -Process= "BGP Router", ipl= 0, pid= 132
%% Low on memory; try again later
GigabitEthernet1/1/0: keepalive not sent
We are running the latest S train IOS patched for the IPV4 issue - however downgrading to the code we had run for the previous year did not solve the problem, nor did replacing the RSPs, VIPs and interfaces with new cards. In addition, we have complied with the Cisco recommendations for mitigating the effects of the Nachi Worm.
http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186...
We also shut down one of the routers totally and the other router still experienced the same issue.
None of these updates or fixes have solved the problem.
I am thinking it may have something to do with all the virus stuff running around (same thing was crashing my Lucent TNT's), but I cannot seem to get an answer from Cisco, nor can I find anyone seeing the same issues.
Hopefully someone here can shed some light on this problem.
Thanks in Advance
Richard
I fly because it releases my mind from the tyranny of petty things . .
Stephen J. Wilcox wrote:
Hi, we've seen this.. yuo need to make sure you filter the nachi worm 92 byte icmp echo's on your interfaces and it will be fine. The problem seems to be input buffers which use all the memory up for some reason.
This sounds vaguely similar to the recent IOS buffers stuck issue. Pete
On Fri, 12 Sep 2003, Petri Helenius wrote:
Stephen J. Wilcox wrote:
Hi, we've seen this.. yuo need to make sure you filter the nachi worm 92 byte icmp echo's on your interfaces and it will be fine. The problem seems to be input buffers which use all the memory up for some reason.
This sounds vaguely similar to the recent IOS buffers stuck issue.
No, its quite different 1: On the vuln. the buffer filled up and could not be emptied without a reboot On nachi the buffer doesnt seem to fill and an acl or shutting the interface will solve the problem whilst the router stays up 2: On the vuln. the outcome was that the particular interface stopped forwarding traffic On nachi the router runs out of main memory and starts dropping processes because of malloc failure FYI I have only encountered the nachi problem on a few PE routers which were old and had little memory anyway eg Cisco 2500.. presumably the buffer filling isnt a memory leak and providnig there is enough spare memory the router wont be affected in this way. Steve
participants (7)
-
Christopher Bird -
Crist Clark -
Mike Lewinski -
Petri Helenius -
Richard J.Sears -
Robert Blayzor -
Stephen J. Wilcox