- Test - lists.nanog.org

RE: Status of FCAPS model? Useful? Obsolete?
by Hannigan, Martin 11 Nov '04

11 Nov '04

> -----Original Message----- > From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] > Sent: Monday, November 08, 2004 8:52 PM > To: nanog(a)merit.edu > Subject: Status of FCAPS model? Useful? Obsolete? > > > > Someone at Forrester research wrote an article in 2003 that > said FCAPS was > an obsolete model because > it was conceived during a time when mainframes were in use. > I haven't > read the article but the > premise of it seemed a bit overboard to me. > > Does the FCAPS model still hold currency among network > managers/engineers > today? What's FCAPS? -M<

6 5

Cisco Security Advisory: Cisco IOS DHCP Blocked Interface Denial-of-Service
by Cisco Systems Product Security Incident Response Team 10 Nov '04

10 Nov '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Cisco Security Advisory: Cisco IOS DHCP Blocked Interface Denial-of-Service =============================================================================== Revision 1.0 For Public Release 2004 November 10 1700 UTC (GMT) - ------------------------------------------------------------------------------- Contents Summary Affected Products Details Impact Software Versions and Fixes Obtaining Fixed Software Workarounds Exploitation and Public Announcements Status of This Notice: FINAL Distribution Revision History Cisco Security Procedures - ------------------------------------------------------------------------------- Summary ======= Cisco IOS devices running branches of Cisco IOS version 12.2S that have Dynamic Host Configuration Protocol (DHCP) server or relay agent enabled, even if not configured, are vulnerable to a denial of service where the input queue becomes blocked when receiving specifically crafted DHCP packets. Cisco is providing free fixed software to address this issue. There are also workarounds to mitigate this vulnerability. This issue was introduced by the fix included in CSCdx46180 and is being tracked by Cisco Bug ID CSCee50294 ( registered customers only) . This advisory is available at http://www.cisco.com/warp/public/707/cisco-sa-20041110-dhcp.shtml. Affected Products ================= This vulnerability was introduced by the fix for CSCdx46180, and was integrated in Cisco IOS 12.2(14)SZ and 12.2(18)S. The following Cisco products running Cisco IOS version 12.2(14)SZ, or a variant of Cisco IOS 12.2(18)S (as listed in the following section) and higher are affected by this vulnerability. * Cisco 7200, 7300, 7500 platforms * Cisco 2650, 2651, 2650XM, 2651XM Multiservice platform * Cisco ONS15530, ONS15540 * Cisco Catalyst 4000, Sup2plus, Sup3, Sup4 and Sup5 modules * Cisco Catalyst 4500, Sup2Plus TS * Cisco Catalyst 4948, 2970, 3560, and 3750 * Cisco Catalyst 6000, Sup2/MSFC2 and Sup720/MSFC3 * Cisco 7600 Sup2/MSFC2 and Sup720/MSFC3 Vulnerable Products - ------------------- This issue affects only Cisco devices running affected Cisco IOS versions 12.2 (18)EW, 12.2(18)EWA, 12.2(14)SZ, 12.2(18)S, 12.2(18)SE, 12.2(18)SV, 12.2(18)SW and higher that do not have the configuration command no service dhcp. It is not necessary for DHCP server or relay agent to be configured in order for this vulnerability to be present and exploited; "service dhcp" is enabled by default in IOS and is the only setting necessary (in addition to interface addresses) for exploitation of this vulnerability. This includes routers as well as switches and line cards which run Cisco IOS software. Cisco devices which do not run Cisco IOS software are not affected. Cisco devices running affected Cisco IOS software with the command no service dhcp enabled are not affected. To determine the software running on a Cisco product, log in to the device and issue the show version command to display the system banner. Cisco IOS software will identify itself as "Internetwork Operating System Software" or simply "IOS." On the next line of output, the image name will be displayed between parentheses, followed by "Version" and the IOS release name. Other Cisco devices will not have the show version command or will give different output. The following example identifies a Cisco product running IOS release 12.0(3) with an installed image name of C2500-IS-L: Cisco Internetwork Operating System Software IOS (TM) 2500 Software (C2500-IS-L), Version 12.0(3), RELEASE SOFTWARE The release train label is "12.0." The next example shows a product running Cisco IOS release 12.0(2a)T1 with an image name of C2600-JS-MZ: Cisco Internetwork Operating System Software IOS (tm) C2600 Software (C2600-JS-MZ), Version 12.0(2a)T1, RELEASE SOFTWARE (fc1) Additional information about Cisco IOS release naming can be found at http://www.cisco.com/warp/public/620/1.html. Products Confirmed Not Vulnerable - --------------------------------- Cisco devices running affected Cisco IOS software with the command no service dhcp enabled are not affected. Cisco products that run any versions of IOS not listed in the Software Versions and Fixes table below, are not affected. Cisco products that do not run Cisco IOS software and are not affected by this vulnerability include, but are not limited to: * 700 series dialup routers (750, 760, and 770 series) are not affected. * WAN switching products such as the IGX, BPX and MGX lines are not affected. * No host-based software is affected. * The Cisco PIX Firewall is not affected * The Cisco LocalDirector is not affected. * The Cisco Content Engine and ACNS is not affected. * The Catalyst 2901/2902, 2948G, 2980G, 4000, 5000, and 6000 switches running CatOS. * Cisco Network Registrar is not affected. * Cisco VPN 3000 series is not affected * Cisco IOS-XR platform is not affected. * Cisco MDS 9000 family is not affected. Details ======= DHCP services allow devices to request and receive basic host configuration information from the DHCP server via the network. Cisco routers can be configured to both provide dynamic host configuration information (termed DHCP server function), and forward DHCP and BootP packets across separate broadcast domains (termed DHCP relay agent function). Cisco routers are configured to process and accept DHCP packets by default, therefore the command "service dhcp" does not appear in the running configuration display, and only the command for the disabled feature, no service dhcp, will appear in the running configuration display when the feature is disabled. The vulnerability is present, regardless if the DHCP server or relay agent configurations are present on an affected product. The only required configuration for this vulnerability in affected versions is the lack of the no service dhcp command. Certain crafted DHCP packets may be undeliverable, but will remain in the queue instead of being dropped. If a number of packets are sent that equal the size of the input queue, no more traffic will be accepted on that interface. On a blocked Ethernet interface, Address Resolution Protocol (ARP) times out after a default time of four hours, and no inbound or outbound traffic can be processed, including both IP and non-IP traffic such as IPX. The device must be rebooted to clear the input queue on the interface, and will not reload without user intervention. The attack may be repeated on all interfaces, causing the router to be remotely inaccessible, excluding the console port where DHCP packets are not processed by default and which can be used for out-of-band management and configured for remote access. Workarounds are available, and are documented in the Workarounds section below. Other types of interfaces, including but not limited to ATM, Serial and POS interfaces, are affected, but ARP is not a factor. To identify a blocked input interface, use the show interfaces command and look for the Input Queue line. The size of the input queue may keep increasing. If the current size (in this case, 76) is larger than the maximum size (75), the input queue is blocked. Router#show interface ethernet 0/0 Ethernet0/0 is up, line protocol is up Hardware is AmdP2, address is 0050.500e.f1e0 (bia 0050.500e.f1e0) Internet address is 172.16.1.9/24 MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 1/255 Encapsulation ARPA, loopback not set, keepalive set (10 sec) ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:41, output 00:00:07, output hang never Last clearing of "show interface" counters 00:07:18 Input queue: 76/75/1091/0 (size/max/drops/flushes); Total output drops: 0 !--- The 76/75 shows that this is blocked Impact ====== A device receiving these specifically crafted DHCP packets will force the inbound interface to stop processing traffic. The device may stop processing packets destined to the router, including routing protocol packets and ARP packets. No alarms will be triggered, nor will the router reload to correct itself. This vulnerability may be exercised repeatedly resulting in loss of availability until a workaround has been applied or the device has been upgraded to a fixed version of code. The effects of this vulnerability can be monitored by the same methods outlined in the white paper entitled Uses of Network Management for Monitoring the "IP Packet Blocks Input Queue" PSIRT Advisory which details methods to identify impacted devices via SNMP, RMON, and Network Management products. Software Versions and Fixes =========================== Each row of the Cisco IOS software table below describes a release train and the platforms or products for which it is intended. If a given release train is vulnerable, then the earliest possible releases that contain the fix (the "First Fixed Release") and the anticipated date of availability for each are listed in the "Rebuild" and "Maintenance" columns. A device running a release in the given train that is earlier than the release in a specific column (less than the First Fixed Release) is known to be vulnerable. The release should be upgraded at least to the indicated release or a later version (greater than or equal to the First Fixed Release label). When selecting a release, keep in mind the following definitions: * Maintenance: Most heavily tested, stable, and highly recommended release of a release train in any given row of the table. * Rebuild: Constructed from the previous maintenance or major release in the same train, it contains the fix for a specific defect. Although it receives less testing, it contains only the minimal changes necessary to repair the vulnerability. In all cases, customers should exercise caution to confirm that the devices to be upgraded contain sufficient memory and that current hardware and software configurations will continue to be supported properly by the new software release. If the information is not clear, contact the Cisco TAC for assistance as shown in the Obtaining Fixed Software section below. More information on Cisco IOS software release names and abbreviations is available at http://www.cisco.com/warp/public/620/1.html. The fixes will be available at the Software Center located at http://www.cisco.com/tacpage/sw-center/. For software installation and upgrade procedures, see http://www.cisco.com/warp/public/130/upgrade_index.shtml. For a current view of all posted and repaired images for Cisco IOS, please check the listing available to registered Cisco.com users at: http://www.cisco.com/tacpage/sw-center/sw-ios.shtml. +---------------------------------------------+ | Major Release | Availability of Repaired | | | Releases | |----------------+----------------------------| | Affected | | | | 12.2-Based | Rebuild | Maintenance | | Release | | | |----------------+--------------+-------------| | 12.2(18)EW | 12.2(18)EW2 | | |----------------+--------------+-------------| | 12.2(18)EWA | | 12.2(20)EWA | |----------------+--------------+-------------| | | 12.2(18)S6 | | | |--------------+-------------| | | 12.2(20)S4 | | |12.2(18)S |--------------+-------------| | | 12.2(22)S2 | | | |--------------+-------------| | | | 12.2(25)S | |----------------+--------------+-------------| | 12.2(18)SE | 12.2(20)SE3 | | |----------------+--------------+-------------| | 12.2(18)SV | | 12.2(24)SV | |----------------+--------------+-------------| | 12.2(18)SW | | 12.2(25)SW | |----------------+----------------------------| | 12.2(18)SXD | Not impacted | |----------------+----------------------------| | 12.2(14)SZ | Migrates to | | | | 12.2(20)S4 | | +---------------------------------------------+ Obtaining Fixed Software ======================== Customers with Service Contracts - -------------------------------- Customers with contracts should obtain upgraded software through their regular update channels. For most customers, this means that upgrades should be obtained through the Software Center on Cisco's worldwide website at http:// www.cisco.com. Customers using Third-party Support Organizations - ------------------------------------------------- Customers whose Cisco products are provided or maintained through prior or existing agreement with third-party support organizations such as Cisco Partners, authorized resellers, or service providers should contact that support organization for assistance with the upgrade, which should be free of charge. Customers without Service Contracts - ----------------------------------- Customers who purchase direct from Cisco but who do not hold a Cisco service contract and customers who purchase through third-party vendors but are unsuccessful at obtaining fixed software through their point of sale should get their upgrades by contacting the Cisco Technical Assistance Center (TAC). TAC contacts are as follows. * +1 800 553 2447 (toll free from within North America) * +1 408 526 7209 (toll call from anywhere in the world) * e-mail: tac(a)cisco.com Please have your product serial number available and give the URL of this notice as evidence of your entitlement to a free upgrade. Free upgrades for non-contract customers must be requested through the TAC. Please do not contact either "psirt(a)cisco.com" or "security-alert(a)cisco.com" for software upgrades. See http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml for additional TAC contact information, including special localized telephone numbers and instructions and e-mail addresses for use in various languages. Customers may only install and expect support for the feature sets they have purchased. By installing, downloading, accessing or otherwise using such software upgrades, customers agree to be bound by the terms of Cisco's software license terms found at http://www.cisco.com/public/sw-license-agreement.html, or as otherwise set forth at Cisco.com Downloads at http://www.cisco.com/public/sw-center/sw-usingswc.shtml. Workarounds =========== There are four possible workarounds for this vulnerability: * Disabling the dhcp service * Control Plane Policing * Two versions of Access Control Lists Disabling the DHCP Service - -------------------------- This vulnerability can be mitigated by utilizing the command: no service dhcp However, this workaround will disable all DHCP processing on the device, including the DHCP helper functionality that may be necessary in some network configurations. Control Plane Policing Feature - ------------------------------ The Control Plane Policy feature may be used to mitigate this vulnerability, as in the following example: access-list 140 deny udp host 192.168.13.1 any eq bootps access-list 140 deny udp any host 192.168.13.1 eq bootps access-list 140 deny udp any host 255.255.255.255 eq bootps access-list 140 permit udp any any eq bootps class-map match-all bootps-class match access-group 140 policy-map control-plane-policy class bootps-class police 8000 1500 1500 conform-action drop exceed-action drop control-plane service-policy input control-plane-policy For this example 192.168.13.1 is a legitimate DHCP server. Additional information on the configuration and use of the CPP feature can be found at this link: http://www.cisco.com/en/US/partner/products/sw/iosswrel/ps1838/ products_feature_guide09186a00801afad4.html. This workaround is only applicable to Cisco IOS 12.2S, as this feature is only available in Cisco IOS versions 12.2S and 12.3T. Cisco IOS 12.3T is not impacted by this advisory. Access Lists - Two Methods - -------------------------- Access lists can be applied to block DHCP/BootP traffic destined to any router interface addresses, as in the following example: In this example, the IP address 192.168.13.1 represents a legitimate DHCP server, the addresses 10.89.236.147 and 192.168.13.2 represent router interface addresses, and 192.168.61.1 represents a loopback interface on the router. In this example, any bootp/dhcp packets destined to the router interface addresses are blocked. access-list 100 remark permit bootps from the DHCP server access-list 100 permit udp host 192.168.13.1 any eq bootps access-list 100 remark deny bootps from any to router f1/0 access-list 100 deny udp any host 10.89.236.147 eq bootps access-list 100 remark deny bootps from any to router f0/0 access-list 100 deny udp any host 192.168.13.2 eq bootps access-list 100 remark deny bootps from any to router loopback1 access-list 100 deny udp any host 192.168.61.1 eq bootps access-list 100 remark permit all other traffic access-list 100 permit ip any any access-list 100 is applied to f0/0 and f1/0 physical interfaces. interface FastEthernet0/0 ip address 192.168.13.2 255.255.255.0 ip access-group 100 in interface FastEthernet1/0 ip address 10.89.236.147 255.255.255.240 ip access-group 100 in ip helper-address 192.168.13.1 An alternate configuration for the interface access-list workaround. This example would also need to be applied to all physical interfaces, but deny statements for all of the IP addresses configured on the router are not necessary in this approach. In this example, the address 192.168.13.1 represents a legitimate DHCP server. access-list 100 permit udp host 192.168.13.1 any eq bootps access-list 100 permit udp any host 192.168.13.1 eq bootps access-list 100 permit udp any host 255.255.255.255 eq bootps access-list 100 deny udp any any eq bootps interface FastEthernet0/0 ip address 192.168.13.2 255.255.255.0 ip access-group 100 in interface FastEthernet1/0 ip address 10.89.236.147 255.255.255.240 ip access-group 100 in ip helper-address 192.168.13.1 Note: These workarounds will not prevent spoofed IP packets with the source IP address set to that of the DHCP server For more information on anti-spoofing refer to http://www.cisco.com/en/US/tech/ tk648/tk361/technologies_tech_note09186a0080120f 48.shtml#sec_ip and http://www.ietf.org/rfc/rfc2827.txt. The Unicast Reverse Path Forwarding (Unicast RPF) feature helps to mitigate problems that are caused by spoofed IP source addresses. For further details, please refer to http://www.cisco.com/univercd/cc/td/doc/product/software/ios122 /122cgcr/fsecur_c/fothersf/scfrpf.htm. Exploitation and Public Announcements ===================================== The Cisco PSIRT is not aware of any public announcements or malicious use of the vulnerability described in this advisory. This vulnerability was reported to Cisco by a customer. Status of This Notice: FINAL ============================ THIS ADVISORY IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY. YOUR USE OF THE INFORMATION ON THE ADVISORY OR MATERIALS LINKED FROM THE ADVISORY IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS NOTICE AT ANY TIME. Distribution This advisory will be posted on Cisco's worldwide website at http://www.cisco.com/warp/public/707/cisco-sa-20041110-dhcp.shtml. In addition to worldwide web posting, a text version of this notice is clear-signed with the Cisco PSIRT PGP key and is posted to the following e-mail and Usenet news recipients. * cust-security-announce(a)cisco.com * first-teams(a)first.org (includes CERT/CC) * bugtraq(a)securityfocus.com * vulnwatch(a)vulnwatch.org * cisco(a)spot.colorado.edu * cisco-nsp(a)puck.nether.net * full-disclosure(a)lists.netsys.com * comp.dcom.sys.cisco(a)newsgate.cisco.com Future updates of this advisory, if any, will be placed on Cisco's worldwide website, but may or may not be actively announced on mailing lists or newsgroups. Users concerned about this problem are encouraged to check the above URL for any updates. Revision History +---------------------------------------------+ | Revision | | Initial | | 1.0 | 2004-November-10 | public | | | | release. | +---------------------------------------------+ Cisco Security Procedures Complete information on reporting security vulnerabilities in Cisco products, obtaining assistance with security incidents, and registering to receive security information from Cisco, is available on Cisco's worldwide website at http://www.cisco.com/en/US/products/products_security_vulnerability_policy.…. This includes instructions for press inquiries regarding Cisco security notices. All Cisco security advisories are available at http://www.cisco.com/go/psirt. - ------------------------------------------------------------------------------- All contents are Copyright © 1992-2004 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. -----BEGIN PGP SIGNATURE----- Version: PGP 7.0 iQA/AwUBQZJN63sxqM8ytrWQEQIT9gCg1F3Il3xJrvye98FMPwMAP8/WlfkAoOx3 +61bikQKb5rc/PFvzelpZfuP =XQ2O -----END PGP SIGNATURE-----

1 0

Fwd: Consensus declared: IETF Administrative restructuring
by Fergie (Paul Ferguson) 10 Nov '04

10 Nov '04

FYI. - ferg ---------- Forwarded Message ---------- On October 26, the IESG and IAB sent out a call for consensus to the IETF on administrative restructuring, referencing the Internet-Draft: - IAB and IESG Recommendation for IETF Administrative Restructuring <draft-iab-iesg-adminrest-rec-00.txt> Based on the feedback on this Call for Consensus, which ended November 8, we conclude that we have verified that the IETF community opinion has converged on a rough consensus to restructure our administrative support activity as described in this recommendation ("Scenario O"). This verification allows the IAB and IESG to focus our efforts on executing the AdminRest transition plan and on refining and completing the IASA BCP. Thanks to all who participated in this effort! Harald Alvestrand, for the IESG and IAB. _______________________________________________ IETF-Announce mailing list IETF-Announce(a)ietf.org https://www1.ietf.org/mailman/listinfo/ietf-announce -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(a)netzero.net or fergdawg(a)sbcglobal.net

1 0

Re: Important IPv6 Policy Issue -- Your Input Requested
by Jerry Eyers 10 Nov '04

10 Nov '04

"Get a firewall" is not a valid response when you have lusers to drop the latest netgear whatever onto their PC and dial to some provider somewhere. Your firewall is useless to protect that segment. In many cases NAT is the ONLY protection you end up with in this scenario, a scenario that is far to common in the corporate world. Jerry -------Original Message------- From: Theo de Whaat Date: 11/09/04 15:28:44 To: jeyers(a)sloancc.net Subject: Re: Important IPv6 Policy Issue -- Your Input Requested Get a firewall. You shouldn't rely on NAT to provide this functionality. How long has IPv6 been on the horizon, promising to make NAT unnecessary? >I want my inside addresses to be >non accessible from the 'real world', ever.

6 6

Sandvine
by Paul Bradford 10 Nov '04

10 Nov '04

I just want to thank everybody for the OL e-mails about sandvine...time will tell how successful service flow management tools like this will be... the geek inside sad bad bad bad messing with my packets.... Thanks, Paul -- Paul A. Bradford Senior Network Engineer Adelphia Communications AS19548 and AS7843 Voice: 814-274-1353 AIM: llyolf

1 0

OT: Akamai DNS and Yahoo
by John Neiberger 10 Nov '04

10 Nov '04

I saw many references about this in the archives but I haven't seen a solution to it and we just started seeing this problem. Beginning sometime very recently, our DNS servers are not able to resolve www.yahoo.com. I have no problem if I point my PC to an external DNS server, but when I point it back at our internal servers I get timeouts when trying to resolve that domain. A Google search turned up dozens of posts over the past few years regarding people not being able to resolve www.yahoo.com, but the solutions don't seem to apply to our environment. We're running a version of BIND 8 on Solaris 9, and it's likely that this behavior began this weekend after we applied the most recent patch cluster for Solaris 9 and rebooted the server. For quite a while, all external DNS was failing and we still have some odd intermittent problems but the most noticable issue that is 100% reproducible is the failure to resolve Yahoo addresses. I saw a few Usenet posts that mentioned this could be a problem with Extensions for DNS and the fact that DNS replies could be larger than 512 bytes. This would be a problem if you were behind a PIX firewall running a certain version of software and with a certain feature configured because it would drop all UDP DNS packets over 512 bytes. This doesn't really fit our environment so I'm still looking for answers. Any thoughts? Thanks, John --

2 1

Earth to someone with a clue at Verizon mail operations
by alex＠yuriev.com 09 Nov '04

09 Nov '04

Earth to someone Verizon... Earth to verizon... Come in verizon.... come in Verizon... We have an urgent message for you - This is called mail debugging 101. IF upon a successful authentication to your POP servers, the POP servers claim that there is no mail in the inbox while your webmail shows dozen of new unread unopened messages, (you are presented the printout of a telnet session to port 110 of mail.verizon.net)) THEN the claim that it is the "MTU on a customer DSL router" that is causing the problem is wrong especially if the problem manifests itself on a NON-DSL connection. Making these claims for several days after three levels of escalation makes your well-ran operation look incompetent. END IF;

1 0

RE: Newedge Networks meltdown?
by Peering 09 Nov '04

09 Nov '04

I can understand not wanting to get Alcatel support...we use Alcatel's NMS too and it's pricey for everything, support, upgrades, etc. (thank goodness it's now on the edge of our network and not on the core...hail Juniper!!). But you gotta do what you gotta do. I can't imagine us being down that hard that long before we got the vendor involved. Diane Turley Network Engineer Xspedius Communications Co. 636-625-7178 -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of netadm Sent: Tuesday, November 09, 2004 2:32 PM To: nanog(a)merit.edu Subject: RE: Newedge Networks meltdown? Statement from New Edge to Customers: ------------------------------------- Edger Customers and Partners, We are experiencing major network issues with our Alcatel Network Management System (NMS). Early last evening we had degradation in our primary NMS and we switched to our secondary backup NMS as per standard procedure. We then experienced similar degradation difficulties with our backup NMS. We have worked through the night on this and we have brought in Alcatel. Much of our network is stable. We estimate about 30% of our permanent virtual circuits (PVC) for customers may have been impacted. Customer affecting symptoms can take the form of performance degradation on a circuit or an actual outage in the form of failure to sync at Layer 3. We are now back on our primary NMS and we believe we have this situation stabilized. We are rebuilding our customer PVC's and the situation should improve throughout the day. We have added extra resources in NOC and our call centers. Our Service Level Agreements (SLA) call for 99.5 % up time and we have always far surpassed these SLA targets. This is the first outage of its type in the five year history of New Edge Networks. We are deeply embarrassed by this network outage and we apologize to our customers. We will do everything possible to fix this and we will communicate on the hour as we do so. Dan Moffat CEO New Edge Networks -----Original Message----- From: Bubba Parker [mailto:sysadmin@citynetwireless.net] Sent: Tuesday, November 09, 2004 11:10 AM To: Mark Radabaugh Cc: nanog(a)merit.edu Subject: Re: Newedge Networks meltdown? I had a DS3 go down with them last night, they said that a lot of their PVCs all showed active must had to be bounced to continue working, and they'd only know which ones were down if people called and told so. On Tue, Nov 09, 2004 at 07:40:36AM -0500, Mark Radabaugh wrote: > > Seems there is a major problem within NewEdge networks this morning. > From the recording on the NOC line it sounds like they lost an internal > database and PVC's at many of the switches. > > Anyone know what happened? Let the speculation begin :-) > > Mark Radabaugh > Amplex > -- Bubba Parker sysadmin(a)citynetwireless.net CityNet LLC http://www.citynetinfo.com/

1 0

RE: Newedge Networks meltdown?
by netadm 09 Nov '04

09 Nov '04

Statement from New Edge to Customers: ------------------------------------- Edger Customers and Partners, We are experiencing major network issues with our Alcatel Network Management System (NMS). Early last evening we had degradation in our primary NMS and we switched to our secondary backup NMS as per standard procedure. We then experienced similar degradation difficulties with our backup NMS. We have worked through the night on this and we have brought in Alcatel. Much of our network is stable. We estimate about 30% of our permanent virtual circuits (PVC) for customers may have been impacted. Customer affecting symptoms can take the form of performance degradation on a circuit or an actual outage in the form of failure to sync at Layer 3. We are now back on our primary NMS and we believe we have this situation stabilized. We are rebuilding our customer PVC's and the situation should improve throughout the day. We have added extra resources in NOC and our call centers. Our Service Level Agreements (SLA) call for 99.5 % up time and we have always far surpassed these SLA targets. This is the first outage of its type in the five year history of New Edge Networks. We are deeply embarrassed by this network outage and we apologize to our customers. We will do everything possible to fix this and we will communicate on the hour as we do so. Dan Moffat CEO New Edge Networks -----Original Message----- From: Bubba Parker [mailto:sysadmin@citynetwireless.net] Sent: Tuesday, November 09, 2004 11:10 AM To: Mark Radabaugh Cc: nanog(a)merit.edu Subject: Re: Newedge Networks meltdown? I had a DS3 go down with them last night, they said that a lot of their PVCs all showed active must had to be bounced to continue working, and they'd only know which ones were down if people called and told so. On Tue, Nov 09, 2004 at 07:40:36AM -0500, Mark Radabaugh wrote: > > Seems there is a major problem within NewEdge networks this morning. > From the recording on the NOC line it sounds like they lost an internal > database and PVC's at many of the switches. > > Anyone know what happened? Let the speculation begin :-) > > Mark Radabaugh > Amplex > -- Bubba Parker sysadmin(a)citynetwireless.net CityNet LLC http://www.citynetinfo.com/

1 0

RE: Newedge Networks meltdown?
by Kathryn Kessey 09 Nov '04

09 Nov '04

> Anyone know what happened? Let the speculation begin :-) they use alcatel's centralized nms with one nms instance to reroute all paths (no rerouting capability without the nms, no pnni.) They won't pay for upgrades or support, and have such a contentious relationship with alcatel that alcatel demands cash to answer a support call (and that's not alcatel's fault.) Once they had a major nms database problem, it took their finance staff ~8 hours of outage time to decide to spend the money for the support call, then the ceo demanded to know why their support staff took so long to fix the problem. Don't yell at their support people, this attitude comes from the top. -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Bubba Parker Sent: Tuesday, November 09, 2004 10:10 AM To: Mark Radabaugh Cc: nanog(a)merit.edu Subject: Re: Newedge Networks meltdown? I had a DS3 go down with them last night, they said that a lot of their PVCs all showed active must had to be bounced to continue working, and they'd only know which ones were down if people called and told so. On Tue, Nov 09, 2004 at 07:40:36AM -0500, Mark Radabaugh wrote: > > Seems there is a major problem within NewEdge networks this morning. > From the recording on the NOC line it sounds like they lost an internal > database and PVC's at many of the switches. > > Anyone know what happened? Let the speculation begin :-) > > Mark Radabaugh > Amplex > -- Bubba Parker sysadmin(a)citynetwireless.net CityNet LLC http://www.citynetinfo.com/

1 0