Anyone having issues with Equinix IX out of Ashburn?
At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. Equinix is not responding to our support requests, either they are overloaded with support requests or all on holiday. Curious if others know if there are known issues at this site or is it just us. Sincerely, Nick Ellermann - CTO & VP Cloud Services BroadAspect E: nellermann@broadaspect.com<mailto:nellermann@broadaspect.com> P: 703-297-4639 F: 703-996-4443 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
We've seen issues as well. We've just started to turn the exchange up again and check if it's fixed. On 27/11/15 15:03, Nick Ellermann wrote:
At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. Equinix is not responding to our support requests, either they are overloaded with support requests or all on holiday. Curious if others know if there are known issues at this site or is it just us.
Sincerely, Nick Ellermann - CTO & VP Cloud Services BroadAspect
E: nellermann@broadaspect.com<mailto:nellermann@broadaspect.com> P: 703-297-4639 F: 703-996-4443
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
-- Jeroen Wunnink IP Engineering Manager - Hibernia Networks Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300 Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623 jeroen.wunnink@hibernianetworks.com www.hibernianetworks.com This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Networks has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment.
Hi, Seems it was some icmpv6 neighbor solicitation multicast storm. Got about > 800kpps this morning for about 4hours... Strange there is no rate limits somewhere... Regards
Storm caused by an L2 loop, malicious attack, bug in router code, or something else? Frank -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Xavier Beaudouin Sent: Friday, November 27, 2015 8:22 AM To: nanog@nanog.org Subject: Re: Anyone having issues with Equinix IX out of Ashburn? Hi, Seems it was some icmpv6 neighbor solicitation multicast storm. Got about > 800kpps this morning for about 4hours... Strange there is no rate limits somewhere... Regards
All of our IX peers dropped around 11 UTC. All recovered about 30 mins later. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jeroen Wunnink Sent: Friday, November 27, 2015 8:13 AM To: nanog@nanog.org Subject: Re: Anyone having issues with Equinix IX out of Ashburn? We've seen issues as well. We've just started to turn the exchange up again and check if it's fixed. On 27/11/15 15:03, Nick Ellermann wrote:
At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. Equinix is not responding to our support requests, either they are overloaded with support requests or all on holiday. Curious if others know if there are known issues at this site or is it just us.
Sincerely, Nick Ellermann - CTO & VP Cloud Services BroadAspect
E: nellermann@broadaspect.com<mailto:nellermann@broadaspect.com> P: 703-297-4639 F: 703-996-4443
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
-- Jeroen Wunnink IP Engineering Manager - Hibernia Networks Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300 Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623 jeroen.wunnink@hibernianetworks.com www.hibernianetworks.com This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Networks has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment.
I think we saw an issue like this a few weeks back in Chicago. It took them longer than I would have expected to fix it, later they ultimately ended up upgrading software I think. Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnstong@westmancom.com think green; don't print this email. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Baker, Byrn Sent: Friday, November 27, 2015 8:32 AM To: nanog@nanog.org Subject: RE: Anyone having issues with Equinix IX out of Ashburn? All of our IX peers dropped around 11 UTC. All recovered about 30 mins later. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jeroen Wunnink Sent: Friday, November 27, 2015 8:13 AM To: nanog@nanog.org Subject: Re: Anyone having issues with Equinix IX out of Ashburn? We've seen issues as well. We've just started to turn the exchange up again and check if it's fixed. On 27/11/15 15:03, Nick Ellermann wrote:
At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. Equinix is not responding to our support requests, either they are overloaded with support requests or all on holiday. Curious if others know if there are known issues at this site or is it just us.
Sincerely, Nick Ellermann - CTO & VP Cloud Services BroadAspect
E: nellermann@broadaspect.com<mailto:nellermann@broadaspect.com> P: 703-297-4639 F: 703-996-4443
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
-- Jeroen Wunnink IP Engineering Manager - Hibernia Networks Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300 Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623 jeroen.wunnink@hibernianetworks.com www.hibernianetworks.com This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Networks has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment.
Received a notification shortly after reporting the problem at 6am this morning about "issues" on the Exchange. We saw packet input on our routers in upwards of 350K pps, with all peers disabled. Looked to be a broadcast/multicast storm. Seems to be good now, but no explanation of what was found. -evt
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jeroen Wunnink Sent: Friday, November 27, 2015 9:13 AM To: nanog@nanog.org Subject: Re: Anyone having issues with Equinix IX out of Ashburn?
We've seen issues as well. We've just started to turn the exchange up again and check if it's fixed.
At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. Equinix is not responding to our support requests, either they are overloaded with support requests or all on holiday. Curious if others know if there are known issues at this site or is it just us.
Sincerely, Nick Ellermann - CTO & VP Cloud Services BroadAspect
E: nellermann@broadaspect.com<mailto:nellermann@broadaspect.com> P: 703-297-4639 F: 703-996-4443
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received
On 27/11/15 15:03, Nick Ellermann wrote: this in error, please contact the sender and delete the e-mail and its attachments from all computers.
--
Jeroen Wunnink IP Engineering Manager - Hibernia Networks Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300 Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623 jeroen.wunnink@hibernianetworks.com www.hibernianetworks.com
This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Networks has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment.
participants (7)
-
Baker, Byrn -
Eric Van Tol -
Frank Bulk -
Graham Johnston -
Jeroen Wunnink -
Nick Ellermann -
Xavier Beaudouin