- Test - lists.nanog.org

Re: AS4788 Telecom Malaysia major route leak?
by Jürgen Jaritsch 15 Jun '15

15 Jun '15

This is the official feedback: Level 3's network, alongside some other ISP's, experienced service disruptions affecting customers in Europe, Asia and multiple other markets. IP, Voice and Content Delivery Network (CDN) services were affected for Level 3. The root cause of the issue was isolated to a third party Internet Service Provider in Asia that leaked internet routes resulting in traffic being sent to a destination that could not route them, which affected IP, Voice and CDN services in multiple markets. The issue has been resolved, but the provider continues working to determine the specific root cause of the incident. At this time, customer services are restored with the exception of any that pose any possible risk to the Level 3 network. Maintaining a reliable, high-performing network for our customers is our top priority. Level 3 will continue to work with the provider to prevent a recurrence. Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj(a)anexia.at<mailto:jj@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

14 21

Re: AS4788 Telecom Malaysia major route leak?
by Brandon Butterworth 15 Jun '15

15 Jun '15

> From: Rafael Possamai <rafael(a)gav.ufsc.br> > ... I assume they would have to, granted the issue lasted for a > couple hours. Now, it depends on how they define the outage A L3 outage is something you manage to open a ticket for, if you don't then it didn't happen (been there before, one of their DC lost power, transit down for around 7h, nothing) With the world trying to call them on this you can't get through so no SLA payout for you brandon

1 0

Hardware monitoring
by Rafael Possamai 15 Jun '15

15 Jun '15

Hi everyone, I know this is slightly off-topic, but since it's still related to the list, I thought I'd give it a try. I am wondering what systems are out there (open source, preferably) for data collection and processing of hardware health data (temperature, CPU clock, fan speeds, etc). Ideally brand agnostic and location agnostic as well. I know of Cacti, but it would require SNMP enabled devices AFAIK, so room/generator/misc monitors wouldn't necessarily be included. Thanks in advance. Rafael

7 10

RE: AS4788 Telecom Malaysia major route leak?
by Jürgen Jaritsch 14 Jun '15

14 Jun '15

They should verify the GBLX customer ports as well ... Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj(a)anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

1 0

Fwd: Question about EX - SRX redundancy
by Anurag Bhatia 14 Jun '15

14 Jun '15

Hello everyone! I have got two Juniper EX series switches (on virtual chassis) and two SRX devices on native clustering. I am trying to have a highly available redundancy between them with atleast 2Gbps capacity all the time but kind of failing. I followed Juniper's official page here <http://kb.juniper.net/InfoCenter/index?page=content&id=KB22474> as well as this detailed forum link here <http://forums.juniper.net/t5/SRX-Services-Gateway/Best-way-of-redundancy-be…> . I wish to have a case where devices are connected criss cross and following the documentation I get two ae bundles in EX side and one single reth bundle on SRX side. Both ae bundles on EX side have identical configuration and VLAN has both ae interfaces called up. If I do not go for criss cross connectivity like this: EX0 (ae1) >> Two Patches to SRX0 (reth1) EX1 (ae2) >> Two Patches to SRX1 (reth1) Then it works all well and redundancy works fine. In this case as long as 1 out of 4 patch is connected connectivity stays live but this has trade off that if one EX goes down then I cannot make use of other corresponding SRX. If I do criss connectivity, something like: EX0 (ae1) >> Two Patches to SRX0 (reth1) EX0 (ae1) >> One patch to SRX1 (reth1) EX1 (ae2) >> Two Patches to SRX1 (reth1) EX1 (ae2) >> One patch to SRX0 (reth1) In this config system behaves very oddly with one ae pair (and it's corresponding physical ports) working well while failover to other ae bundle fails completely. I was wondering if someone can point me out here. Appreciate your time and help! -- Anurag Bhatia anuragbhatia.com Linkedin <http://in.linkedin.com/in/anuragbhatia21> | Twitter <https://twitter.com/anurag_bhatia> Skype: anuragbhatia.com PGP Key Fingerprint: 3115 677D 2E94 B696 651B 870C C06D D524 245E 58E2

11 18

Open letter to Level3 concerning the global routing issues on June 12th
by Martin Millnert 14 Jun '15

14 Jun '15

Dear Level3, The Internet is a cooperative effort, and it works well only when its participants take constructive actions to address errors and remedy problems. Your position as a major Internet Carrier bestows upon you a certain degree of responsibility for the correct operation of the Internet all across (and beyond) the planet. You have many customers. Customers will always occasionally make mistakes. You as a major Internet Carrier have a responsibility to limit, not amplify, your customers' mistakes. Other major carriers implement technical measures that severely limits the damages from customer mistakes from having global impact. Other major carriers also implement operational procedures in addition to technical measures. In combination, these measures drastically reduce the outage-hours as a result of customer configuration errors. At 08:44 UTC on Friday 12th of June, one of your transit customers, Telekom Malaysia (AS4788) began announcing the full Internet table back to you, which you accepted and propagated to your peers and customers, causing global outages for close to 3 hours. [ https://twitter.com/DynResearch/status/609340592036970496 ] During this 3 hour window, it appears (from your own service outage reports) that you did nothing to stop the global Internet outage, but that Telekom Malaysia themselves eventually resolved it. This lack of action on your end, and your disregard for the correct operation of the global Internet is astonishing. These mistakes do not need to happen. AS4788 under normal circumstances announces ~1900 IPv4 prefixes to the Internet. You accepted multiple hundred thousand prefixes from them - a max prefix setting would have severely limited the damage. We expect that these are your practices as well, but they failed. When they do, it should not take ~3 hours to shut down the session(s). Many operators, in despair, turned down their peering sessions with you once it was clear you were causing the outages and no immediate fix was in sight. This improved the situation for some - but not all did. Had you deployed proper IRR-filtering to filter the bad announcements the impact would've been far less critical. As a direct consequence of your ~3 hours of inaction, as a local example, Swedish payment terminals were experiencing problems all over the country. The Swedish economy was directly affected by your inaction. There were queues when I was buying lunch! Imagine the food rage. The situation was probably similar at other places around the globe where people were awake. Operators around the planet are curious: - Did Level3 not detect or understand that it was causing global Internet outages for ~3 hours? - If Level3 did in fact detect or understand it was causing global Internet outages, why did it not properly and immediately remedy the situation? - What is Level3 going to do to address these questions and begin work on restoring its credibility as a carrier? We all understand that mistakes do happen (in applying customer interface templates, etc.). However the Internet is all too pervasive in everyday life today for anything but swift action by carriers to remedy breakage after the fact. It is absolutely not sufficient to let a customer spend 3 hours to detect and fix a situation like this one. It is unacceptable that no swift action was taken on your end to limit the global routing issues you caused. Sincerely, Martin Millnert Member of Internet Community - no carrier / ISP affiliation.

18 21

Re: NANOG Digest, Vol 89, Issue 15
by Abdulkareem H. Ali 14 Jun '15

14 Jun '15

Hi, >Message: 15 >Date: Sun, 14 Jun 2015 01:09:44 +0530 >From: Anurag Bhatia <me(a)anuragbhatia.com> >To: NANOG Mailing List <nanog(a)nanog.org> >Subject: Re: Question about EX - SRX redundancy >Message-ID: > <CAJ0+aXabO5GCbWgoCERjqcX8DWiK9q3Pj7bmQkFkY41P8TBSLA(a)mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > 3. In case of SRX only one device runs at a time and ports of other SRX > (slave) do not access traffic at all as long as it see the master is up via > heartbeat. This is true with regards to RETH interfaces, but any interface that is not part of a reth group on the slave SRX will function all the time. The SRX HA works with RE being up only on the master, while the PFE (which is what all interfaces use) is live on both systems all the time. In fact, you can make a reth group to have it's master interface on the slave SRX, depending on your redundancy group priorities. Also, I'd recommend assigning more than one interface for FAB links, providing your SRXes have the spares to use. Kareem. -- Abdulkareem H. Ali Network Operations Engineer CentralNic Group PLC London Stock Exchange Symbol: CNIC +44 20 3388 0600 www.CentralNic.com CentralNic Group PLC is a company registered in England and Wales with company number 8576358. Registered Offices: 35-39 Moorgate, London, EC2R 6AR.

1 0

Re: Hardware monitoring
by Dovid Bender 14 Jun '15

14 Jun '15

Anything on a linux box can have a bash script that snmpd can call. You can always wire up your own contraption and connect it to a Pi. ------Original Message------ From: Rafael Possamai Sender: NANOG To: nanog(a)nanog.org Subject: Hardware monitoring Sent: Jun 13, 2015 12:54 Hi everyone, I know this is slightly off-topic, but since it's still related to the list, I thought I'd give it a try. I am wondering what systems are out there (open source, preferably) for data collection and processing of hardware health data (temperature, CPU clock, fan speeds, etc). Ideally brand agnostic and location agnostic as well. I know of Cacti, but it would require SNMP enabled devices AFAIK, so room/generator/misc monitors wouldn't necessarily be included. Thanks in advance. Rafael Regards, Dovid

1 0

Enterprise network as an ISP with a single huge customer
by Stepan Kucherenko 13 Jun '15

13 Jun '15

Hello, I'm sure lots of you work for big enterprises, and some of you work for biggest of them. How many of you architect your network as an ISP, with that enterprise as the biggest customer ? Office networks in l3vpn, VPLS/EVPN on top of your own network for DCI, etc ? Or is it usually just a single IGP domain with no unnecessary bells and whistles ? Do you think one approach is better than the other ? If so, why ? I understand that it usually comes down to specific circumstances and most likely scale but I'd still love to hear about your experience.

8 12

DC Circuit denies stay on Neutrality
by Jay Ashworth 13 Jun '15

13 Jun '15

Here is a delightful wacky weekend starter culture for you: a backgrounder on exactly what it means that the DC Circuit denied Verizon et alia a stay of execution on Title II reclassification. Complete with bonus brony references. http://www.wetmachine.com/tales-of-the-sausage-factory/net-neutrality-litig… Cheers, --jra -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

3 2