AS4788 Telecom Malaysia major route leak?
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 3333 I, validation-state: valid > to 64.210.69.85 via xe-1/1/0.0 Tore
On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson wrote:
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them.
E.g. for the RIPE NCC (193.0.0.0/21):
[BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 3333 I, validation-state: valid > to 64.210.69.85 via xe-1/1/0.0
It appears that AS3549 propagated the (almost?) full routing table leak to its peers, where in lots of instances max prefix kicked in. This has global impact, lots of alerts on the SQA collector page http://sqa.ring.nlnog.net/ Kind regards, Job
On 12 Jun 2015, at 16:16, Job Snijders wrote:
This has global impact, lots of alerts on the SQA collector page http://sqa.ring.nlnog.net/
I'm reaching out to them now. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
It *looks* like GBLX stopped accepting the leak. Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 20 3514 6970 UK (Office) +44 7584 906 055 UK (Mobile) +1 888 993 5273 US (Office) smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335
On 12 Jun 2015, at 10:27, Roland Dobbins <rdobbins@arbor.net> wrote:
On 12 Jun 2015, at 16:16, Job Snijders wrote:
This has global impact, lots of alerts on the SQA collector page http://sqa.ring.nlnog.net/
I'm reaching out to them now.
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
* Marty Strong via NANOG <nanog@nanog.org>
It *looks* like GBLX stopped accepting the leak.
If so, it's a partial fix at best, I still see plenty of leaked routes, both via 3356 and 3549, e.g.: tore@cr1-osl3> show route 195.24.168.98 all Jun 12 12:03:54 +0200 inet.0: 544405 destinations, 1591203 routes (543086 active, 3 holddown, 526626 hidden) + = Active Route, - = Last Active, * = Both 195.24.160.0/19 *[BGP/170] 00:03:59, MED 2000, localpref 50, from 87.238.63.5 AS path: 3356 3549 4788 6939 39648 I, validation-state: unverified > to 87.238.63.56 via ae0.0 [BGP/170] 00:05:24, MED 0, localpref 50, from 87.238.63.2 AS path: 3356 3549 4788 6939 39648 I, validation-state: unverified > to 87.238.63.56 via ae0.0 [BGP ] 01:16:00, MED 25245, localpref 100 AS path: 3549 4788 6939 39648 I, validation-state: unverified > to 64.210.69.85 via xe-1/1/0.0 It seems to have started around 08:47 UTC, that's when I got my first alarm from ring-sqa at least. Tore
On 06/12/2015 10:43 AM, Marty Strong via NANOG wrote:
It *looks* like GBLX stopped accepting the leak
I think you just saw it flapping. :-) That's what I've been seeing since ~ 0845 UTC :-( -- rrbone UG (haftungsbeschraenkt) - Leibnizstr. 8a - 44147 Dortmund HR B 23168 Amtsgericht Dortmund - Geschaeftsfuehrer: Dominik Bay
Yes, you’re right, I was too trigger happy :( Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 20 3514 6970 UK (Office) +44 7584 906 055 UK (Mobile) +1 888 993 5273 US (Office) smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335
On 12 Jun 2015, at 11:18, Job Snijders <job@instituut.net> wrote:
On Fri, Jun 12, 2015 at 10:43:09AM +0100, Marty Strong via NANOG wrote:
It *looks* like GBLX stopped accepting the leak.
I disagree. Since 08:44 UTC up until now (10:15) the DFZ has been a radio-active wasteland with hordes of unwelcome announcements.
Kind regards,
Job
On Fri, Jun 12, 2015 at 12:18:38PM +0200, Job Snijders wrote:
On Fri, Jun 12, 2015 at 10:43:09AM +0100, Marty Strong via NANOG wrote:
It *looks* like GBLX stopped accepting the leak.
I disagree. Since 08:44 UTC up until now (10:15) the DFZ has been a radio-active wasteland with hordes of unwelcome announcements.
OK, as of now (~ 10:40) UTC things look normalised. Kind regards, Job
* Roland Dobbins <rdobbins@arbor.net> [2015-06-12 12:57]:
On 12 Jun 2015, at 17:46, Job Snijders wrote:
OK, as of now (~ 10:40) UTC things look normalised.
Just got off the phone, I think things may be in hand, now.
Still seeing a lot more updates than usual: http://www.karotte.org/pics/bgp-stability-2.png Is this just folks turning up their sessions again? Looks a bit much... Regards Sebastian -- GPG Key: 0x93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
On Fri, Jun 12, 2015 at 01:21:14PM +0200, Sebastian Wiesinger wrote:
* Roland Dobbins <rdobbins@arbor.net> [2015-06-12 12:57]:
On 12 Jun 2015, at 17:46, Job Snijders wrote:
OK, as of now (~ 10:40) UTC things look normalised.
Just got off the phone, I think things may be in hand, now.
Still seeing a lot more updates than usual:
http://www.karotte.org/pics/bgp-stability-2.png
Is this just folks turning up their sessions again? Looks a bit much...
Yes, I suspect tons of 3356 / 3549 customers shut down their BGP sessions waiting for the storm to blow over. I expect more churn then usual the next 6 ~ 12 hours, due to customers slowly turning session back on. Kind regards, Job
First news: http://www.xgn.nl/nieuws/69593/grote-internetstoring-in-europa-problemen-doo... -- Alessandro Martins +55 11 94715-4700 On Fri, Jun 12, 2015 at 8:27 AM, Job Snijders <job@instituut.net> wrote:
On Fri, Jun 12, 2015 at 01:21:14PM +0200, Sebastian Wiesinger wrote:
* Roland Dobbins <rdobbins@arbor.net> [2015-06-12 12:57]:
On 12 Jun 2015, at 17:46, Job Snijders wrote:
OK, as of now (~ 10:40) UTC things look normalised.
Just got off the phone, I think things may be in hand, now.
Still seeing a lot more updates than usual:
http://www.karotte.org/pics/bgp-stability-2.png
Is this just folks turning up their sessions again? Looks a bit much...
Yes, I suspect tons of 3356 / 3549 customers shut down their BGP sessions waiting for the storm to blow over. I expect more churn then usual the next 6 ~ 12 hours, due to customers slowly turning session back on.
Kind regards,
Job
* Job Snijders <job@instituut.net> [2015-06-12 13:30]:
Yes, I suspect tons of 3356 / 3549 customers shut down their BGP sessions waiting for the storm to blow over. I expect more churn then usual the next 6 ~ 12 hours, due to customers slowly turning session back on.
Yes. It's nice and stable now. http://www.karotte.org/pics/bgp-stability-3.png So after this interesting morning let's hope for a boring weekend. :) Let's wait and see what explanation will be given for this hiccup. Regards Sebastian -- GPG Key: 0x93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
On Fri, 2015-06-12 at 10:43 +0100, Marty Strong via NANOG wrote:
It *looks* like GBLX stopped accepting the leak.
Nope. Churn is ongoing, nothing has been fixed. Global outage began 08:44 UTC and is still ongoing. It's been so long people have now had time to come up with things like "33.333%". Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :) /M
Still on hold with Level3, but some of my sites are clearing up. Chris -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Martin Millnert Sent: Friday, June 12, 2015 3:24 AM To: Marty Strong Cc: NANOG Subject: Re: AS4788 Telecom Malaysia major route leak? On Fri, 2015-06-12 at 10:43 +0100, Marty Strong via NANOG wrote:
It *looks* like GBLX stopped accepting the leak.
Nope. Churn is ongoing, nothing has been fixed. Global outage began 08:44 UTC and is still ongoing. It's been so long people have now had time to come up with things like "33.333%". Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :) /M
* millnert@gmail.com (Martin Millnert) [Fri 12 Jun 2015, 12:54 CEST]:
Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :) https://scontent-sea1-1.xx.fbcdn.net/hphotos-xat1/t31.0-8/10914977_101528099...
Is that tweet for real? How is that company (not TM) still in business? -- Niels.
Looks to be edited from their original tweet. On Fri, Jun 12, 2015 at 9:07 AM, <niels=nanog@bakker.net> wrote:
* millnert@gmail.com (Martin Millnert) [Fri 12 Jun 2015, 12:54 CEST]:
Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :)
https://scontent-sea1-1.xx.fbcdn.net/hphotos-xat1/t31.0-8/10914977_101528099...
Is that tweet for real? How is that company (not TM) still in business?
-- Niels.
* Tore Anderson <tore@fud.no> [2015-06-12 11:12]:
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them.
E.g. for the RIPE NCC (193.0.0.0/21):
[BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 3333 I, validation-state: valid > to 64.210.69.85 via xe-1/1/0.0
I confirm, something is going on: http://www.karotte.org/pics/bgp-stability.png Regards Sebastian -- GPG Key: 0x93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson <tore@fud.no> wrote a message of 10 lines which said:
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them.
E.g. for the RIPE NCC (193.0.0.0/21):
[BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 3333 I, validation-state: valid
Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
These aren't just leaks - they're more specifics of what's normally advertised, but keeping the proper origin. Hard to see how that could be accidental... Chris On Fri, 12 Jun 2015, Stephane Bortzmeyer wrote:
On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson <tore@fud.no> wrote a message of 10 lines which said:
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them.
E.g. for the RIPE NCC (193.0.0.0/21):
[BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 3333 I, validation-state: valid
Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
These aren't just leaks - they're more specifics of what's normally advertised, but keeping the proper origin. Hard to see how that could be accidental...
Having looked further - the examples of these I was looking at (advertisements from AS34556 & AS17709) were being advertised before the leak, but only with limited visibility. The leak caused them to be (intermittently) globally visible. Tin foil hat off - can all just be accidental. Chris
On Fri, 12 Jun 2015, Stephane Bortzmeyer wrote:
On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson <tore@fud.no> wrote a message of 10 lines which said:
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them.
E.g. for the RIPE NCC (193.0.0.0/21):
[BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 3333 I, validation-state: valid
Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
Hi Marty,
Noted. We are still checking this issue.
Regards,
LEE BON SHENG | leebonsheng@tm.com.my | ipmc_ipcore@tm.com.my NOC2 IPCORE, ISP Network Management, Telekom Malaysia, AS4788 TOLLFREE: 1-800-88-2646 (Opt 4) / International: +603-22466646 (Opt 4)
We're committed to perform. We strive to excel. We deliver THE BEST!
Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty@cloudflare.com +44 20 3514 6970 UK (Office) +44 7584 906 055 UK (Mobile) +1 888 993 5273 US (Office) smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335
On 12 Jun 2015, at 10:41, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson <tore@fud.no> wrote a message of 10 lines which said:
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them.
E.g. for the RIPE NCC (193.0.0.0/21):
[BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 3333 I, validation-state: valid
Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
participants (13)
-
Alessandro Martins -
Chris Burton -
Chris Wilson -
Dominik Bay -
Job Snijders -
Martin Millnert -
Marty Strong -
niels=nanog＠bakker.net -
Roland Dobbins -
Sebastian Wiesinger -
Stephane Bortzmeyer -
Tom Paseka -
Tore Anderson