Re: AS4788 Telecom Malaysia major route leak?
This is the official feedback: Level 3's network, alongside some other ISP's, experienced service disruptions affecting customers in Europe, Asia and multiple other markets. IP, Voice and Content Delivery Network (CDN) services were affected for Level 3. The root cause of the issue was isolated to a third party Internet Service Provider in Asia that leaked internet routes resulting in traffic being sent to a destination that could not route them, which affected IP, Voice and CDN services in multiple markets. The issue has been resolved, but the provider continues working to determine the specific root cause of the incident. At this time, customer services are restored with the exception of any that pose any possible risk to the Level 3 network. Maintaining a reliable, high-performing network for our customers is our top priority. Level 3 will continue to work with the provider to prevent a recurrence. Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at<mailto:jj@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
On 12/Jun/15 22:25, Jürgen Jaritsch wrote:
This is the official feedback:
Level 3's network, alongside some other ISP's, experienced service disruptions affecting customers in Europe, Asia and multiple other markets. IP, Voice and Content Delivery Network (CDN) services were affected for Level 3. The root cause of the issue was isolated to a third party Internet Service Provider in Asia that leaked internet routes resulting in traffic being sent to a destination that could not route them, which affected IP, Voice and CDN services in multiple markets. The issue has been resolved, but the provider continues working to determine the specific root cause of the incident. At this time, customer services are restored with the exception of any that pose any possible risk to the Level 3 network. Maintaining a reliable, high-performing network for our customers is our top priority. Level 3 will continue to work with the provider to prevent a recurrence.
While I agree that TM needs to look into its operational procedures, I think Level(3) needs to shoulder more of the blame, and not simply pass the buck to TM. TM has several more upstreams other than Level(3). Assuming their issue affected all their border routers, we did not see an issue via their other upstreams other than Level(3) - although this is conjecture on my part. Level(3) should have filtered at the time they were turning up TM. Simple as that. We all know we should never trust customers. So... Mark.
On 6/13/15 3:39 AM, Mark Tinka wrote:
On 12/Jun/15 22:25, Jürgen Jaritsch wrote:
This is the official feedback:
Level 3's network, alongside some other ISP's, experienced service disruptions affecting customers in Europe, Asia and multiple other markets. IP, Voice and Content Delivery Network (CDN) services were affected for Level 3. The root cause of the issue was isolated to a third party Internet Service Provider in Asia that leaked internet routes resulting in traffic being sent to a destination that could not route them, which affected IP, Voice and CDN services in multiple markets. The issue has been resolved, but the provider continues working to determine the specific root cause of the incident. At this time, customer services are restored with the exception of any that pose any possible risk to the Level 3 network. Maintaining a reliable, high-performing network for our customers is our top priority. Level 3 will continue to work with the provider to prevent a recurrence.
While I agree that TM needs to look into its operational procedures, I think Level(3) needs to shoulder more of the blame, and not simply pass the buck to TM.
if you localpref your customer up, you should probably not be willing to accept the whole internet from them.
TM has several more upstreams other than Level(3). Assuming their issue affected all their border routers, we did not see an issue via their other upstreams other than Level(3) - although this is conjecture on my part.
they also have ~ 180 ASNs in their downstream cone who presumably get a full table have the export policy that did the business in this case applied all the time.
Level(3) should have filtered at the time they were turning up TM. Simple as that.
We all know we should never trust customers. So...
Mark.
On Fri, Jun 12, 2015 at 08:25:40PM +0000, Jürgen Jaritsch wrote:
This is the official [level3] feedback:
[ ... ]
For completeness sake: here is what Telekom Malaysia published about the issue: Telekom Malaysia Berhad (TM) wishes to update on the service related issue detected yesterday, 12 June 2015 affecting a number of our Internet services customers that caused a deterioration in connection performance. We identified the root cause and our network team immediately took steps to optimise traffic flows, while we worked to restore connectivity to its expected level of performance. The services were restored at 6.30pm on the same day. We would like to clarify that during a network reconfiguration exercise, we had unintentionally updated traffic routing information which caused congestion and packet loss to our international connectivity. This had affected the internet traffic flow for some of our customers and some international traffic routes. We apologise for any inconvenience caused by the service disruption and would like to assure customers that we are undertaking all the necessary measures to ensure customers continue to experience uninterrupted services. Meanwhile, customers who have any enquiry or require further assistance can email us at help@tm.com.my or tweet to us via @tmconnects on Twitter. source: https://www.tm.com.my/OnlineHelp/Announcement/Pages/INTERNET-SERVICES-DISRUP... Kind regards, Job
Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is <tm> ... Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 20:27 heeft Job Snijders <job@instituut.net> het volgende geschreven:
On Fri, Jun 12, 2015 at 08:25:40PM +0000, Jürgen Jaritsch wrote: This is the official [level3] feedback:
[ ... ]
For completeness sake: here is what Telekom Malaysia published about the issue:
Telekom Malaysia Berhad (TM) wishes to update on the service related issue detected yesterday, 12 June 2015 affecting a number of our Internet services customers that caused a deterioration in connection performance.
We identified the root cause and our network team immediately took steps to optimise traffic flows, while we worked to restore connectivity to its expected level of performance. The services were restored at 6.30pm on the same day.
We would like to clarify that during a network reconfiguration exercise, we had unintentionally updated traffic routing information which caused congestion and packet loss to our international connectivity. This had affected the internet traffic flow for some of our customers and some international traffic routes.
We apologise for any inconvenience caused by the service disruption and would like to assure customers that we are undertaking all the necessary measures to ensure customers continue to experience uninterrupted services.
Meanwhile, customers who have any enquiry or require further assistance can email us at help@tm.com.my or tweet to us via @tmconnects on Twitter.
source: https://www.tm.com.my/OnlineHelp/Announcement/Pages/INTERNET-SERVICES-DISRUP...
Kind regards,
Job
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote:
Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu> het volgende geschreven:
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
Raymond, They provided a "simple sorry": "We apologise for any inconvenience caused by the service disruption." It doesn't get much more simple than that. -mel beckman
On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn <raymond@prolocation.net> wrote:
Hai!
Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes.
A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread?
I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements.
I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu> het volgende geschreven:
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
Hello Mel, Must just be me then. I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak. Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:27 heeft Mel Beckman <mel@beckman.org> het volgende geschreven:
Raymond,
They provided a "simple sorry":
"We apologise for any inconvenience caused by the service disruption."
It doesn't get much more simple than that.
-mel beckman
On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn <raymond@prolocation.net> wrote:
Hai!
Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes.
A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread?
I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements.
I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu> het volgende geschreven:
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
Raymond, But you said "A simple 'sorry' would have done." Now you're asking for lots more detail. Why the change? -mel beckman
On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn <raymond@prolocation.net> wrote:
Hello Mel,
Must just be me then.
I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:27 heeft Mel Beckman <mel@beckman.org> het volgende geschreven:
Raymond,
They provided a "simple sorry":
"We apologise for any inconvenience caused by the service disruption."
It doesn't get much more simple than that.
-mel beckman
On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn <raymond@prolocation.net> wrote:
Hai!
Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes.
A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread?
I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements.
I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu> het volgende geschreven:
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
Does anyone know if there's an official "ruling" as to who gets to pay for the SLA breaches? On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman <mel@beckman.org> wrote:
Raymond,
But you said "A simple 'sorry' would have done." Now you're asking for lots more detail. Why the change?
-mel beckman
On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn <raymond@prolocation.net> wrote:
Hello Mel,
Must just be me then.
I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:27 heeft Mel Beckman <mel@beckman.org> het volgende geschreven:
Raymond,
They provided a "simple sorry":
"We apologise for any inconvenience caused by the service disruption."
It doesn't get much more simple than that.
-mel beckman
On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn < raymond@prolocation.net> wrote:
Hai!
Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes.
A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread?
I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements.
I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu> het volgende geschreven:
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a "best effort" network, with zero guarantees. -mel beckman On Jun 14, 2015, at 4:06 PM, Rafael Possamai <rafael@gav.ufsc.br<mailto:rafael@gav.ufsc.br>> wrote: Does anyone know if there's an official "ruling" as to who gets to pay for the SLA breaches? On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Raymond, But you said "A simple 'sorry' would have done." Now you're asking for lots more detail. Why the change? -mel beckman
On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn <raymond@prolocation.net<mailto:raymond@prolocation.net>> wrote:
Hello Mel,
Must just be me then.
I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:27 heeft Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> het volgende geschreven:
Raymond,
They provided a "simple sorry":
"We apologise for any inconvenience caused by the service disruption."
It doesn't get much more simple than that.
-mel beckman
On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn <raymond@prolocation.net<mailto:raymond@prolocation.net>> wrote:
Hai!
Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes.
A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread?
I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements.
I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu<mailto:mark.tinka@seacom.mu>> het volgende geschreven:
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
On Sun, Jun 14, 2015 at 9:07 PM, Mel Beckman <mel@beckman.org> wrote:
SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a "best effort" network, with zero guarantees.
-mel beckman
Ok, I'll bite: my $dayjob is a Level 3 client that was directly affected by lack of availability due to recovery attempt Level 3 tried in our region. Where $dayjob can collect $ for this incident ? Rubens
keep in mind their target audience with that message is probably local malaysian customers, not the world. On Sun, Jun 14, 2015 at 5:09 PM Mel Beckman <mel@beckman.org> wrote:
SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a "best effort" network, with zero guarantees.
-mel beckman
On Jun 14, 2015, at 4:06 PM, Rafael Possamai <rafael@gav.ufsc.br<mailto: rafael@gav.ufsc.br>> wrote:
Does anyone know if there's an official "ruling" as to who gets to pay for the SLA breaches?
On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> wrote: Raymond,
But you said "A simple 'sorry' would have done." Now you're asking for lots more detail. Why the change?
-mel beckman
On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn <raymond@prolocation.net <mailto:raymond@prolocation.net>> wrote:
Hello Mel,
Must just be me then.
I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:27 heeft Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> het volgende geschreven:
Raymond,
They provided a "simple sorry":
"We apologise for any inconvenience caused by the service disruption."
It doesn't get much more simple than that.
-mel beckman
On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn < raymond@prolocation.net<mailto:raymond@prolocation.net>> wrote:
Hai!
Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes.
A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread?
I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements.
I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu <mailto:mark.tinka@seacom.mu>> het volgende geschreven:
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai!
Wouw! This is what they came up with?!
Hopefully Level3 will take appropriate measures. Its amazing. Really.
'Some internationally routes'
Have they any idea what they did at all?
Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
In addition to that, losing face in SE Asia is "not done". On Mon, Jun 15, 2015 at 12:14:43AM +0000, ryanL wrote:
keep in mind their target audience with that message is probably local malaysian customers, not the world.
On Sun, Jun 14, 2015 at 5:09 PM Mel Beckman <mel@beckman.org> wrote:
SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a "best effort" network, with zero guarantees.
-mel beckman
On Jun 14, 2015, at 4:06 PM, Rafael Possamai <rafael@gav.ufsc.br<mailto: rafael@gav.ufsc.br>> wrote:
Does anyone know if there's an official "ruling" as to who gets to pay for the SLA breaches?
On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> wrote: Raymond,
But you said "A simple 'sorry' would have done." Now you're asking for lots more detail. Why the change?
-mel beckman
I get that much, just wondering if Level3 would have to pay an SLA breach to its customers given the mess started with TM (even though it could have been avoided). And I am guessing if they do, they wouldn't be able to recover anything from TM. On Sun, Jun 14, 2015 at 7:07 PM, Mel Beckman <mel@beckman.org> wrote:
SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a "best effort" network, with zero guarantees.
-mel beckman
On Jun 14, 2015, at 4:06 PM, Rafael Possamai <rafael@gav.ufsc.br> wrote:
Does anyone know if there's an official "ruling" as to who gets to pay for the SLA breaches?
On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman <mel@beckman.org> wrote:
Raymond,
But you said "A simple 'sorry' would have done." Now you're asking for lots more detail. Why the change?
-mel beckman
On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn < raymond@prolocation.net> wrote:
Hello Mel,
Must just be me then.
I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:27 heeft Mel Beckman <mel@beckman.org> het volgende geschreven:
Raymond,
They provided a "simple sorry":
"We apologise for any inconvenience caused by the service disruption."
It doesn't get much more simple than that.
-mel beckman
On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn < raymond@prolocation.net> wrote:
Hai!
Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes.
A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread?
I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements.
I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally.
Thanks, Raymond Dijkxhoorn
Op 14 jun. 2015 om 23:04 heeft Mark Tinka <mark.tinka@seacom.mu> het volgende geschreven:
> On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: > Hai! > > Wouw! This is what they came up with?! > > Hopefully Level3 will take appropriate measures. Its amazing. Really. > > 'Some internationally routes' > > Have they any idea what they did at all? > > Its amazing that with parties like that the internet still works as is <tm> ...
I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed.
Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do.
But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern...
Mark.
what i have yet to understand (probably my fault) is how L(3) propagated the disease or, more correctly, what has happened over there that they did not stop the propagation? the crew that went there from mci ran a very tight ship and L(3) has always had pretty rigid filters. what happened? and i mean that in the sense of how can i not make a similar mistake? randy
On 15/Jun/15 03:01, Randy Bush wrote:
what i have yet to understand (probably my fault) is how L(3) propagated the disease or, more correctly, what has happened over there that they did not stop the propagation? the crew that went there from mci ran a very tight ship and L(3) has always had pretty rigid filters. what happened? and i mean that in the sense of how can i not make a similar mistake?
Given that TM were leaking into 3549, one may infer that Level(3)'s tight screws have not yet completely filtered down to the GBLX network of old. Conjecture on my part... It is no secret that Level(3)'s IRR client is broken, and as others have mentioned before, it's reasonably common to give them a call and get the spanner rammed over its head for the thing to work. Whether they are using the same for the GBLX network, or if the GBLX network is the nasty cousin we don't care about until he leaves the house, is an exercise left to all of us. Mark.
Absolutely on point. Let's solve the problem, not the blame. ERM Evan R Moore Network Engineer and Bitwrangler Sovernet Communications -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Randy Bush Sent: Sunday, June 14, 2015 9:02 PM To: North American Network Operators' Group Subject: [SPAM]Re: AS4788 Telecom Malaysia major route leak? what i have yet to understand (probably my fault) is how L(3) propagated the disease or, more correctly, what has happened over there that they did not stop the propagation? the crew that went there from mci ran a very tight ship and L(3) has always had pretty rigid filters. what happened? and i mean that in the sense of how can i not make a similar mistake? randy
Hi Rafael, I get that much, just wondering if Level3 would have to pay an SLA breach
to its customers given the mess started with TM (even though it could have been avoided). And I am guessing if they do, they wouldn't be able to recover anything from TM.
I doubt if L3 has to pay anything to its customers in terms of SLA breach, its best effort. Are you aware of any such agreement which suggest otherwise? that would be interesting.
Well, I was wondering the same. I am guessing it depends on the SLA contract since they are all very unique and specific. I assume they would have to, granted the issue lasted for a couple hours. Now, it depends on how they define the outage. A fiber cut that yields a customer's service unusable would be an easy SLA breach. Their legal team most likely removed any liability due to someone else's negligence, although you could argue they were negligent as well. So in this case they can claim the whole "best effort" thing and get away with it. I am not a L3 customer, so was just wondering out of curiosity. On Sun, Jun 14, 2015 at 8:07 PM, Aftab Siddiqui <aftab.siddiqui@gmail.com> wrote:
Hi Rafael,
I get that much, just wondering if Level3 would have to pay an SLA breach
to its customers given the mess started with TM (even though it could have been avoided). And I am guessing if they do, they wouldn't be able to recover anything from TM.
I doubt if L3 has to pay anything to its customers in terms of SLA breach, its best effort. Are you aware of any such agreement which suggest otherwise? that would be interesting.
On Sun, Jun 14, 2015 at 9:20 PM, Rafael Possamai <rafael@gav.ufsc.br> wrote:
Well, I was wondering the same. I am guessing it depends on the SLA contract since they are all very unique and specific.
I'm going to bet that aside from a few one-off cases the SLA in question talks about maintaining reachability inside L3's network, or maybe even 'is your link up and can you ping the L3 gateway router you connect to?' SLA's aren't meant to actually get paid out...
participants (14)
-
Aftab Siddiqui -
B -
Christopher Morrow -
Evan Moore -
Job Snijders -
joel jaeggli -
Jürgen Jaritsch -
Mark Tinka -
Mel Beckman -
Rafael Possamai -
Randy Bush -
Raymond Dijkxhoorn -
Rubens Kuhl -
ryanL