Dear NANOGers, is there anyone from Verizon and Level3 who can help me with DNS caching issue? We're running a global service for a customer and we had to change to NS IPs via Glue Records. At the moment at least Verizone and Level3 are caching old NS records. Looking for DNS admins out there. Please contact me off- or on-list! Thanks & best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com<mailto:JJaritsch@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
On 06/01/2016 10:59 AM, Jürgen Jaritsch wrote:
Dear NANOGers,
is there anyone from Verizon and Level3 who can help me with DNS caching issue? We're running a global service for a customer and we had to change to NS IPs via Glue Records. At the moment at least Verizone and Level3 are caching old NS records. Looking for DNS admins out there.
Please contact me off- or on-list!
I totally understand the desire to just be able to go ask major operators for a courtesy cache flush, but there are ways to update dns and procedures to engage that can eliminate the underlaying causes of same. Not that everyone, including myself, is prefect or godly (or has their name in the rfc...!), but at the same time, it's a learning experience being offered to you and I hope that whatever hole you shot in your foot heals soon and hopefull you never have to make another one like it. Mike-
Hi Mike, thanks for your (not so useful :)) answer ... I'm aware of things like TTL etc ... but the situation is that customer is receiving ~130gbit of DNS reflection attack to their original DNS and that's the reason why we had to move over to a new NS set. I'm not allowed to tell you the customers and/or project name but I guess many of you know them ... if you're reading Twitter or reddit you've probably recognized which global service is broken at the moment ... Best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Gesendet: Mittwoch, 01. Juni 2016 20:17 An: nanog@nanog.org Betreff: Re: Verizon and Level3 DNS flush On 06/01/2016 10:59 AM, Jürgen Jaritsch wrote:
Dear NANOGers,
is there anyone from Verizon and Level3 who can help me with DNS caching issue? We're running a global service for a customer and we had to change to NS IPs via Glue Records. At the moment at least Verizone and Level3 are caching old NS records. Looking for DNS admins out there.
Please contact me off- or on-list!
I totally understand the desire to just be able to go ask major operators for a courtesy cache flush, but there are ways to update dns and procedures to engage that can eliminate the underlaying causes of same. Not that everyone, including myself, is prefect or godly (or has their name in the rfc...!), but at the same time, it's a learning experience being offered to you and I hope that whatever hole you shot in your foot heals soon and hopefull you never have to make another one like it. Mike-
On Jun 2, 2016, at 1:24 AM, Jürgen Jaritsch <JJaritsch@anexia-it.com> wrote:
and that's the reason why we had to move over to a new NS set.
Which the attackers (or their attack tools) will immediately discern, & shift their targeting accordingly. Playing games like this with addressing seldom, if ever, accomplishes anything useful in terms of successfully defending against DDoS attacks. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Hi Roland, the difference between old and new DNS are way more capacity and extra DDoS protection ... it IS expected behavior that traffic will switch over to the new DNS. best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Roland Dobbins Gesendet: Donnerstag, 02. Juni 2016 10:38 An: nanog@nanog.org Betreff: Re: AW: Verizon and Level3 DNS flush On Jun 2, 2016, at 1:24 AM, Jürgen Jaritsch <JJaritsch@anexia-it.com> wrote:
and that's the reason why we had to move over to a new NS set.
Which the attackers (or their attack tools) will immediately discern, & shift their targeting accordingly. Playing games like this with addressing seldom, if ever, accomplishes anything useful in terms of successfully defending against DDoS attacks. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Jun 2, 2016, at 3:42 PM, Jürgen Jaritsch <JJaritsch@anexia-it.com> wrote:
it IS expected behavior that traffic will switch over to the new DNS.
Altering routing and/or adding capacity/capabilities to the existing infrastructure is generally better, whenever possible, due to the cache-flushing challenges you're now experiencing. Sometimes it isn't possible, of course. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Altering routing and/or adding capacity/capabilities to the existing infrastructure is generally better
Yes ... but as mentioned in one of the off-list replies: the original DNS are from a 3rd party and they had no chance to expand resources ... best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Roland Dobbins Gesendet: Donnerstag, 02. Juni 2016 11:30 An: nanog@nanog.org Betreff: Re: AW: AW: Verizon and Level3 DNS flush On Jun 2, 2016, at 3:42 PM, Jürgen Jaritsch <JJaritsch@anexia-it.com> wrote:
it IS expected behavior that traffic will switch over to the new DNS.
Altering routing and/or adding capacity/capabilities to the existing infrastructure is generally better, whenever possible, due to the cache-flushing challenges you're now experiencing. Sometimes it isn't possible, of course. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On 01/06/2016 21:16, Mike wrote:
On 06/01/2016 10:59 AM, Jürgen Jaritsch wrote:
Dear NANOGers,
is there anyone from Verizon and Level3 who can help me with DNS caching issue? We're running a global service for a customer and we had to change to NS IPs via Glue Records. At the moment at least Verizone and Level3 are caching old NS records. Looking for DNS admins out there.
Please contact me off- or on-list!
I totally understand the desire to just be able to go ask major operators for a courtesy cache flush, but there are ways to update dns and procedures to engage that can eliminate the underlaying causes of same. Not that everyone, including myself, is prefect or godly (or has their name in the rfc...!), but at the same time, it's a learning experience being offered to you and I hope that whatever hole you shot in your foot heals soon and hopefull you never have to make another one like it.
Mike-
Those "procedures" were attempted to be documented in an RFC: https://tools.ietf.org/html/draft-jabley-dnsop-flush-reqs-00 https://tools.ietf.org/html/draft-jabley-dnsop-dns-flush-00 Unfortunately, nothing ever came of it, so people are forced to post to NANOG pleading for help. -Hank
participants (4)
-
Hank Nussbacher -
Jürgen Jaritsch -
Mike -
Roland Dobbins