No DDoS or Anonymous attack appears to have been involved. On Tue, Sep 11, 2012 at 10:54 AM, Kyle Creyts <kyle.creyts@gmail.com> wrote:

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

On Mon, Sep 10, 2012 at 1:27 PM, Operations Dallas <operations.tcdallas@hotmail.com> wrote:

...

I thought I saw an article on routergod.com from Dance Patrick regarding anycast DNS...... ~oliver

Sent via DynaTAC. Please forgive spelling and grammar.

-----Original Message----- From: Bill.Ingrum@t-systems.com Date: Mon, 10 Sep 2012 19:13:27 To: <aaron@heyaaron.com>; <nanog@nanog.org> Subject: RE: Heads-Up: GoDaddy Broke the Interwebs...

Looks like this may be a DDoS attack from Anonymous:

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites...

-----Original Message----- From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com] Sent: Monday, September 10, 2012 1:07 PM To: NANOG mailing list Subject: Heads-Up: GoDaddy Broke the Interwebs...

For the last ~15 minutes I've been receiving complaints about DNS issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of tweets about their hosting and VPS being down. I'm unable to access the control panel for one of my customer accounts.

-A

-- Kyle Creyts

Information Assurance Professional BSidesDetroit Organizer

-- Kyle Creyts Information Assurance Professional BSidesDetroit Organizer

And this is bad why? Sent from my iPhone On 2012-09-11, at 1:14 PM, "Jason Bertoch" <Jason_Bertoch@nwrdc.fsu.edu> wrote:

Now it's CNN

/Jason

-----Original Message----- From: Kyle Creyts [mailto:kyle.creyts@gmail.com] Sent: Tuesday, September 11, 2012 1:55 PM To: Operations Dallas Cc: nanog@nanog.org Subject: Re: Heads-Up: GoDaddy Broke the Interwebs...

No DDoS or Anonymous attack appears to have been involved.

On Tue, Sep 11, 2012 at 10:54 AM, Kyle Creyts <kyle.creyts@gmail.com> wrote:

...

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

On Mon, Sep 10, 2012 at 1:27 PM, Operations Dallas <operations.tcdallas@hotmail.com> wrote:

...

I thought I saw an article on routergod.com from Dance Patrick regarding anycast DNS...... ~oliver

Sent via DynaTAC. Please forgive spelling and grammar.

-----Original Message----- From: Bill.Ingrum@t-systems.com Date: Mon, 10 Sep 2012 19:13:27 To: <aaron@heyaaron.com>; <nanog@nanog.org> Subject: RE: Heads-Up: GoDaddy Broke the Interwebs...

Looks like this may be a DDoS attack from Anonymous:

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-o f-sites/

-----Original Message----- From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com] Sent: Monday, September 10, 2012 1:07 PM To: NANOG mailing list Subject: Heads-Up: GoDaddy Broke the Interwebs...

For the last ~15 minutes I've been receiving complaints about DNS issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of tweets about their hosting and VPS being down. I'm unable to access the control panel for one of my customer accounts.

-A

-- Kyle Creyts

Information Assurance Professional BSidesDetroit Organizer

-- Kyle Creyts

Information Assurance Professional BSidesDetroit Organizer

On Tue, Sep 11, 2012 at 03:15:55PM -0400, William Herrin wrote:

On Tue, Sep 11, 2012 at 1:54 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:

...

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

"many of our customers experienced intermittent service outages"

Must be that new definition of the word "intermittent." The one roughly synonymous with "total."

Yeah. Doubleplusungood. -- Mike Andrews, W5EGO mikea@mikea.ath.cx Tired old sysadmin

What time did the anon first say there would be GoDaddy problems? Earliest timestamp I can find is 10:45am (pacific) and the problems had started much earlier, around 10:15. Curious if you found evidence of any anons claiming an attack or responsibility for the attack within, say, 5 minutes of it starting. Also, the time it stopped wasn't exactly tied to anything the anon said, other than his vague statements like "it can last one hour or one month" and "soon u guys can acess". And he said that latter statement at 1:59pm while the outage ended at 3:45pm. Summary: 30 minutes late on the start time, and off by well over an hour on the stop time. Damian On Tue, Sep 11, 2012 at 12:25 PM, Blake Pfankuch <blake@pfankuch.me> wrote:

As someone else nicely pointed out "network problems starting when the anon post said they would, and ending when they said they would stop.... ironic?"

-----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Tuesday, September 11, 2012 1:16 PM To: Kyle Creyts Cc: nanog@nanog.org Subject: Re: Heads-Up: GoDaddy Broke the Interwebs...

On Tue, Sep 11, 2012 at 1:54 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:

...

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

"many of our customers experienced intermittent service outages"

Must be that new definition of the word "intermittent." The one roughly synonymous with "total."

-Bill

-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

-- Damian Menscher :: Security Reliability Engineer :: Google

On Sep 11, 2012, at 16:04 , Christopher Morrow <morrowc.lists@gmail.com> wrote:

On Tue, Sep 11, 2012 at 3:47 PM, Damian Menscher <damian@google.com> wrote:

...

Summary: 30 minutes late on the start time, and off by well over an hour on the stop time.

even a broken clock is right 2x/day? nostrodamus was eventually right a few times? 'If you're cold, shoot until you get hot, then keep shooting!' - dick vitale

folk like to look for the most complicated/spooky/crazy reason... most often it's just a simple reason for failure :( so far godaddy seems to agree with the 'it was a simple mistake on our part' (paraphrased, they probably won't say 'simple')

No large flows reported to the affected NSes, tweets were suspicious at best, other anon-ops denied the attack was them, and GoDaddy admitted internal error. I'm going to take GoDaddy at their word, and give them major kudos for owning up to the mistake - in public. -- TTFN, patrick

The blog says 99.999% uptime, but I'm guessing this "outage" lasted more them 5.49300000002 minutes and they probably had other issues during the year. On 11 September 2012 21:53, Rubens Kuhl <rubensk@gmail.com> wrote:

...

No large flows reported to the affected NSes, tweets were suspicious at best, other anon-ops denied the attack was them, and GoDaddy admitted internal error.

I'm going to take GoDaddy at their word, and give them major kudos for owning up to the mistake - in public.

That doesn't mean that their description of the internal error fits what happened. Not to say that there were an attack, just that there can be more internal failures, including processes, to be accounted for. Whether they will publish a root-cause analysis/swiss chesse model/<insert your preferred methodology> or not is up to them, but to tech-savvy stakeholders I think they are still in debt.

Rubens

-- ฤ๊๊๊๊๊็็็็็๊๊๊๊๊็็็็ ฮ้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้้ ฦ้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้ BaconZombie LOAD "*",8,1

On Sep 11, 2012, at 17:04 , ryanL <ryan.landry@gmail.com> wrote:

when patrick is referring to "taking their word for it", he's referring to a post on outages@ by godaddy's network engineering manager that stated "bgp, and more details to follow".

Well, mostly I'm taking GoDaddy at their word that this was not a DoS attack. I also believe it was related to BGP, and am happy to get more info. But we are discussing Anonymous vs. Self-inflicted wound here. -- TTFN, patrick

i tend to align with patrick's thought. i'm also interested to see the details, which they are really under no obligation to provide.

On Tue, Sep 11, 2012 at 1:53 PM, Rubens Kuhl <rubensk@gmail.com> wrote:

...

No large flows reported to the affected NSes, tweets were suspicious at best, other anon-ops denied the attack was them, and GoDaddy admitted internal error.

I'm going to take GoDaddy at their word, and give them major kudos for owning up to the mistake - in public.

That doesn't mean that their description of the internal error fits what happened. Not to say that there were an attack, just that there can be more internal failures, including processes, to be accounted for. Whether they will publish a root-cause analysis/swiss chesse model/<insert your preferred methodology> or not is up to them, but to tech-savvy stakeholders I think they are still in debt.

Rubens

On Tue, Sep 11, 2012 at 11:16 PM, Naveen Nathan <naveen@lastninja.net> wrote:

...

Well, mostly I'm taking GoDaddy at their word that this was not a DoS attack.

I also believe it was related to BGP, and am happy to get more info. But we are discussing Anonymous vs. Self-inflicted wound here.

I'm skeptical, BGPlay (http://bgplay.routeviews.org/) doesn't show any withdrawn routes for any of their prefixes over Sep 9-11. Infact, their BGP operation looks fairly operational during the time from what I can gather.

a bgp error doesn't HAVE to mean that they withdrew (or even re-announced!) anything to the outside world, does it? for instance: border-router -> internet redistribute your aggregate networks from statics to Null0 on the border-router accept full routes so you can send them to the other borders and make good decisions at the external edge border-router -> internal send default or some version of default via a fitler to internal datacenter routers/aggregation/distribution devices. accept from them (maybe) local subnets that are part of your aggregates now, accidently remove the filter content for the sessions between the border and internal ... oops, your internal devices bounce with 'corrupted tables' (blown tables)... you still send your aggs steadily to the interwebs, wee! -chris

(Arrive at the intended destination, that is) On Tue, Sep 11, 2012 at 9:18 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:

+1

Announcing a prefix doesn't mean that the traffic to those IPs found within shall ever arrive.

On Tue, Sep 11, 2012 at 8:43 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

...

On Tue, Sep 11, 2012 at 11:16 PM, Naveen Nathan <naveen@lastninja.net> wrote:

...

...

Well, mostly I'm taking GoDaddy at their word that this was not a DoS attack.

I also believe it was related to BGP, and am happy to get more info. But we are discussing Anonymous vs. Self-inflicted wound here.

I'm skeptical, BGPlay (http://bgplay.routeviews.org/) doesn't show any withdrawn routes for any of their prefixes over Sep 9-11. Infact, their BGP operation looks fairly operational during the time from what I can gather.

a bgp error doesn't HAVE to mean that they withdrew (or even re-announced!) anything to the outside world, does it?

for instance: border-router -> internet redistribute your aggregate networks from statics to Null0 on the border-router accept full routes so you can send them to the other borders and make good decisions at the external edge

border-router -> internal send default or some version of default via a fitler to internal datacenter routers/aggregation/distribution devices. accept from them (maybe) local subnets that are part of your aggregates

now, accidently remove the filter content for the sessions between the border and internal ... oops, your internal devices bounce with 'corrupted tables' (blown tables)... you still send your aggs steadily to the interwebs, wee!

-chris

-- Kyle Creyts

Information Assurance Professional BSidesDetroit Organizer

-- Kyle Creyts Information Assurance Professional BSidesDetroit Organizer

On Sep 11, 2012, at 4:53 PM, Rubens Kuhl <rubensk@gmail.com> wrote:

That doesn't mean that their description of the internal error fits what happened

Anytime I've seen a real RFO, it takes more than 24 hours to collect data. Sometimes you actually don't know what happened. There's a reason for this comic: http://www.dilbert.com/strips/comic/1999-08-04/ (the reboot cleared the problem). I've seen many odd behaviors of devices that nobody could explain, including the vendors.. sometimes it takes a few years to understand what happened. I recall a case where 2-3 years after a major outage someone made some minor comment about their architecture and a light came on. I welcome more information about mistakes/errors that we can all learn from. Sharing that information can be hard or uncomfortable at times, but can help others learn and not make the same mistakes again. I took the recommendation of others and have started to read "Normal Accidents". amazon link: http://tinyurl.com/9dc6x98 The whole multiple-failures problem really makes me concerned about cascading system failures when things go wrong. - Jared

we do not know what happened. we have an apology, not an explanation or reasonable post mortem. all else is conjecturbation.

Agreed. And as Chris and Kyle pointed out, there is no indication that the problems were present in the BGP DFT, and the issues could've occured over iBGP. I completely concur with this, and do not preclude it as an explanation. But I would just like to put this out there. In the past, GoDaddy has clashed with the Internet due to their initial stance on SOPA, which resulted in a noticeable loss of customers and generated a significant amount of bad press. Now, there's a lot of conjecture as to what caused their outage. But the most harm to GoDaddy would be reporting that they had a security breach or DoS/DDoS attack which would instill fear in their customer base. The major media outlets had already picked this up and started to report foul play by Anonymous, denial of service attacks, or whatever. To save face, it would make the most sense not to mention that a security breach or DoS/DDoS attack occured. Indicating a security breach would be immediate concern for any customer. If it was a DoS/DDoS attack, they're basically admitting that they don't have an infrastructure capable of withstanding or mitigating such attacks (which competitors such as Cloudfare do claim). So the best option would be to spread disinformation if either occured, and offer /generous/ service credit to earn back customer goodwill and confidence. This is simply why I remain skeptical. And as I said earlier, it would be nice to receive more information of what actually happened, if GoDaddy, or anyone in the know with GoDaddy, would oblige. - Naveen