Yes tis down. Watch productivity go up ------Original Message------ From: Ernie Rubi To: nanog@nanog.org Subject: Facebook Issues/Outage in Southeast? Sent: Sep 23, 2010 12:39 Anyone else having trouble? We're colo'ed at the NOTA in Miami and directly peer with them - even though our session hasn't gone down we still can't reach them. Ernesto M. Rubi Sr. Network Engineer AMPATH/CIARA Florida International Univ, Miami Reply-to: ernesto@cs.fiu.edu Cell: 786-282-6783
Productivity grinds to a hault as everyone goes onto twitter to talk about facebook being down....
-----Original Message----- From: chaim rieger [mailto:chaim.rieger@gmail.com] Sent: Thursday, September 23, 2010 12:41 PM To: Ernie Rubi; nanog@nanog.org Subject: Re: Facebook Issues/Outage in Southeast?
Yes tis down. Watch productivity go up ------Original Message------ From: Ernie Rubi To: nanog@nanog.org Subject: Facebook Issues/Outage in Southeast? Sent: Sep 23, 2010 12:39
Anyone else having trouble? We're colo'ed at the NOTA in Miami and directly peer with them - even though our session hasn't gone down we still can't reach them.
Ernesto M. Rubi Sr. Network Engineer AMPATH/CIARA Florida International Univ, Miami Reply-to: ernesto@cs.fiu.edu Cell: 786-282-6783
On Thu, 2010-09-23 at 12:47 -0700, Justin Horstman wrote:
Productivity grinds to a halt as everyone goes onto twitter to talk about facebook being down....
I'm hoping (desperately) that someone other than me sees the full irony in this statement? I also have visions of hundreds of techs worldwide thinking their snort boxes have hung. Gord -- oink
I can ping them but not access them over Cogent here in Tysons Corner, Virginia - looks like a server issue to me. server:temp tme$ wget www.facebook.com --15:52:17-- http://www.facebook.com/ => `index.html' Resolving www.facebook.com... 216.66.8.178, 216.66.8.225 Connecting to www.facebook.com|216.66.8.178|:80... connected. HTTP request sent, awaiting response... 500 Internal Server Error 15:53:03 ERROR 500: Internal Server Error. server:temp tme$ ping -c 5 216.66.8.178 PING 216.66.8.178 (216.66.8.178): 56 data bytes 64 bytes from 216.66.8.178: icmp_seq=0 ttl=55 time=15.865 ms 64 bytes from 216.66.8.178: icmp_seq=1 ttl=55 time=15.693 ms 64 bytes from 216.66.8.178: icmp_seq=2 ttl=55 time=15.851 ms 64 bytes from 216.66.8.178: icmp_seq=3 ttl=55 time=15.953 ms 64 bytes from 216.66.8.178: icmp_seq=4 ttl=55 time=15.823 ms --- 216.66.8.178 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max/stddev = 15.693/15.837/15.953/0.084 ms server:temp tme$ Regards Marshall On Sep 23, 2010, at 3:41 PM, chaim rieger wrote:
Yes tis down. Watch productivity go up ------Original Message------ From: Ernie Rubi To: nanog@nanog.org Subject: Facebook Issues/Outage in Southeast? Sent: Sep 23, 2010 12:39
Anyone else having trouble? We're colo'ed at the NOTA in Miami and directly peer with them - even though our session hasn't gone down we still can't reach them.
Ernesto M. Rubi Sr. Network Engineer AMPATH/CIARA Florida International Univ, Miami Reply-to: ernesto@cs.fiu.edu Cell: 786-282-6783
It may be some chunk of akamai being down/unreachable, no noticeable issues from TWC in SATX. J
On Thu, Sep 23, 2010 at 3:03 PM, Jorge Amodio <jmamodio@gmail.com> wrote:
It may be some chunk of akamai being down/unreachable, no noticeable issues from TWC in SATX.
Change that, it is down from here too now ... packets don't get out of tbone.rr.com from here. J
Looks that it came back up now ... root@tango:/var/cache/bind# ping -c 10 www.facebook.com PING a1254.g.akamai.net (209.18.43.169) 56(84) bytes of data. 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=1 ttl=57 time=15.8 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=2 ttl=57 time=16.9 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=3 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=4 ttl=57 time=16.6 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=5 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=6 ttl=57 time=16.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=7 ttl=57 time=17.0 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=8 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=9 ttl=57 time=16.7 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=10 ttl=57 time=16.7 ms J
We've always been able to ping facebook, it seems the people with the most trouble are hitting akamai's caching. The 209.165.150.24 is local to us, as we host akamai for facebook in Alaska. ping www.facebook.com Pinging a1254.g.akamai.net [209.165.150.24] with 32 bytes of data: Reply from 209.165.150.24: bytes=32 time=41ms TTL=57 Reply from 209.165.150.24: bytes=32 time=4ms TTL=57 Reply from 209.165.150.24: bytes=32 time=4ms TTL=57 Reply from 209.165.150.24: bytes=32 time=8ms TTL=57 Ping statistics for 209.165.150.24: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 4ms, Maximum = 41ms, Average = 14ms -----Original Message----- From: Jorge Amodio [mailto:jmamodio@gmail.com] Sent: Thursday, September 23, 2010 12:16 PM To: nanog@nanog.org Subject: Re: Facebook Issues/Outage in Southeast? Looks that it came back up now ... root@tango:/var/cache/bind# ping -c 10 www.facebook.com PING a1254.g.akamai.net (209.18.43.169) 56(84) bytes of data. 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=1 ttl=57 time=15.8 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=2 ttl=57 time=16.9 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=3 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=4 ttl=57 time=16.6 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=5 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=6 ttl=57 time=16.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=7 ttl=57 time=17.0 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=8 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=9 ttl=57 time=16.7 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=10 ttl=57 time=16.7 ms J
its up and running now. try 69.63.189.16 On Thu, Sep 23, 2010 at 1:52 PM, Warren Bailey <wbailey@gci.com> wrote:
We've always been able to ping facebook, it seems the people with the most trouble are hitting akamai's caching. The 209.165.150.24 is local to us, as we host akamai for facebook in Alaska.
ping www.facebook.com
Pinging a1254.g.akamai.net [209.165.150.24] with 32 bytes of data:
Reply from 209.165.150.24: bytes=32 time=41ms TTL=57 Reply from 209.165.150.24: bytes=32 time=4ms TTL=57 Reply from 209.165.150.24: bytes=32 time=4ms TTL=57 Reply from 209.165.150.24: bytes=32 time=8ms TTL=57
Ping statistics for 209.165.150.24: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 4ms, Maximum = 41ms, Average = 14ms
-----Original Message----- From: Jorge Amodio [mailto:jmamodio@gmail.com] Sent: Thursday, September 23, 2010 12:16 PM To: nanog@nanog.org Subject: Re: Facebook Issues/Outage in Southeast?
Looks that it came back up now ...
root@tango:/var/cache/bind# ping -c 10 www.facebook.com PING a1254.g.akamai.net (209.18.43.169) 56(84) bytes of data. 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=1 ttl=57 time=15.8 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=2 ttl=57 time=16.9 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=3 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=4 ttl=57 time=16.6 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=5 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=6 ttl=57 time=16.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=7 ttl=57 time=17.0 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=8 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=9 ttl=57 time=16.7 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=10 ttl=57 time=16.7 ms
J
I heard it's an issue with users going through Quest. -----Original Message----- From: Ramanpreet Singh [mailto:sikandar.raman@gmail.com] Sent: Thursday, September 23, 2010 2:24 PM To: Warren Bailey Cc: nanog@nanog.org Subject: Re: Facebook Issues/Outage in Southeast? its up and running now. try 69.63.189.16 On Thu, Sep 23, 2010 at 1:52 PM, Warren Bailey <wbailey@gci.com> wrote:
We've always been able to ping facebook, it seems the people with the most trouble are hitting akamai's caching. The 209.165.150.24 is local to us, as we host akamai for facebook in Alaska.
ping www.facebook.com
Pinging a1254.g.akamai.net [209.165.150.24] with 32 bytes of data:
Reply from 209.165.150.24: bytes=32 time=41ms TTL=57 Reply from 209.165.150.24: bytes=32 time=4ms TTL=57 Reply from 209.165.150.24: bytes=32 time=4ms TTL=57 Reply from 209.165.150.24: bytes=32 time=8ms TTL=57
Ping statistics for 209.165.150.24: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 4ms, Maximum = 41ms, Average = 14ms
-----Original Message----- From: Jorge Amodio [mailto:jmamodio@gmail.com] Sent: Thursday, September 23, 2010 12:16 PM To: nanog@nanog.org Subject: Re: Facebook Issues/Outage in Southeast?
Looks that it came back up now ...
root@tango:/var/cache/bind# ping -c 10 www.facebook.com PING a1254.g.akamai.net (209.18.43.169) 56(84) bytes of data. 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=1 ttl=57 time=15.8 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=2 ttl=57 time=16.9 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=3 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=4 ttl=57 time=16.6 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=5 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=6 ttl=57 time=16.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=7 ttl=57 time=17.0 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=8 ttl=57 time=17.2 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=9 ttl=57 time=16.7 ms 64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169): icmp_seq=10 ttl=57 time=16.7 ms
J
yes, and Qwest is no longer experiencing issues according to IHR. -b -- Bill Blackford Network Engineer Logged into reality and abusing my sudo privileges.....
http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-ou... Apparently, our surmise about Akamai notwithstanding, the problem was actually internal to their app-specific caching facilities, which went into Sorcerer's Apprentice mode, and they had to kill them all and let ghod sort them out. More if I get it; hope that posting's public. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Start a man a fire, and he'll be warm all night. Set a man on fire, and he'll be warm for the rest of his life.
Agreed; my reading of this suggests database caching issues (i.e. all the frontend/middleware clients hitting the main sql cluster at once instead of the memcached farm they normally use), not HTTP/CDN caching issues. -C On Sep 23, 2010, at 7:17 12PM, Jay R. Ashworth wrote:
http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-ou...
Apparently, our surmise about Akamai notwithstanding, the problem was actually internal to their app-specific caching facilities, which went into Sorcerer's Apprentice mode, and they had to kill them all and let ghod sort them out.
More if I get it; hope that posting's public.
Cheers, -- jra
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Start a man a fire, and he'll be warm all night. Set a man on fire, and he'll be warm for the rest of his life.
On Thu, Sep 23, 2010 at 7:17 PM, Jay R. Ashworth <jra@baylink.com> wrote:
http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-ou...
Apparently, our surmise about Akamai notwithstanding, the problem was actually internal to their app-specific caching facilities, which went into Sorcerer's Apprentice mode, and they had to kill them all and let ghod sort them out.
More if I get it; hope that posting's public.
That was a model postmortem. Wish more companies had that sort of detail and clarity around what went wrong and what was being done to fix it. /vijay
Cheers, -- jra
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Start a man a fire, and he'll be warm all night. Set a man on fire, and he'll be warm for the rest of his life.
I can ping them but not access them over Cogent here in Tysons Corner, Virginia - looks like a server issue to me.
Want to see something funnier: http://downrightnow.com/ Exactly the same as what your seeing for facebook. Working icmp, broken http. I wonder if 10^234 people are all trying to find out if facebook is down by going to this site, and crashing it.. :)
Want to see something funnier: http://downrightnow.com/
Exactly the same as what your seeing for facebook. Working icmp, broken http.
downforeveryoneorjustme.com is/was returning intermittent 500 errors, too. fun day. ..d (twiddling his thumbs waiting to test newly built servers that require facebook) -- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html
IPv6 seems to be working fine for me www.v6.facebook.com :) Cameron ========== http://groups.google.com/group/tmoipv6beta ===========
are they down coz of DDoS? On Thu, Sep 23, 2010 at 2:04 PM, Cameron Byrne <cb.list6@gmail.com> wrote:
IPv6 seems to be working fine for me www.v6.facebook.com :)
Cameron ========== http://groups.google.com/group/tmoipv6beta ===========
-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
participants (19)
-
andrew.wallace -
Beavis -
Bill Blackford -
Cameron Byrne -
chaim rieger -
Chris Woodfield -
david raistrick -
gordon b slater -
Greg Olson -
Heath Jones -
Jay R. Ashworth -
jim deleskie -
Jorge Amodio -
Justin Horstman -
Marshall Eubanks -
Ramanpreet Singh -
Seth Mattinen -
vijay gill -
Warren Bailey