July 2012 - Test - lists.nanog.org

FYI Netflix is down
by Joe Blanchard 04 Jul '12

04 Jul '12

Seems that they are unreachable at the moment. Called and theres a recorded message stating they are aware of an issue, no details. -Joe

42 84

Arbor network
by Waseem 04 Jul '12

04 Jul '12

Hi, Anybody using Arbor Peakflow? Can tell us about its efficiency? Any other DDoS detection and mitigation product ? Regards, Waseem

1 0

Re: FYI Netflix is down
by Steven Bellovin 03 Jul '12

03 Jul '12

On Jul 2, 2012, at 3:43 PM, Greg D. Moore wrote: > At 03:08 PM 7/2/2012, George Herbert wrote: > > If folks have not read it, I would suggest reading Normal Accidents by Charles Perrow. Strong second to that suggestion. --Steve Bellovin, https://www.cs.columbia.edu/~smb

3 2

DNS poisoning at Google?
by Matthew Black 03 Jul '12

03 Jul '12

Google Safe Browsing and Firefox have marked our website as containing malware. They claim our home page returns no results, but redirects users to another compromised website couchtarts.com. We have thoroughly examined our root .htaccess and httpd.conf files and are not redirecting to the problem target site. No recent changes either. We ran some NSLOOKUPs against various public DNS servers and intermittently get results that are NOT our servers. We believe the DNS servers used by Google's crawler have been poisoned. Can anyone shed some light on this? matthew black information technology services california state university, long beach www.csulb.edu<http://www.csulb.edu>

15 23

Re: FYI Netflix is down
by Jon Lewis 03 Jul '12

03 Jul '12

On Mon, 2 Jul 2012, Greg D. Moore wrote: > As for pulling the plug to test stuff. I recall a demo at Netapps in the > early 00's. They were talking about their fault tolerance and how great it > was. So I walked up to their demo array and said, "So, it shouldn't be a > problem if I pulled this drive right here?" Before I could the salesperson > or tech guy, can't remember, told me to stop. He didn't want to risk it. Lightweight. Your story reminded me of this Sun ZFS demo. http://www.youtube.com/watch?v=QGIwg6ye1gE ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

1 0

Fwd: Re: F-ckin Leap Seconds, how do they work?
by shawn wilson 03 Jul '12

03 Jul '12

---------- Forwarded message ---------- From: "shawn wilson" <ag4ve.us(a)gmail.com> Date: Jul 3, 2012 11:33 AM Subject: Re: F-ckin Leap Seconds, how do they work? To: "Joel jaeggli" <joelja(a)bogus.com> I agree with TAI. Epoch is supposed to be an unsigned long int starting ~1970 (there are are 4 epochs iirc, but never mind that). I don't recall the rfc, but I don't recall this spec mentioning leap seconds (and it shouldn't). Frankly our time system is buggy as hell (no year 0, base 60 seconds and minutes, base 24 hours, no standard month base, e/m isn't a part of the system, etc). I find my last issue the most disconcerting with our system and makes it really unreliable - GPS time is *not* earth time and we rely on that skew for everything. To that point, I hate to think how many missile tests it took them to figure that one out :) However there is no reason to add more crap to an already messed up system. On Jul 3, 2012 10:03 AM, "Joel jaeggli" <joelja(a)bogus.com> wrote: > On 7/3/12 01:54 , Wolfgang S. Rupprecht wrote: > > > > Steven Bellovin <smb(a)cs.columbia.edu> writes: > >> See > >> > http://landslidecoding.blogspot.com/2012/07/linuxs-leap-second-deadlocks.ht… > > > > Maybe we should stop wrenching the poor system time back and forth. We > > no longer add or subtract daylight savings time (or timezones) to the > > kernel time, why do we do it with leapseconds? We should really move > > the leapseconds correction into the display routines like DST and > > timezones already are. I believe the Olson time code already has ifdefs > > for doing this. I wonder why the system's internal time isn't run that > > way. > > Neither timezones nor dst impact length of the mean solar day. > > TAI is some 35 seconds ahead of UTC this point. and will continue to > diverge in a fashion which is not sufficiently predictable that you can > know over the long term. > > Not using utc as the timebase is certainly possible, gps does that for > example. > > Apps are buggy sounds like a really poor excuse for doing so. > > > > -wolfgang > > > > > >

3 2

Re: FYI Netflix is down
by Hal Murray 03 Jul '12

03 Jul '12

George Herbert <george.herbert(a)gmail.com> said: > I worked for a Sun clone vendor (Axil) for a while and took some of our > systems and storage to Comdex one year in the 90s. We had a RAID unit > (Mylex controller) we had just introduced. Beforehand, I made REALLY REALLY > SURE that the pull-the-disk and pull-the-redundant-power tricks worked. And > showed them to people with the "Please keep in mind that this voids the > warranty, but here we *rip* go...". All of the other server vendors were > giving me dirty looks for that one. Apparently I sold a few systems that > way. :) Nice. Thanks. Many years ago, I worked for one of DEC's research groups. We built a network using FDDI 4B/5B link technology based on AMD TAXI chips. (They were state of the art back then.) The switches were 3U(?) boxes with 12 ports. It took a rack of 6 or 8 of them in the phone closet to cover a floor. Workstations had 2 cables plugged into different switches. In theory, we covered any single point of failure. My office was near the phone closet. I got to watch my boss give demos to visiting VIPs. He was pretty good at it. In the middle of explaining things, he would grab a power cord and yank it. Blinka-blinka=blinka and the remaining switches would reconfigure and go back to work. (It took under a second.) It was interesting to watch the VIPs. Most of them got it: the network really could recover quickly. The interesting ones had a telco background. They were really surprised. The concept of disrupting live traffic for something as insignificant as a demo was off scale in their culture. It was just a research lab. We were used to eating our own dog food. ---------- "Greg D. Moore" <mooregr(a)greenms.com> said: > If folks have not read it, I would suggest reading Normal Accidents by > Charles Perrow. +1 > The "it can't happen" is almost guaranteed to happen. ;-) And when it > does, it'll often interact in ways we can't predict or sometimes even > understand. My memory of that sort of event is roughly... (see above for context) The hardware broke and turned a vanilla packet into a super-long packet. My FPGA code was supposed to catch that case and do something sane. It was never tested and didn't work. It poured crap all over memory. Needless to say, things went downhill from there. Easy to spot in hindsight. None of us thought that was an interesting case while we were testing. -- These are my opinions. I hate spam.

1 0

Contributing to the community
by Matt Chung 03 Jul '12

03 Jul '12

I've been so fortunate and appreciative over the years to have colleagues (many whom I consider my close friends) cultivate my career by providing sound advise that I will continue to pass on. In addition to those I've known personally, I have gleaned a substantial amount of information through many of you who've contributed to these threads, blogs, and so on. Within the organizations I've worked for, I have always been an advocate for sharing knowledge in order for the company to grow collectively; I truly believe its infectious. But I digress... At my previous company (regional WISP) as a network engineer, I was able to get buy in from the partners to conduct training for our call center in effort to better support our customers. By institutionalizing a methodical approach to troubleshooting (and performing root cause analysis), we can filter out many potential issues (i.e why check if there is network connectivity if you are getting an HTTP response - ruled out the lower stack). That was great however...despite contributing to my organization, I've always felt that I haven't performed due diligence when it comes to contributing back to the network/IT community as an entity. Excuses have been made ("I don't have time") on my part and I realized that everyone here is a working professional as well. I've never been an active participant like many of you. As a person who is passionate about this field (as well as a working professional), how do you find the time in order to contribute? Do you ever feel that the post may be redundant? Another factor I've always took into consideration was the fact that although I may be knowledgeable and proficient in one facet, someone out there is the true expert (i.e assisted in developing the RFC) and has a deeper understanding than I do (which I feel my contribution may be inadequate). -- -Matt Chung

1 0

Northern Virginia 9-1-1 service after storm
by Sean Donelan 03 Jul '12

03 Jul '12

Probably not as interesting as talking about Amazon/Netflix. http://www.washingtonpost.com/local/after-storm-911-phone-service-remains-s… Fairfax County's 911 emergency center operated at just half capacity Monday as Verizon struggled to figure out why both its primary and backup power systems failed after Friday night's storm and left much of Northern Virginia without 911 service through the weekend.

1 0

Re: FYI Netflix is down
by Greg D. Moore 02 Jul '12

02 Jul '12

At 05:04 PM 7/2/2012, George Herbert wrote: >On Mon, Jul 2, 2012 at 12:43 PM, Greg D. Moore <mooregr(a)greenms.com> wrote: > > At 03:08 PM 7/2/2012, George Herbert wrote: > > > > If folks have not read it, I would suggest reading Normal Accidents by > > Charles Perrow. > > > > The "it can't happen" is almost guaranteed to happen. ;-) And > when it does, > > it'll often interact in ways we can't predict or sometimes even understand. > >Seconded. I figured you had probably read it. :-) >There are also aerospace and nuclear and failure analysis books which >are good, but I often encourage people to start with that one. > > > As for pulling the plug to test stuff. I recall a demo at Netapps in the > > early 00's. They were talking about their fault tolerance and how great it > > was. So I walked up to their demo array and said, "So, it shouldn't be a > > problem if I pulled this drive right here?" Before I could the salesperson > > or tech guy, can't remember, told me to stop. He didn't want to risk it. > > > > That right there said loads about their confidence in their own system. > >I worked for a Sun clone vendor (Axil) for a while and took some of >our systems and storage to Comdex one year in the 90s. We had a RAID >unit (Mylex controller) we had just introduced. Beforehand, I made >REALLY REALLY SURE that the pull-the-disk and pull-the-redundant-power >tricks worked. And showed them to people with the "Please keep in >mind that this voids the warranty, but here we *rip* go...". All of >the other server vendors were giving me dirty looks for that one. >Apparently I sold a few systems that way. I can imagine. Back when we were testing a cluster from MicronPC, the techs were in our office and they encouraged us to do that. It was re-assuring. >You have to watch for connector wear-out and things like that, but ... > >All the clusters I've built, I've insisted on a burn-in time plug pull >test on all the major components. We caught things with those from >time to time. Especially with N+1, if it is really N+0 due to a bug >or flaw you need to know that... About 7 years back, we were about to move a production platform to a cluster+SAN that an outside vendor had installed. I was brought in at the last minute to lead the project. Before we did the move, I said, "Umm, has anyone tried a remote reboot of the servers?" "Oh they rebooted fine when we were at the datacenter with the vendor. We're good." I repeated my question and finally did the old, "Ok, I know I'm being a pain, but please, let's just try it once, remotely before we're committed." So we rebooted, and wait, and waited, and waited. It took a trip out to the datacenter (we couldn't afford good remote KVM tools back then) to see that the server was trying to mount stuff off of something on the network. At first we couldn't figure out what it was. Finally realized it was looking for files on the vendor's laptop. So of course it had worked fine when the vendor was at the datacenter. Despite all that, the vendor still denied it being their problem. Anyway, enough reminiscing. Things happen. We can only do so much to prevent them, and never assume. >-- >-george william herbert >george.herbert(a)gmail.com Greg D. Moore http://greenmountainsoftware.wordpress.com/ CEO QuiCR: Quick, Crowdsourced Responses. http://www.quicr.net

1 0