- NANOG - lists.nanog.org

DNS poisoning at Google?
by Matthew Black 03 Jul '12

03 Jul '12

Google Safe Browsing and Firefox have marked our website as containing malware. They claim our home page returns no results, but redirects users to another compromised website couchtarts.com. We have thoroughly examined our root .htaccess and httpd.conf files and are not redirecting to the problem target site. No recent changes either. We ran some NSLOOKUPs against various public DNS servers and intermittently get results that are NOT our servers. We believe the DNS servers used by Google's crawler have been poisoned. Can anyone shed some light on this? matthew black information technology services california state university, long beach www.csulb.edu<http://www.csulb.edu>

15 23

Re: FYI Netflix is down
by Jon Lewis 03 Jul '12

03 Jul '12

On Mon, 2 Jul 2012, Greg D. Moore wrote: > As for pulling the plug to test stuff. I recall a demo at Netapps in the > early 00's. They were talking about their fault tolerance and how great it > was. So I walked up to their demo array and said, "So, it shouldn't be a > problem if I pulled this drive right here?" Before I could the salesperson > or tech guy, can't remember, told me to stop. He didn't want to risk it. Lightweight. Your story reminded me of this Sun ZFS demo. http://www.youtube.com/watch?v=QGIwg6ye1gE ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

1 0

Fwd: Re: F-ckin Leap Seconds, how do they work?
by shawn wilson 03 Jul '12

03 Jul '12

---------- Forwarded message ---------- From: "shawn wilson" <ag4ve.us(a)gmail.com> Date: Jul 3, 2012 11:33 AM Subject: Re: F-ckin Leap Seconds, how do they work? To: "Joel jaeggli" <joelja(a)bogus.com> I agree with TAI. Epoch is supposed to be an unsigned long int starting ~1970 (there are are 4 epochs iirc, but never mind that). I don't recall the rfc, but I don't recall this spec mentioning leap seconds (and it shouldn't). Frankly our time system is buggy as hell (no year 0, base 60 seconds and minutes, base 24 hours, no standard month base, e/m isn't a part of the system, etc). I find my last issue the most disconcerting with our system and makes it really unreliable - GPS time is *not* earth time and we rely on that skew for everything. To that point, I hate to think how many missile tests it took them to figure that one out :) However there is no reason to add more crap to an already messed up system. On Jul 3, 2012 10:03 AM, "Joel jaeggli" <joelja(a)bogus.com> wrote: > On 7/3/12 01:54 , Wolfgang S. Rupprecht wrote: > > > > Steven Bellovin <smb(a)cs.columbia.edu> writes: > >> See > >> > http://landslidecoding.blogspot.com/2012/07/linuxs-leap-second-deadlocks.ht… > > > > Maybe we should stop wrenching the poor system time back and forth. We > > no longer add or subtract daylight savings time (or timezones) to the > > kernel time, why do we do it with leapseconds? We should really move > > the leapseconds correction into the display routines like DST and > > timezones already are. I believe the Olson time code already has ifdefs > > for doing this. I wonder why the system's internal time isn't run that > > way. > > Neither timezones nor dst impact length of the mean solar day. > > TAI is some 35 seconds ahead of UTC this point. and will continue to > diverge in a fashion which is not sufficiently predictable that you can > know over the long term. > > Not using utc as the timebase is certainly possible, gps does that for > example. > > Apps are buggy sounds like a really poor excuse for doing so. > > > > -wolfgang > > > > > >

3 2

Re: FYI Netflix is down
by Hal Murray 03 Jul '12

03 Jul '12

George Herbert <george.herbert(a)gmail.com> said: > I worked for a Sun clone vendor (Axil) for a while and took some of our > systems and storage to Comdex one year in the 90s. We had a RAID unit > (Mylex controller) we had just introduced. Beforehand, I made REALLY REALLY > SURE that the pull-the-disk and pull-the-redundant-power tricks worked. And > showed them to people with the "Please keep in mind that this voids the > warranty, but here we *rip* go...". All of the other server vendors were > giving me dirty looks for that one. Apparently I sold a few systems that > way. :) Nice. Thanks. Many years ago, I worked for one of DEC's research groups. We built a network using FDDI 4B/5B link technology based on AMD TAXI chips. (They were state of the art back then.) The switches were 3U(?) boxes with 12 ports. It took a rack of 6 or 8 of them in the phone closet to cover a floor. Workstations had 2 cables plugged into different switches. In theory, we covered any single point of failure. My office was near the phone closet. I got to watch my boss give demos to visiting VIPs. He was pretty good at it. In the middle of explaining things, he would grab a power cord and yank it. Blinka-blinka=blinka and the remaining switches would reconfigure and go back to work. (It took under a second.) It was interesting to watch the VIPs. Most of them got it: the network really could recover quickly. The interesting ones had a telco background. They were really surprised. The concept of disrupting live traffic for something as insignificant as a demo was off scale in their culture. It was just a research lab. We were used to eating our own dog food. ---------- "Greg D. Moore" <mooregr(a)greenms.com> said: > If folks have not read it, I would suggest reading Normal Accidents by > Charles Perrow. +1 > The "it can't happen" is almost guaranteed to happen. ;-) And when it > does, it'll often interact in ways we can't predict or sometimes even > understand. My memory of that sort of event is roughly... (see above for context) The hardware broke and turned a vanilla packet into a super-long packet. My FPGA code was supposed to catch that case and do something sane. It was never tested and didn't work. It poured crap all over memory. Needless to say, things went downhill from there. Easy to spot in hindsight. None of us thought that was an interesting case while we were testing. -- These are my opinions. I hate spam.

1 0

Contributing to the community
by Matt Chung 03 Jul '12

03 Jul '12

I've been so fortunate and appreciative over the years to have colleagues (many whom I consider my close friends) cultivate my career by providing sound advise that I will continue to pass on. In addition to those I've known personally, I have gleaned a substantial amount of information through many of you who've contributed to these threads, blogs, and so on. Within the organizations I've worked for, I have always been an advocate for sharing knowledge in order for the company to grow collectively; I truly believe its infectious. But I digress... At my previous company (regional WISP) as a network engineer, I was able to get buy in from the partners to conduct training for our call center in effort to better support our customers. By institutionalizing a methodical approach to troubleshooting (and performing root cause analysis), we can filter out many potential issues (i.e why check if there is network connectivity if you are getting an HTTP response - ruled out the lower stack). That was great however...despite contributing to my organization, I've always felt that I haven't performed due diligence when it comes to contributing back to the network/IT community as an entity. Excuses have been made ("I don't have time") on my part and I realized that everyone here is a working professional as well. I've never been an active participant like many of you. As a person who is passionate about this field (as well as a working professional), how do you find the time in order to contribute? Do you ever feel that the post may be redundant? Another factor I've always took into consideration was the fact that although I may be knowledgeable and proficient in one facet, someone out there is the true expert (i.e assisted in developing the RFC) and has a deeper understanding than I do (which I feel my contribution may be inadequate). -- -Matt Chung

1 0

Northern Virginia 9-1-1 service after storm
by Sean Donelan 03 Jul '12

03 Jul '12

Probably not as interesting as talking about Amazon/Netflix. http://www.washingtonpost.com/local/after-storm-911-phone-service-remains-s… Fairfax County's 911 emergency center operated at just half capacity Monday as Verizon struggled to figure out why both its primary and backup power systems failed after Friday night's storm and left much of Northern Virginia without 911 service through the weekend.

1 0

Re: FYI Netflix is down
by Greg D. Moore 02 Jul '12

02 Jul '12

At 05:04 PM 7/2/2012, George Herbert wrote: >On Mon, Jul 2, 2012 at 12:43 PM, Greg D. Moore <mooregr(a)greenms.com> wrote: > > At 03:08 PM 7/2/2012, George Herbert wrote: > > > > If folks have not read it, I would suggest reading Normal Accidents by > > Charles Perrow. > > > > The "it can't happen" is almost guaranteed to happen. ;-) And > when it does, > > it'll often interact in ways we can't predict or sometimes even understand. > >Seconded. I figured you had probably read it. :-) >There are also aerospace and nuclear and failure analysis books which >are good, but I often encourage people to start with that one. > > > As for pulling the plug to test stuff. I recall a demo at Netapps in the > > early 00's. They were talking about their fault tolerance and how great it > > was. So I walked up to their demo array and said, "So, it shouldn't be a > > problem if I pulled this drive right here?" Before I could the salesperson > > or tech guy, can't remember, told me to stop. He didn't want to risk it. > > > > That right there said loads about their confidence in their own system. > >I worked for a Sun clone vendor (Axil) for a while and took some of >our systems and storage to Comdex one year in the 90s. We had a RAID >unit (Mylex controller) we had just introduced. Beforehand, I made >REALLY REALLY SURE that the pull-the-disk and pull-the-redundant-power >tricks worked. And showed them to people with the "Please keep in >mind that this voids the warranty, but here we *rip* go...". All of >the other server vendors were giving me dirty looks for that one. >Apparently I sold a few systems that way. I can imagine. Back when we were testing a cluster from MicronPC, the techs were in our office and they encouraged us to do that. It was re-assuring. >You have to watch for connector wear-out and things like that, but ... > >All the clusters I've built, I've insisted on a burn-in time plug pull >test on all the major components. We caught things with those from >time to time. Especially with N+1, if it is really N+0 due to a bug >or flaw you need to know that... About 7 years back, we were about to move a production platform to a cluster+SAN that an outside vendor had installed. I was brought in at the last minute to lead the project. Before we did the move, I said, "Umm, has anyone tried a remote reboot of the servers?" "Oh they rebooted fine when we were at the datacenter with the vendor. We're good." I repeated my question and finally did the old, "Ok, I know I'm being a pain, but please, let's just try it once, remotely before we're committed." So we rebooted, and wait, and waited, and waited. It took a trip out to the datacenter (we couldn't afford good remote KVM tools back then) to see that the server was trying to mount stuff off of something on the network. At first we couldn't figure out what it was. Finally realized it was looking for files on the vendor's laptop. So of course it had worked fine when the vendor was at the datacenter. Despite all that, the vendor still denied it being their problem. Anyway, enough reminiscing. Things happen. We can only do so much to prevent them, and never assume. >-- >-george william herbert >george.herbert(a)gmail.com Greg D. Moore http://greenmountainsoftware.wordpress.com/ CEO QuiCR: Quick, Crowdsourced Responses. http://www.quicr.net

1 0

Re: FYI Netflix is down
by George Herbert 02 Jul '12

02 Jul '12

On Mon, Jul 2, 2012 at 12:43 PM, Greg D. Moore <mooregr(a)greenms.com> wrote: > At 03:08 PM 7/2/2012, George Herbert wrote: > > If folks have not read it, I would suggest reading Normal Accidents by > Charles Perrow. > > The "it can't happen" is almost guaranteed to happen. ;-) And when it does, > it'll often interact in ways we can't predict or sometimes even understand. Seconded. There are also aerospace and nuclear and failure analysis books which are good, but I often encourage people to start with that one. > As for pulling the plug to test stuff. I recall a demo at Netapps in the > early 00's. They were talking about their fault tolerance and how great it > was. So I walked up to their demo array and said, "So, it shouldn't be a > problem if I pulled this drive right here?" Before I could the salesperson > or tech guy, can't remember, told me to stop. He didn't want to risk it. > > That right there said loads about their confidence in their own system. I worked for a Sun clone vendor (Axil) for a while and took some of our systems and storage to Comdex one year in the 90s. We had a RAID unit (Mylex controller) we had just introduced. Beforehand, I made REALLY REALLY SURE that the pull-the-disk and pull-the-redundant-power tricks worked. And showed them to people with the "Please keep in mind that this voids the warranty, but here we *rip* go...". All of the other server vendors were giving me dirty looks for that one. Apparently I sold a few systems that way. You have to watch for connector wear-out and things like that, but ... All the clusters I've built, I've insisted on a burn-in time plug pull test on all the major components. We caught things with those from time to time. Especially with N+1, if it is really N+0 due to a bug or flaw you need to know that... -- -george william herbert george.herbert(a)gmail.com

1 0

RE: FYI Netflix is down
by Dan Golding 02 Jul '12

02 Jul '12

> -----Original Message----- > From: Greg D. Moore [mailto:mooregr@greenms.com] > > > If folks have not read it, I would suggest reading Normal Accidents by > Charles Perrow. > Also, Human Error by James Reason.

1 0

Re: FYI Netflix is down
by Greg D. Moore 02 Jul '12

02 Jul '12

At 03:08 PM 7/2/2012, George Herbert wrote: If folks have not read it, I would suggest reading Normal Accidents by Charles Perrow. The "it can't happen" is almost guaranteed to happen. ;-) And when it does, it'll often interact in ways we can't predict or sometimes even understand. As for pulling the plug to test stuff. I recall a demo at Netapps in the early 00's. They were talking about their fault tolerance and how great it was. So I walked up to their demo array and said, "So, it shouldn't be a problem if I pulled this drive right here?" Before I could the salesperson or tech guy, can't remember, told me to stop. He didn't want to risk it. That right there said loads about their confidence in their own system. >Late reply, but: > >On Sat, Jun 30, 2012 at 12:30 AM, Lynda <shrdlu(a)deaddrop.org> wrote: > >... > > Second, and more important. I *was* a "computer science guy" in a > past life, > > and this is nonsense. You can have astonishingly large software projects > > that just continue to run smoothly, day in, day out, and they don't hit the > > news, because they don't break. There are data centers that don't hit the > > news, in precisely the same way. > >I really need to write the book on IT reliability I keep meaning to. > >There's reliability - backwards looking statistical, which can be 100% >for a given service or datacenter - and then there's dependability, >forwards-predicted outage risks, which people often *assert* equals >the prior reliability record, but in reality you often have a number >of latent failures (and latent cascade paths) that you do not >understand, did not identify previously, and are not aware of. > >I've had or had to respond to over a billion dollars of culminative IT >disaster loss over my consulting career so far; I have NEVER seen >anyone who did it perfect, even the best pros. And I include myself >in that list. > >Looking at other fields like aerospace and nuclear engineering, what >is done in IT is not anywhere close to the same level of QA and >engineering analysis and testing. We cannot assert better results >with less work. > >"Oh, that never happens", except I've had my stuff in three locations >that had catastrophic generator failures. "Oh, that never happens" >when you're doing power maintenance and the best-rated electrical >company in California, in conjunction with the generator vendor and a >couple of independent power EEs, mis-balance the maintenance generator >loads between legs and blow the generators and datacenter. "Oh, that >never happens" that the datacenter burns (or starts to burn and then >gets flooded). "Oh, that never happens" that the FM-200 goes off or >preaction breaks and water leaks. "Oh, that never happens" that well >maintained and monitored and triple-redundant AC units all trip >offline due to a common mode failure over the course of a weekend and >the room gets up to 106 degrees. Oh thank god the next thing didn't >go wrong in THAT situation, because the spot temperature meters >indicated that the ceiling height of that particular room peaked at 1 >degree short of the temp at which the sprinkler heads are supposed to >discharge, so we nearly lost that room to flooding rather than just a >10% disk and 15% power supply attrition over the next year... > >Don't be so confident in the infrastructure. It's not engineered or >built or maintained well enough to actually support that assertion. >The same can be said of the application software and application >architecture and integration. > > >-- >-george william herbert >george.herbert(a)gmail.com Greg D. Moore http://greenmountainsoftware.wordpress.com/ CEO QuiCR: Quick, Crowdsourced Responses. http://www.quicr.net

1 0