On Sun, Jan 15, 2006 at 03:32:02PM -0800, Matt Ghali wrote:

On Sun, 15 Jan 2006, Elijah Savage wrote:

Any validatity to this and if so I am suprised that our team has got no calls on not be able to get to certain websites.

http://webhostingtalk.com/showthread.php?t=477562

I for one applaud godaddy's response. If more piddling "Hosting Providers" with "Datacenters" got turned off when they started spewing abusive traffic, the net would be a much nicer place.

Whoever the heck "nectartech" is, I guess they might act a little more responsibly in the future. Or, more probably, they'll just change to another DNS registrar who doesn't care as much about abuse.

FYI, Nectartech is a small hosting shop out of 55 S Market in San Jose. I wouldn't describe them as a "datacenter", since I don't think they own or operate any facilities. Perhaps if they ever managed to find "the command to make two routers talk to each other and be redundant" (a real quote from what has been loosely described as their network admin, I'm not kidding, you can't make stuff like this up :P), their next step might be to find the command to make dns servers talk to each other and be redundant. Reality check time, what we have here is a small hosting shop with a long history of shady customers. I doubt GoDaddy nukes nameservers on a whim, my money is that there was a lot of abuse which went on for a long time without getting any response. Its amazing how quickly some people who don't respond or address abuse issues at all when you're asking nicely will appear and take care of things once you turn them off. The rest is just some random blowhard web hosting customer who gets off on being an ass and blaming everyone but himself and his choice in hosting companies. Hardly an uncommon sight. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Mon, 16 Jan 2006, Joe McGuckin wrote:

Richard,

On the other hand , I'm not comfortable with the idea that an organization that provides network infrastructure services under the aegis of the US Government could unilaterally revoke those services for something that is not illegal.

It does not have to be illegal. All that is necessary is that customer who purchased the service beware and agree to the policies prior to making the purchase (of course, almost nobody fully reads that long agreement you get presented on the website, but that's another story...) Not being somebody who've ever used godaddy's services, I'm just speculating based on various reports, but I think their registration service agreement is more extensive then domain registration agreement from most other registrars and prohibits use of the domain in connection with spamming as well as in connection with illegal activities. If policies are violated then domain maybe suspended until problem is resolved. I suspect they don't suspend right away and have system of requiring domain owner be available for notification and conversation in case such use (prohibited by their service agreement) is reported. If they do not hear anything about it and reports continue then they take action as allowed by domain registration agreement. What we probably saw is such action after nectartech failed to respond to several notifications and probably kept server running without fully cleaning it up and possibly more then one of their servers was hacked too. This is similar enough situation to what may happen when you run servers on the connection purchased from your ISP and that ISP actually takes abuse reports seriously and has working abuse department that follows up on what is sent them. That this was spinned around as datacenter shutdown on WHT and even got here is a result of both how nectartech wanted itself seen and who they had for dealing with such vendor actions.

On Mon, 16 Jan 2006, Richard A Steenbergen wrote:

...

The rest is just some random blowhard web hosting customer

I disagree with this particular part. I think its quite clear that this was not "random blowhard hosting customer" but somebody close to nectartech owner who owner knew could get through walls put by some companies and if not annoy the hell out of them afterward and spin it around in [in]appropriate way. -- William Leibzon Elan Networks william@elan.net

william(at)elan.net wrote:

...

...

On Mon, 16 Jan 2006, Richard A Steenbergen wrote:

...

The rest is just some random blowhard web hosting customer

I disagree with this particular part. I think its quite clear that this was not "random blowhard hosting customer" but somebody close to nectartech owner who owner knew could get through walls put by some companies and if not annoy the hell out of them afterward and spin it around in [in]appropriate way.

Precisely. It wasn't just some random blowhard web hosting customer. It was a carefully selected web hosting customer specifically chosen for his expertise at being a blowhard.

He sounds like a blowhard to me, and he delayed them getting back online as quick as he could. GoDaddy gave him the same sphiel I've heard 100 times i.e. here's our procedures please do x, y, and z. If you look at the guys web page, he takes pride in being a blow hard so don't fret, he'd disagree with you too. No doubt he's reading NANOG and probably yelling at the mailing admins about how he has to sign up for two lists vs. one and how stupid we all are. -M<

On Jan 17, 2006, at 1:32 AM, Jim Popovitch wrote:

I want to say, from an outsider's perspective, that I whole heartily applaud GoDaddy on the actions they took [...]

There seems to be a wide split on this topic. I was wondering if people would privately tell me yes or no on a few questions so I can understand the issue better. 1) Do you think it is acceptable to cause any collateral damage to innocent bystanders if it will stop network abuse? 2) If yes, do you still think it is acceptable to take down 100s of innocent bystanders because one customer of a provider is misbehaving? 3) If yes, do you still think it is acceptable if the "misbehaving" customer is not intentionally misbehaving - i.e. they've been hacked? 3) If yes, do you still think it is acceptable if the collateral damage (taking out 100s of innocent businesses) doesn't actually stop the spam run / DoS attack / etc.? These are important question to me, and I'm surprised at the number of people who seem to feel so very differently than I thought they would feel - than I personally feel. Would people mind sending me private e-mails with yes/no answers? Longer answers are welcome, but yes/no will do. Using the case under discussion as an example, I am wondering why anyone thinks taking down 100s of innocent domains is a good way to stop a single hacked machine from doing whatever it is doing? If you somehow think all that is worth it, take a close look at your cost / benefit analysis. At this rate, every business on the Internet will be out of business before we take out even a single moderately large botnet. I am also wondering why anyone thinks the miscreant will stop just because the legitimate owner's domain no longer resolves? Not only is the machine likely to continue sending spam as if nothing happened, we aren't even "catching" the guy. I guess you could say "well, it put pressure on his hosting provider to clean the infected machine", which is true. I just think that's a bit silly. But maybe I'm the one who's silly. Lastly, I wonder what "average" people - people who run businesses on hosting providers who really don't understand all this computer stuff - think about such actions. How many 100s of people have we just alienated for life to stop - er, NOT stop - a single zombie? And how many of their friends are going to hear over an over how the Internet is not a real business and no one should put any faith in it? Is this really a good thing? -- TTFN, patrick

On Tue, 2006-01-17 at 03:19 -0500, Richard A Steenbergen wrote:

The question at hand is, at what point does a registrar providing services have an ethical or moral obligation to step in and do something when they do encounter an excessive level of abuse by someone using their services?

I think the issue here is not so much what happened, but how it happened. The phishing problem was originally reported to godaddy and then passed on to nectar on 1/9 (a Monday). It also appears the nectar folks resolved the problem on the same day. After that point godaddy continued to receive complains about the same problem and rather than checking to see if the problem still existed, they just assumed it did. Nectar appears to have even responded to godaddy stating that the problem had already been resolved long before service was cut. IMHO the big issue is that service was cut on a Friday night just as the only folks empowered to resolve the situation have left for the weekend. I can see cutting service during a weekday morning to get the client's attention on the matter. Doing it at a time when you know you'll be causing a long term outage is just plain nasty. HTH, Chris

Patrick W. Gilmore wrote:

On Jan 17, 2006, at 1:32 AM, Jim Popovitch wrote:

...

I want to say, from an outsider's perspective, that I whole heartily applaud GoDaddy on the actions they took [...]

There seems to be a wide split on this topic. I was wondering if people would privately tell me yes or no on a few questions so I can understand the issue better.

1) Do you think it is acceptable to cause any collateral damage to innocent bystanders if it will stop network abuse?

In some cases. Our policy is to minimize such. Example: Customer has a NATted network with multiple machines sharing one global address. One of the machines at customer's premise is causing abuse (virus, etc.) Null-routing one specific IP address will cause collateral damage to the non-infected machines at that customer, but I think most of here would agree that such is justified. Obviously, if the impact of the abuse is minimal, having the customer fix the problem before shutting anything down is preferred. Another example would be a customer's webserver which has many name-based virtual hosts, one of which is abusive, and you are providing IP connectivity. By null-routing one IP you are causing collateral damage to the non-abusive virtual host customers of your customer, but I think most would think that justified.

2) If yes, do you still think it is acceptable to take down 100s of innocent bystanders because one customer of a provider is misbehaving?

I assume here that you mean "Customer of a customer". Again, it depends. If the customer has continual problems controlling abuse from his customers, or you suspect that your customer is playing "whack-a-mole", or the abuse is ongoing and/or serious and you can't identify which of customer's customers is the cause (spoofed source addresses, etc.) in some cases yes.

3) If yes, do you still think it is acceptable if the "misbehaving" customer is not intentionally misbehaving - i.e. they've been hacked?

Again, it depends on the seriousness of the abuse and its affect on the network, as well as the frequency thereof and the seriousness of the customer in rectifying the problem. Also whether you can reasonably isolate the abuse and disconnect only the customer's abusive customer.

3) If yes, do you still think it is acceptable if the collateral damage (taking out 100s of innocent businesses) doesn't actually stop the spam run / DoS attack / etc.?

If it doesn't stop it but stops your network from being a part of it, yes. If it has no affect on it at all, then you're probably pulling the wrong plug.

These are important question to me, and I'm surprised at the number of people who seem to feel so very differently than I thought they would feel - than I personally feel. Would people mind sending me private e-mails with yes/no answers? Longer answers are welcome, but yes/no will do.

This is IMHO operational, so posting publicly. I don't think this is as black-and-white as to warrant simple yes-no answers. There are policies involved as well as your agreements with your peers/upstreams. If the issue is serious enough that you risk losing your own connectivity because you can't stem the abuse from a customer's customer, then you may need to do so, or the end result will be that you become part of greater collateral damage.

Using the case under discussion as an example, I am wondering why anyone thinks taking down 100s of innocent domains is a good way to stop a single hacked machine from doing whatever it is doing? If you somehow think all that is worth it, take a close look at your cost / benefit analysis. At this rate, every business on the Internet will be out of business before we take out even a single moderately large botnet.

The present example seems to be a combination of poor communication, bad attitude and sloppy network design from what I've seen here. It's unclear to me exactly what GoDaddy shut down, and the only data points we have to go on are admittedly edited conversations that took place after the plug was pulled. What went on beforehand? Did Nectar indeed make a good faith effort to correct the original problem? Was their attitude the same as shown on the phone calls? How long had the problem existed, had it happened before, and did Nectar keep an open dialogue as to the steps they were taking to fix it? Did GoDaddy have less intrusive options to shut down just the abuser?

I am also wondering why anyone thinks the miscreant will stop just because the legitimate owner's domain no longer resolves? Not only is the machine likely to continue sending spam as if nothing happened, we aren't even "catching" the guy. I guess you could say "well, it put pressure on his hosting provider to clean the infected machine", which is true. I just think that's a bit silly. But maybe I'm the one who's silly.

I think this was a case of a fake phishing website rather than outgoing spam spew. If the domain was the target of a phish, then causing it not to resolve would keep the phisher from reaping any benefit from the abuse although the spam run would likely continue, at least for a while until the phisher realizes it is in vain.

Lastly, I wonder what "average" people - people who run businesses on hosting providers who really don't understand all this computer stuff - think about such actions. How many 100s of people have we just alienated for life to stop - er, NOT stop - a single zombie? And how many of their friends are going to hear over an over how the Internet is not a real business and no one should put any faith in it?

Well, "average" people who run businesses on hosting providers" probably should hire someone who does understand all this computer stuff to do some due diligence on the providers they are considering. If their prospective providers netblocks are repeatedly mentioned in SPEWS, Spamhaus, Spamcop, and NANAE, they may want to look elsewhere. Googling "Nectartech abuse" is interesting. As far back as July of last year they were battling GoDaddy over spam and abuse issues. It doesn't look like this should have been all that big of a surprise. In fact, Nectartech's predictions in post 23 of the following thread are eerily accurate. http://www.webhostingtalk.com/showthread.php?s=&threadid=422612

Is this really a good thing?

If steps are taken to minimize collateral damage, yes. Allowing the abuse to continue causes collateral damage to the rest of the Internet for as long as it continues. The choice often boils down to severe collateral damage to a few or raising the noise level and collateral damage to the Internet as a whole. Is cutting off ten customers of an infected customer better than allowing this customer's virus to infect tens of thousands of random hosts on the net worth it? If you're one of the tens of thousands, yes. If you're one of the ten customers, no. -- Jay Hennigan - CCIE #7880 - Network Administration - jay@west.net NetLojix Communications, Inc. - http://www.netlojix.com/ WestNet: Connecting you to the planet. 805 884-6323

On Mon, 16 Jan 2006, Jim Popovitch wrote: [jim, please wrap your text!]

I have never been a Go Daddy customer, but I certainly appreciate their stand on this issue. I will probably never be a Nectartech customer after this episode.

Hear Hear. After reading the GoDaddy domain registration legal agreement, available at: https://www.godaddy.com/gdshop/legal_agreements/show_doc.asp?se=%2B&ci=1839&pageid=REG%5FSA especially section 7, "Restriction of Services, Right of Refusal", I have to give them a big thumbs up. It is good to see that wielding a Big Stick, and actively working for the Good Guys has not hindered GoDaddy from achieving quite a bit of success in the market. matto --matt@snark.net------------------------------------------<darwin>< The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke

--On January 17, 2006 7:27:20 AM -0500 "Robert E.Seastrom" <rs@seastrom.com> wrote:

Now that Go Daddy has ensured that I'll never do business with them (which is a shame; I liked certain lawsuits that they brought in the past, but if being their customer means subscribing to their thought police, count me out), I think it's time to carefully go over the registration agreements with the registrars I use... never know when someone will slip in something truly odious, and the argument that none of them would be so crazy as to try it appears to be incorrect.

This thread gets less and less operational....however...I'm trying to keep this in scope...I think this relates operationally because we all have and enforce AUPs and ToS on our customer bases, both internal, and external. We also have AUPs and ToS enforced on us, by business relationships and peerings, etc. Most ToS and AUP out there at the consumer level state basically the service is worthless, that we can and will d/c you at will, without cause, at our whim. Overzealous lawyering has made this a necessity. How much any of these might or might not stand up in court, I have no clue. As you get into the business world some ToS and AUP become more weighty, but far more structured. Giving both sides clearer and well defined policies and practices for responding to issues. Requiring notification, escalation, etc. I think what matters is the way that the AUPs are applied. This case...the facts...don't match up. webhosting.info (not an authoritative source mind you, but a datapoint) only sees ~150 hosts by this ISP. From what I understand this number is from whois data with nameservers pointing to theirs. Contrast this with mydyndns.org, google.com, ebay.com, prioritycolo.com, wellsfargo.com (ok so this ones not that much more, at ~800), even sun.com has more domains listed. Those last two aren't even 'in the business' and they have more. While they may have a large datacenter, I'm not even remotely sure that this incident darkened the whole thing. It might've taken rDNS offline, but that's far from darkening a whole datacenter. It sounds like another WHTer puffing themselves up to being bigger than they are. They *must* be small to let a *CUSTOMER* advocate for them to a third party! Nectartech clearly knew about this and sanctioned it, and the person recording the phone calls has pointed this out more than once. There are no facts in this case either way, because it is really Go Daddy against Nectartech. And Nectartech has a lot more reason to lie to make itself look better in front of its customers. If their whole datacenter went dark then it's some unrelated thing, or some really bad practice (such as somehow establishing iBGP based on domain names maybe? hell I dunno). I've seen so much utter BS spouted by a lot of the self proclaimed web hosts on WHT that I'm not inclined to believe his side of the story any more (or any less) because of it. Go Daddy has to my knowledge never been draconian in applying their AUP (I think atleast some of us here would know about it if so).

On Tue, 17 Jan 2006, Matt Ghali wrote:

On Tue, 17 Jan 2006, Robert E.Seastrom wrote:

...

The first and second paragraphs are sane. The last paragraph gives Go Daddy the right to capriciously and arbitrarily delete your domain for any reason they wish ("Morally objectionable activities will include, but not be limited to...")

Do you believe that your philosophical objections to the language absolves you as a customer from the minimal due dilligence of knowing what you are agreeing to?

Find me a registrar that DOESN'T have that kind of language in their user agreements, then tell me if anyone wishing to do any kind of e-commerce has a choice. I've gone off on a tear about this before: A registrar has a license to print money. Boilerplate user agreements that leave the user zero recourse are the standard. I haven't seen a registrar yet that doesn't have this kind of verbiage completely freeing them from liability for *any* action taken on a domain registration, including none. - billn

--On January 16, 2006 10:32:58 PM -0800 Jim Popovitch <jimpop@yahoo.com> wrote:

I want to say, from an outsider's perspective, that I whole heartily applaud GoDaddy on the actions they took and the consistent professionalism exhibited by their tech support representative. Despite obvious (and heavily edited) calls to the same agent, the consumer was informed in a professional manner of his/her avenue for resolution. No doubt remains in my mind that the caller was not caught blind by this situation. Go Daddy has a privacy policy that no doubt prohibits them from releasing details of their side of this case, however to me the recording suggests that the caller knew this was the end result, not a sudden surprise move, and they just wanted to circumvent standard procedure. The caller's prior thought to record, what appears as a standard call to tech-support, is insightful and should be an obvious sign of his motivation.

Theres a clear case of he said they said going on with this case. Nectartech is making claims that they fixed the issue. Also note that the caller is not a Nectartech employee at all. He's a customer who's also friends with the owner. Atleast that's what he says in WHT thread. In any event I don't think Nectartech handled this very well, and more likely than not still had a problem and were given ample time to properly correct it.

I'm not sure how on-topic this is/was, but considering long thread and different opinions that were expressed before, I believe some here may want to have additional information I recently read: http://www.emailbattles.com/archive/battles/phish_aacgebeeje_hc/

The article author talked to both nectartech and godaddy and is also including copies of emails from nectartech side as to their conversations with godaddy. The last one (on how domain can be reactivated) you may find most interesting if you're not otherwise familiar with godaddy's policies: http://www.trimmail.com/news/archive/extra/godaddy_v_nectartech/14012006/

The customer service aspects of it are less impressive. I originally thought, based on information available at that time, that GoDaddy did a decent, or even a good job, at "handling" the call. Today, I think they did an OK job. Nothing exemplary, but definately not bad from an operations perspective. What is interesting is the concept of calling a rack, or a row, a "datacenter". It's becoming more commonplace for terms to be exaggerated these days i.e. "datacenter". Another interesting point is that GoDaddy charged a $199 reconnect fee. They punished the operator for the behavoir of their customers. -M<

Joe McGuckin wrote:

On the other hand �, I'm not comfortable with the idea that an organization that provides network infrastructure services under the aegis of the US Government could unilaterally revoke those services for something that is not illegal.

You could say I do that. I am not a registrar, but I do host DNS for many domains. So if my customer spams and I cut them off, including DNS, do you have a problem with that too? -- Steve Sobol, Professional Geek 888-480-4638 PGP: 0xE3AE35ED Company website: http://JustThe.net/ Personal blog, resume, portfolio: http://SteveSobol.com/ E: sjsobol@JustThe.net Snail: 22674 Motnocab Road, Apple Valley, CA 92307

On Mon, 16 Jan 2006, Richard A Steenbergen wrote:

FYI, Nectartech is a small hosting shop out of 55 S Market in San Jose. I wouldn't describe them as a "datacenter", since I don't think they own or operate any facilities.

Heh, I used to work at a small hosting shop out of 55 S. Market- it was (then) called BBN Planet. I guess these schmoes rent a cage from Genuity (or whatever they are called now).

Perhaps if they ever managed to find "the command to make two routers talk to each other and be redundant" (a real quote from what has been loosely described as their network admin, I'm not kidding, you can't make stuff like this up :P), their next step might be to find the command to make dns servers talk to each other and be redundant.

Seriously. You need to be spewing a lot of cak onto the net for your _domain registrar_ to take notice.

The rest is just some random blowhard web hosting customer who gets off on being an ass and blaming everyone but himself and his choice in hosting companies. Hardly an uncommon sight. :)

The priceless part is that we probably never would have noticed, had he not had the hubris to record the conversations, and then publish the URL to them. I love it when the lusers are nice enough to clearly identify themselves. matto --matt@snark.net------------------------------------------<darwin>< The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke

On 15-Jan-2006, at 18:15, Elijah Savage wrote:

...

Any validatity to this and if so I am suprised that our team has got no calls on not be able to get to certain websites.

http://webhostingtalk.com/showthread.php?t=477562

I think the main thing I learned from that is that there are a surprising number of hosting companies and self-professed data centre operators who really don't know much about the DNS.

The GoDaddy guy didn't do such a bad job. It sounds like they had some procedures and they followed them. http://marc.perkel.com/audio/godaddy.mp3 Operationally, not having someone on the shift who can make decisions is not a good thing. It's like having a NOC with no shift supervisor. If you're big enough - a manager. Disclaimer: In now way, shape, or form, should that be inferred as a plug for or against GoDaddy. I'm nuetral. Best! -M<

I think the main thing I learned from that is that there are a surprising number of hosting companies and self-professed data centre operators who really don't know much about the DNS.

Or even what the word "datacenter" means. Sounds to me like a rack of servers or a cage was suspended, not "an entire datacenter" which was claimed several times. The recorded phone call was basically a lesson in how NOT to escalate a call, from both sides involved. From the customer's side if he'd not been so confrontational, he probably would have gotten his problem solved. From the operator's side, they should have a procedure for dealing with abuse and critical escalations 7/24. Just my perception. --chuck

On Tuesday 17 Jan 2006 01:04, you wrote:

Not having all your DNS servers in the same domain, or registered through the same registrar, isn't a "best practice" that has previously occurred to me, but it makes a lot of sense now that I think about it.

I think the general consensus in the DNS field is that for security reasons it is preferable to have as small a set of DNS servers (or perhaps as small as set of differently configured servers! Hmm physical security....) in the hierarchy above you as possible, since compromise of any of these could affect the results obtained for your domain. See also DJBs "Trusted Servers" note. http://cr.yp.to/djbdns/notes.html Here there is a clear conflict between security through redundancy against accident, and resistant to compromise. Although it can be mitigated by choosing well managed parents zones. Incidently we have DNS servers in two domains, but that is historical, and both top level domains are managed by Verisign, and delivered via the same set of servers. Thus we are dependent on "root-servers.net", "gltd-servers.net" and our own servers, only in the resolution of our own domain names (and customer domains, where those domains are in .com/.net). Of course arguably the effective working of some services (email?) are now also dependent on reverse DNS working well, and the delegation of that is different again. That said I think the idea is sound against some issues (at which point one should probably also use different providers for the DNS registration services, since if their procedures are flawed....). However it does increase the risk of certain types of malicious activity, as in general it is sufficent to compromise one DNS server involved in serving a name to compromise the majority of the traffic (at least in theory, I haven't had a chance to prove this in anger yet). Since we are moving a couple of our nameservers from their current domain, I think I'll look at putting them under co.uk, as the UK seems to have tidied up its DNS management quite nicely in recent years. Also during recent event it has struck me that the hierarchy of servers involved in providing DNS services is quite small, and has quite different characteristics to the other records in the DNS. I'm beginning to wonder if having the scaffolding in the protocol itself is the right way, but that is a debate that has raged before, and is off topic here.