----- Forwarded message from Alexis Rosen <alexis@panix.com> ----- X-Original-To: tls@netbsd.org Delivered-To: tls@netbsd.org Resent-Message-Id: <200501170842.j0H8gWi21166@panix5.panix.com> X-Original-To: tls@panix.com Delivered-To: tls@rek.tjls.com Date: Mon, 17 Jan 2005 01:42:04 -0500 From: Alexis Rosen <alexis@panix.com> To: nanog@merit.edu Subject: Panix.com- Some brief comments on the hijacking of our domain User-Agent: Mutt/1.4.2.1i [Please note: I tried to post this five hours ago. It didn't make it, though I resubscribed to nanog-post (and acked the confirmation check) about half an hour previously. I'm resending (with light edits) and CCing this to a few friends; if any of you get this and see that it's not on nanog yet, please resend it for me. Thanks.] We're still digging out from under here, so I can't say nearly as much as I'd like. However, I have a few things that really need to be said sooner rather than later. (A couple of the later points are operational. Skip to "***" if you don't care who I'm grateful to...) First, I want to thank Martin Hannigan at Verisign. Whatever I may think of the (in)action I got from other parties there, he made significant efforts to get them to move, and the incomplete view of events that I have leads me to believe that it's his efforts, and the efforts of others at Verisign that he worked on, that got Melbourne IT to finally get off the dime. This was a very serious effort on his part, for someone who wasn't his direct customer, and I'm very appreciative of the concern and the effort. (This isn't to say that the immense efforts of other parties wasn't also helpful in this respect.) Secondly, I want to thank the MANY people here (and elsewhere), most of whom I don't know and have never had contact with, who devoted time and energy to this issue. Some I do know, and some of them were especially generous. You know who you are, but a partial list includes Thor Simon, Perry Metzger, Steve Bellovin, Bill Manning, and <hm, I don't know if I can say those names>. Thank you. Third (here's the "***"), I want to make a plea for those with operational control over large nameservers to reload their caches or expire out the panix.com entries from their caches, if they haven't yet picked up the correct data for our zone. (Note that having correct "NS" records isn't sufficient if you're caching all types.) The correct zones can be pulled from 198.7.0.1 or 198.7.0.2, for comparison's sake. If any of you have hand-copied our data into your DNS, please delete it so we're not afflicted by odd bits of stale data in the far future, when this incident is long forgotten. I noted something very odd earlier today. The A records for the hosts purporting to be mail.panix.com and mail2.panix.com were changed, with the last octets switched to ".0", making them unreachable. At the time I was grateful (because mail was being queued or bounced at the sender side, rather than bounced- and possibly copied- at the recipient side) but I didn't have time to try to figure out who had done what. I still don't know who/what was responsible, but I thank those who are, and just so I have a fuller understanding, I'd appreciate it if someone who knows what was done would contact me and fill me in. Someone here pointed out that we seem to have an SSH daemon running on port 80. That's intentional. It's on our shell hosts, and it's actually a clever bit of front-end code that switches web clients to a web server and ssh clients to the ssh daemon. It's for the benefit of customers who want to ssh in but are behind dumbass (or rightfully paranoid, take your pick) firewalls that don't allow out anything but connections to port 80. Thor and others have been commenting a bit on the fact that *something* is broken or compromised, either at MelbourneIT, Dotster, or Verisign. I hope that now that it's Monday morning in Australia, and will be in 12-15 hours here in the US, we can make some progress on figuring out what really happened. This would start with Verisign, Dotster, and MelbourneIT producing *all* relevant logs. I'll be discussing that with them tomorrow. There's a lot more to be said here, but for now we're going to finish cleaning up the mess, get the registry back to dotster, and try to catch up on some sleep. Oh, and work with various law enforcement types to try to catch the bastards responsible for this. /a --- Alexis Rosen President Public Access Networks Corp. - Panix.com alexis@panix.com Grand Central Server LLC. alexis@grandcentralserver.com ----- End forwarded message -----