Re: DNS poisoning at Google?
It also redirects with facebook, youtube, and ebay but NOT amazon. -Grant On Wed, Jun 27, 2012 at 12:57 AM, Matthew Black <Matthew.Black@csulb.edu>wrote:
Our web lead was able to run curl. Thanks.****
** **
matthew black****
information technology services****
california state university, long beach****
** **
*From:* Grant Ridder [mailto:shortdudey123@gmail.com] *Sent:* Tuesday, June 26, 2012 10:53 PM *To:* Matthew Black *Cc:* Landon Stewart; nanog@nanog.org; Jeremy Hanmer
*Subject:* Re: DNS poisoning at Google?****
** **
Matt, what happens you get on a subnet that can access the webservers directly and bypass the load balancer. Try curl then and see if its something w/ the webserver or load balancer.****
** **
-Grant****
On Wed, Jun 27, 2012 at 12:40 AM, Matthew Black <Matthew.Black@csulb.edu> wrote:****
Thanks again to everyone who helped. I didn't know what to enter with curl, because Outlook clobbered the line breaks in Jeremy's original message.
Also, curl failed on our primary webserver because of firewall and load balancer magic settings. The Telnet method worked better!
Our team is now scouring for that hidden redirect to couchtarts.****
matthew black information technology services california state university, long beach
****
From: Landon Stewart [mailto:lstewart@superb.net]****
Sent: Tuesday, June 26, 2012 10:37 PM To: Matthew Black Cc: Jeremy Hanmer; nanog@nanog.org****
Subject: Re: DNS poisoning at Google?****
There is definitely a 301 redirect.
$ curl -I --referer http://www.google.com/ http://www.csulb.edu/ HTTP/1.1 <http://www.csulb.edu/%0d%0aHTTP/1.1> 301 Moved Permanently Date: Wed, 27 Jun 2012 05:36:31 GMT Server: Apache/2.0.63 Location: http://www.couchtarts.com/media.php Connection: close Content-Type: text/html; charset=iso-8859-1****
On 26 June 2012 22:05, Matthew Black <Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu>> wrote: Google Webtools reports a problem with our HOMEPAGE "/". That page is not redirecting anywhere. They also report problems with some 48 other primary sites, none of which redirect to the offending couchtarts.
matthew black information technology services california state university, long beach
-----Original Message-----****
From: Jeremy Hanmer [mailto:jeremy.hanmer@dreamhost.com<mailto: jeremy.hanmer@dreamhost.com>] Sent: Tuesday, June 26, 2012 9:58 PM To: Matthew Black****
Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: DNS poisoning at Google? It's not DNS. If you're sure there's no htaccess files in place, check your content (even that stored in a database) for anything that might be altering data based on referrer. This simple test shows what I mean:****
Airy:~ user$ curl -e 'http://google.com' csulb.edu<http://csulb.edu> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head>****
<title>301 Moved Permanently</title> </head><body> <h1>Moved Permanently</h1> <p>The document has moved <a href="http://www.couchtarts.com/media.php ">here</a>.</p> </body></html>
Running curl without the -e argument gives the proper site contents.****
On Jun 26, 2012, at 9:24 PM, Matthew Black <Matthew.Black@csulb.edu <mailto:Matthew.Black@csulb.edu>> wrote:
Running Apache on three Solaris webservers behind a load balancer. No MS Windows!
Not sure how malicious software could get between our load balancer and Unix servers. Thanks for the tip!
matthew black information technology services california state university, long beach
****
From: Landon Stewart [mailto:lstewart@superb.net<mailto: lstewart@superb.net>]****
Sent: Tuesday, June 26, 2012 9:07 PM To: Matthew Black****
Cc: nanog@nanog.org<mailto:nanog@nanog.org>****
Subject: Re: DNS poisoning at Google?
Is it possible that some malicious software is listening and injecting a redirect on the wire? We've seen this before with a Windows machine being infected.****
On 26 June 2012 20:53, Matthew Black <Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu>>> wrote: Google Safe Browsing and Firefox have marked our website as containing malware. They claim our home page returns no results, but redirects users to another compromised website couchtarts.com<http://couchtarts.com><; http://couchtarts.com>.****
We have thoroughly examined our root .htaccess and httpd.conf files and
are not redirecting to the problem target site. No recent changes either.
We ran some NSLOOKUPs against various public DNS servers and
intermittently get results that are NOT our servers.
We believe the DNS servers used by Google's crawler have been poisoned.
Can anyone shed some light on this?
matthew black information technology services california state university, long beach****
www.csulb.edu<http://www.csulb.edu><http://www.csulb.edu><; http://www.csulb.edu>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto: LStewart@Superb.Net>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199> Web hosting and more "Ahead****
of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199 Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<; http://www.superbhosting.net/>****
** **
We found the aberrant .htaccess file and have removed it. What a mess! matthew black information technology services california state university, long beach From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: Tuesday, June 26, 2012 11:02 PM To: Matthew Black; nanog@nanog.org Cc: Jeremy Hanmer Subject: Re: DNS poisoning at Google? It also redirects with facebook, youtube, and ebay but NOT amazon. -Grant On Wed, Jun 27, 2012 at 12:57 AM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>> wrote: Our web lead was able to run curl. Thanks. matthew black information technology services california state university, long beach From: Grant Ridder [mailto:shortdudey123@gmail.com<mailto:shortdudey123@gmail.com>] Sent: Tuesday, June 26, 2012 10:53 PM To: Matthew Black Cc: Landon Stewart; nanog@nanog.org<mailto:nanog@nanog.org>; Jeremy Hanmer Subject: Re: DNS poisoning at Google? Matt, what happens you get on a subnet that can access the webservers directly and bypass the load balancer. Try curl then and see if its something w/ the webserver or load balancer. -Grant On Wed, Jun 27, 2012 at 12:40 AM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>> wrote: Thanks again to everyone who helped. I didn't know what to enter with curl, because Outlook clobbered the line breaks in Jeremy's original message. Also, curl failed on our primary webserver because of firewall and load balancer magic settings. The Telnet method worked better! Our team is now scouring for that hidden redirect to couchtarts. matthew black information technology services california state university, long beach From: Landon Stewart [mailto:lstewart@superb.net<mailto:lstewart@superb.net>] Sent: Tuesday, June 26, 2012 10:37 PM To: Matthew Black Cc: Jeremy Hanmer; nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: DNS poisoning at Google? There is definitely a 301 redirect. $ curl -I --referer http://www.google.com/ http://www.csulb.edu/ HTTP/1.1<http://www.csulb.edu/%0d%0aHTTP/1.1> 301 Moved Permanently Date: Wed, 27 Jun 2012 05:36:31 GMT Server: Apache/2.0.63 Location: http://www.couchtarts.com/media.php Connection: close Content-Type: text/html; charset=iso-8859-1 On 26 June 2012 22:05, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>> wrote: Google Webtools reports a problem with our HOMEPAGE "/". That page is not redirecting anywhere. They also report problems with some 48 other primary sites, none of which redirect to the offending couchtarts. matthew black information technology services california state university, long beach -----Original Message----- From: Jeremy Hanmer [mailto:jeremy.hanmer@dreamhost.com<mailto:jeremy.hanmer@dreamhost.com><mailto:jeremy.hanmer@dreamhost.com<mailto:jeremy.hanmer@dreamhost.com>>] Sent: Tuesday, June 26, 2012 9:58 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google? It's not DNS. If you're sure there's no htaccess files in place, check your content (even that stored in a database) for anything that might be altering data based on referrer. This simple test shows what I mean: Airy:~ user$ curl -e 'http://google.com' csulb.edu<http://csulb.edu><http://csulb.edu> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>301 Moved Permanently</title> </head><body> <h1>Moved Permanently</h1> <p>The document has moved <a href="http://www.couchtarts.com/media.php">here</a>.</p> </body></html> Running curl without the -e argument gives the proper site contents. On Jun 26, 2012, at 9:24 PM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>> wrote:
Running Apache on three Solaris webservers behind a load balancer. No MS Windows!
Not sure how malicious software could get between our load balancer and Unix servers. Thanks for the tip!
matthew black information technology services california state university, long beach
From: Landon Stewart [mailto:lstewart@superb.net<mailto:lstewart@superb.net><mailto:lstewart@superb.net<mailto:lstewart@superb.net>>] Sent: Tuesday, June 26, 2012 9:07 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google?
Is it possible that some malicious software is listening and injecting a redirect on the wire? We've seen this before with a Windows machine being infected. On 26 June 2012 20:53, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>>> wrote: Google Safe Browsing and Firefox have marked our website as containing malware. They claim our home page returns no results, but redirects users to another compromised website couchtarts.com<http://couchtarts.com><http://couchtarts.com><http://couchtarts.com>.
We have thoroughly examined our root .htaccess and httpd.conf files and are not redirecting to the problem target site. No recent changes either.
We ran some NSLOOKUPs against various public DNS servers and intermittently get results that are NOT our servers.
We believe the DNS servers used by Google's crawler have been poisoned.
Can anyone shed some light on this?
matthew black information technology services california state university, long beach www.csulb.edu<http://www.csulb.edu><http://www.csulb.edu><http://www.csulb.edu><http://www.csulb.edu>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net><mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net>>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199><tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
The fun part will be figuring out how it got there. :) Sent from my iPhone On Jun 27, 2012, at 12:06 AM, Matthew Black <Matthew.Black@csulb.edu> wrote:
We found the aberrant .htaccess file and have removed it. What a mess!
matthew black information technology services california state university, long beach
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: Tuesday, June 26, 2012 11:02 PM To: Matthew Black; nanog@nanog.org Cc: Jeremy Hanmer Subject: Re: DNS poisoning at Google?
It also redirects with facebook, youtube, and ebay but NOT amazon.
-Grant
On Wed, Jun 27, 2012 at 12:57 AM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>> wrote: Our web lead was able to run curl. Thanks.
matthew black information technology services california state university, long beach
From: Grant Ridder [mailto:shortdudey123@gmail.com<mailto:shortdudey123@gmail.com>] Sent: Tuesday, June 26, 2012 10:53 PM To: Matthew Black Cc: Landon Stewart; nanog@nanog.org<mailto:nanog@nanog.org>; Jeremy Hanmer
Subject: Re: DNS poisoning at Google?
Matt, what happens you get on a subnet that can access the webservers directly and bypass the load balancer. Try curl then and see if its something w/ the webserver or load balancer.
-Grant On Wed, Jun 27, 2012 at 12:40 AM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>> wrote: Thanks again to everyone who helped. I didn't know what to enter with curl, because Outlook clobbered the line breaks in Jeremy's original message.
Also, curl failed on our primary webserver because of firewall and load balancer magic settings. The Telnet method worked better!
Our team is now scouring for that hidden redirect to couchtarts.
matthew black information technology services california state university, long beach
From: Landon Stewart [mailto:lstewart@superb.net<mailto:lstewart@superb.net>] Sent: Tuesday, June 26, 2012 10:37 PM To: Matthew Black Cc: Jeremy Hanmer; nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: DNS poisoning at Google? There is definitely a 301 redirect.
$ curl -I --referer http://www.google.com/ http://www.csulb.edu/ HTTP/1.1<http://www.csulb.edu/%0d%0aHTTP/1.1> 301 Moved Permanently Date: Wed, 27 Jun 2012 05:36:31 GMT Server: Apache/2.0.63 Location: http://www.couchtarts.com/media.php Connection: close Content-Type: text/html; charset=iso-8859-1 On 26 June 2012 22:05, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>> wrote: Google Webtools reports a problem with our HOMEPAGE "/". That page is not redirecting anywhere. They also report problems with some 48 other primary sites, none of which redirect to the offending couchtarts.
matthew black information technology services california state university, long beach
-----Original Message----- From: Jeremy Hanmer [mailto:jeremy.hanmer@dreamhost.com<mailto:jeremy.hanmer@dreamhost.com><mailto:jeremy.hanmer@dreamhost.com<mailto:jeremy.hanmer@dreamhost.com>>] Sent: Tuesday, June 26, 2012 9:58 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google? It's not DNS. If you're sure there's no htaccess files in place, check your content (even that stored in a database) for anything that might be altering data based on referrer. This simple test shows what I mean: Airy:~ user$ curl -e 'http://google.com' csulb.edu<http://csulb.edu><http://csulb.edu> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>301 Moved Permanently</title> </head><body> <h1>Moved Permanently</h1> <p>The document has moved <a href="http://www.couchtarts.com/media.php">here</a>.</p> </body></html>
Running curl without the -e argument gives the proper site contents. On Jun 26, 2012, at 9:24 PM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>> wrote:
Running Apache on three Solaris webservers behind a load balancer. No MS Windows!
Not sure how malicious software could get between our load balancer and Unix servers. Thanks for the tip!
matthew black information technology services california state university, long beach
From: Landon Stewart [mailto:lstewart@superb.net<mailto:lstewart@superb.net><mailto:lstewart@superb.net<mailto:lstewart@superb.net>>] Sent: Tuesday, June 26, 2012 9:07 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google?
Is it possible that some malicious software is listening and injecting a redirect on the wire? We've seen this before with a Windows machine being infected. On 26 June 2012 20:53, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>>> wrote: Google Safe Browsing and Firefox have marked our website as containing malware. They claim our home page returns no results, but redirects users to another compromised website couchtarts.com<http://couchtarts.com><http://couchtarts.com><http://couchtarts.com>.
We have thoroughly examined our root .htaccess and httpd.conf files and are not redirecting to the problem target site. No recent changes either.
We ran some NSLOOKUPs against various public DNS servers and intermittently get results that are NOT our servers.
We believe the DNS servers used by Google's crawler have been poisoned.
Can anyone shed some light on this?
matthew black information technology services california state university, long beach www.csulb.edu<http://www.csulb.edu><http://www.csulb.edu><http://www.csulb.edu><http://www.csulb.edu>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net><mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net>>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199><tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
I'll take files that shouldn't have level 7 permissions for $400 alex. On Wed, Jun 27, 2012 at 2:09 AM, Bryan Irvine <sparctacus@gmail.com> wrote:
The fun part will be figuring out how it got there. :)
Sent from my iPhone
On Jun 27, 2012, at 12:06 AM, Matthew Black <Matthew.Black@csulb.edu> wrote:
We found the aberrant .htaccess file and have removed it. What a mess!
matthew black information technology services california state university, long beach
From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: Tuesday, June 26, 2012 11:02 PM To: Matthew Black; nanog@nanog.org Cc: Jeremy Hanmer Subject: Re: DNS poisoning at Google?
It also redirects with facebook, youtube, and ebay but NOT amazon.
-Grant
On Wed, Jun 27, 2012 at 12:57 AM, Matthew Black <Matthew.Black@csulb.edu <mailto:Matthew.Black@csulb.edu>> wrote: Our web lead was able to run curl. Thanks.
matthew black information technology services california state university, long beach
From: Grant Ridder [mailto:shortdudey123@gmail.com<mailto: shortdudey123@gmail.com>] Sent: Tuesday, June 26, 2012 10:53 PM To: Matthew Black Cc: Landon Stewart; nanog@nanog.org<mailto:nanog@nanog.org>; Jeremy Hanmer
Subject: Re: DNS poisoning at Google?
Matt, what happens you get on a subnet that can access the webservers directly and bypass the load balancer. Try curl then and see if its something w/ the webserver or load balancer.
-Grant On Wed, Jun 27, 2012 at 12:40 AM, Matthew Black <Matthew.Black@csulb.edu <mailto:Matthew.Black@csulb.edu>> wrote: Thanks again to everyone who helped. I didn't know what to enter with curl, because Outlook clobbered the line breaks in Jeremy's original message.
Also, curl failed on our primary webserver because of firewall and load balancer magic settings. The Telnet method worked better!
Our team is now scouring for that hidden redirect to couchtarts.
matthew black information technology services california state university, long beach
From: Landon Stewart [mailto:lstewart@superb.net<mailto: lstewart@superb.net>] Sent: Tuesday, June 26, 2012 10:37 PM To: Matthew Black Cc: Jeremy Hanmer; nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: DNS poisoning at Google? There is definitely a 301 redirect.
$ curl -I --referer http://www.google.com/ http://www.csulb.edu/ HTTP/1.1<http://www.csulb.edu/%0d%0aHTTP/1.1> 301 Moved Permanently Date: Wed, 27 Jun 2012 05:36:31 GMT Server: Apache/2.0.63 Location: http://www.couchtarts.com/media.php Connection: close Content-Type: text/html; charset=iso-8859-1 On 26 June 2012 22:05, Matthew Black <Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu>>> wrote: Google Webtools reports a problem with our HOMEPAGE "/". That page is not redirecting anywhere. They also report problems with some 48 other primary sites, none of which redirect to the offending couchtarts.
matthew black information technology services california state university, long beach
-----Original Message----- From: Jeremy Hanmer [mailto:jeremy.hanmer@dreamhost.com<mailto: jeremy.hanmer@dreamhost.com><mailto:jeremy.hanmer@dreamhost.com<mailto: jeremy.hanmer@dreamhost.com>>] Sent: Tuesday, June 26, 2012 9:58 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org <mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google? It's not DNS. If you're sure there's no htaccess files in place, check your content (even that stored in a database) for anything that might be altering data based on referrer. This simple test shows what I mean: Airy:~ user$ curl -e 'http://google.com' csulb.edu<http://csulb.edu><; http://csulb.edu> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>301 Moved Permanently</title> </head><body> <h1>Moved Permanently</h1> <p>The document has moved <a href="http://www.couchtarts.com/media.php ">here</a>.</p> </body></html>
Running curl without the -e argument gives the proper site contents. On Jun 26, 2012, at 9:24 PM, Matthew Black <Matthew.Black@csulb.edu <mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu>>> wrote:
Running Apache on three Solaris webservers behind a load balancer. No MS Windows!
Not sure how malicious software could get between our load balancer and Unix servers. Thanks for the tip!
matthew black information technology services california state university, long beach
From: Landon Stewart [mailto:lstewart@superb.net<mailto: lstewart@superb.net><mailto:lstewart@superb.net<mailto:lstewart@superb.net ] Sent: Tuesday, June 26, 2012 9:07 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org <mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google?
Is it possible that some malicious software is listening and injecting a redirect on the wire? We've seen this before with a Windows machine being infected. On 26 June 2012 20:53, Matthew Black <Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu>><mailto:Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto: Matthew.Black@csulb.edu>>>> wrote: Google Safe Browsing and Firefox have marked our website as containing malware. They claim our home page returns no results, but redirects users to another compromised website couchtarts.com<http://couchtarts.com><; http://couchtarts.com><http://couchtarts.com>.
We have thoroughly examined our root .htaccess and httpd.conf files and are not redirecting to the problem target site. No recent changes either.
We ran some NSLOOKUPs against various public DNS servers and intermittently get results that are NOT our servers.
We believe the DNS servers used by Google's crawler have been poisoned.
Can anyone shed some light on this?
matthew black information technology services california state university, long beach www.csulb.edu<http://www.csulb.edu><http://www.csulb.edu><; http://www.csulb.edu><http://www.csulb.edu>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto: LStewart@Superb.Net><mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net
Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199><tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto: LStewart@Superb.Net>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<; http://www.superbhosting.net/>
Ahh, but how did it get there in the first place. Matthew, meet can of worms. I presume you have an opener. -- ian -----Original Message----- From: Matthew Black Sent: 27/06/2012, 08:07 To: Grant Ridder; nanog@nanog.org Cc: Jeremy Hanmer Subject: RE: DNS poisoning at Google? We found the aberrant .htaccess file and have removed it. What a mess! matthew black information technology services california state university, long beach From: Grant Ridder [mailto:shortdudey123@gmail.com] Sent: Tuesday, June 26, 2012 11:02 PM To: Matthew Black; nanog@nanog.org Cc: Jeremy Hanmer Subject: Re: DNS poisoning at Google? It also redirects with facebook, youtube, and ebay but NOT amazon. -Grant On Wed, Jun 27, 2012 at 12:57 AM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>> wrote: Our web lead was able to run curl. Thanks. matthew black information technology services california state university, long beach From: Grant Ridder [mailto:shortdudey123@gmail.com<mailto:shortdudey123@gmail.com>] Sent: Tuesday, June 26, 2012 10:53 PM To: Matthew Black Cc: Landon Stewart; nanog@nanog.org<mailto:nanog@nanog.org>; Jeremy Hanmer Subject: Re: DNS poisoning at Google? Matt, what happens you get on a subnet that can access the webservers directly and bypass the load balancer. Try curl then and see if its something w/ the webserver or load balancer. -Grant On Wed, Jun 27, 2012 at 12:40 AM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>> wrote: Thanks again to everyone who helped. I didn't know what to enter with curl, because Outlook clobbered the line breaks in Jeremy's original message. Also, curl failed on our primary webserver because of firewall and load balancer magic settings. The Telnet method worked better! Our team is now scouring for that hidden redirect to couchtarts. matthew black information technology services california state university, long beach From: Landon Stewart [mailto:lstewart@superb.net<mailto:lstewart@superb.net>] Sent: Tuesday, June 26, 2012 10:37 PM To: Matthew Black Cc: Jeremy Hanmer; nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: DNS poisoning at Google? There is definitely a 301 redirect. $ curl -I --referer http://www.google.com/ http://www.csulb.edu/ HTTP/1.1<http://www.csulb.edu/%0d%0aHTTP/1.1> 301 Moved Permanently Date: Wed, 27 Jun 2012 05:36:31 GMT Server: Apache/2.0.63 Location: http://www.couchtarts.com/media.php Connection: close Content-Type: text/html; charset=iso-8859-1 On 26 June 2012 22:05, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>> wrote: Google Webtools reports a problem with our HOMEPAGE "/". That page is not redirecting anywhere. They also report problems with some 48 other primary sites, none of which redirect to the offending couchtarts. matthew black information technology services california state university, long beach -----Original Message----- From: Jeremy Hanmer [mailto:jeremy.hanmer@dreamhost.com<mailto:jeremy.hanmer@dreamhost.com><mailto:jeremy.hanmer@dreamhost.com<mailto:jeremy.hanmer@dreamhost.com>>] Sent: Tuesday, June 26, 2012 9:58 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google? It's not DNS. If you're sure there's no htaccess files in place, check your content (even that stored in a database) for anything that might be altering data based on referrer. This simple test shows what I mean: Airy:~ user$ curl -e 'http://google.com' csulb.edu<http://csulb.edu><http://csulb.edu> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>301 Moved Permanently</title> </head><body> <h1>Moved Permanently</h1> <p>The document has moved <a href="http://www.couchtarts.com/media.php">here</a>.</p> </body></html> Running curl without the -e argument gives the proper site contents. On Jun 26, 2012, at 9:24 PM, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>> wrote:
Running Apache on three Solaris webservers behind a load balancer. No MS Windows!
Not sure how malicious software could get between our load balancer and Unix servers. Thanks for the tip!
matthew black information technology services california state university, long beach
From: Landon Stewart [mailto:lstewart@superb.net<mailto:lstewart@superb.net><mailto:lstewart@superb.net<mailto:lstewart@superb.net>>] Sent: Tuesday, June 26, 2012 9:07 PM To: Matthew Black Cc: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: DNS poisoning at Google?
Is it possible that some malicious software is listening and injecting a redirect on the wire? We've seen this before with a Windows machine being infected. On 26 June 2012 20:53, Matthew Black <Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu><mailto:Matthew.Black@csulb.edu<mailto:Matthew.Black@csulb.edu>>>> wrote: Google Safe Browsing and Firefox have marked our website as containing malware. They claim our home page returns no results, but redirects users to another compromised website couchtarts.com<http://couchtarts.com><http://couchtarts.com><http://couchtarts.com>.
We have thoroughly examined our root .htaccess and httpd.conf files and are not redirecting to the problem target site. No recent changes either.
We ran some NSLOOKUPs against various public DNS servers and intermittently get results that are NOT our servers.
We believe the DNS servers used by Google's crawler have been poisoned.
Can anyone shed some light on this?
matthew black information technology services california state university, long beach www.csulb.edu<http://www.csulb.edu><http://www.csulb.edu><http://www.csulb.edu><http://www.csulb.edu>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net><mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net>>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199><tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
-- Landon Stewart <LStewart@Superb.Net<mailto:LStewart@Superb.Net<mailto:LStewart@Superb.Net>>> Sr. Administrator Systems Engineering Superb Internet Corp - 888-354-6128 x 4199<tel:888-354-6128%20x%204199> Web hosting and more "Ahead of the Rest": http://www.superbhosting.net<http://www.superbhosting.net/>
On Jun 27, 2012, at 12:06 AM, Matthew Black wrote:
We found the aberrant .htaccess file and have removed it. What a mess!
Trusting you carefully noted the date/time stamp before removing it, as that's an important bit of forensics. Aloha, Michael. -- "Please have your Internet License and Usenet Registration handy..."
On Jun 27, 2012, at 3:36 AM, Michael J Wise wrote:
On Jun 27, 2012, at 12:06 AM, Matthew Black wrote:
We found the aberrant .htaccess file and have removed it. What a mess!
Trusting you carefully noted the date/time stamp before removing it, as that's an important bit of forensics.
And done forget there is a trail on that file on your backups. Tom
This may not help Matt now, but I just came across this today and believe it may help others who have to deal with incidents: http://cert.societegenerale.com/en/publications.html --> "IRM (Incident Response Methodologies)" If you changed the file contents before noting the created date, modified date, etc. then begin looking at your backups. This date will then help you track down the log entries and finally lead you to the root cause. Also, if possible, please post the culprit code that caused this, exif'ing the sensitive data of course :-) -- Thank you, Robert Miller http://www.armoredpackets.com Twitter: @arch3angel On 6/27/12 7:50 AM, TR Shaw wrote:
On Jun 27, 2012, at 3:36 AM, Michael J Wise wrote:
On Jun 27, 2012, at 12:06 AM, Matthew Black wrote:
We found the aberrant .htaccess file and have removed it. What a mess!
Trusting you carefully noted the date/time stamp before removing it, as that's an important bit of forensics. And done forget there is a trail on that file on your backups.
Tom
Yes, we did that and also noted the username and IP address from where the FTP upload originated. matthew black information technology services california state university, long beach -----Original Message----- From: Michael J Wise [mailto:mjwise@kapu.net] Sent: Wednesday, June 27, 2012 12:37 AM To: nanog@nanog.org Subject: Re: DNS poisoning at Google? On Jun 27, 2012, at 12:06 AM, Matthew Black wrote:
We found the aberrant .htaccess file and have removed it. What a mess!
Trusting you carefully noted the date/time stamp before removing it, as that's an important bit of forensics. Aloha, Michael. -- "Please have your Internet License and Usenet Registration handy..."
participants (8)
-
AP NANOG -
Bryan Irvine -
Grant Ridder -
Ian McDonald -
Ishmael Rufus -
Matthew Black -
Michael J Wise -
TR Shaw