This is a coordinated hacking. (Was Re: Need help in flushing DNS)
This is most definitely a coordinated and planned attack. And by 'attack' I mean hijacking of domain names. I show as of this morning nearly fifty thousand domain names that appear suspicious. I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET). Anyone credentialed (credentialed /n/., "I know you or know of you,") wanting data, e-mail me off-list for some TLD goodness. On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan <philfagan@gmail.com> wrote:
Agree'd in these "smaller" scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide "services" are effected. Perhaps the result of a State led campaign ....topic for another day.
On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson <fergdawgster@gmail.com
wrote:
I am betting that Netsol doesn't need any more "coordination" at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS:
; <<>> DiG 9.7.3 <<>> @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;parsonstech.com. IN NS
;; ANSWER SECTION: parsonstech.com. 172800 IN NS ns2617.ztomy.com. parsonstech.com. 172800 IN NS ns1617.ztomy.com.
;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81
- ferg
On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan <philfagan@gmail.com> wrote:
I should caveat.....coordinate the "recovery" of.
On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth <brandon@rd.bbc.co.uk>wrote:
Is there an organization that coordinates outages like this amongst the industry?
No, usually they are surprise outages though Anonymous have tried coordinating a few
brandon
-- Phil Fagan Denver, CO 970-480-7618
-- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
-- Phil Fagan Denver, CO 970-480-7618
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime. I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones. Is this of value? Does it need to be automated? - Jared On Jun 20, 2013, at 3:53 PM, jamie rishaw <j@arpa.com> wrote:
This is most definitely a coordinated and planned attack.
And by 'attack' I mean hijacking of domain names.
I show as of this morning nearly fifty thousand domain names that appear suspicious.
I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET).
Anyone credentialed (credentialed /n/., "I know you or know of you,") wanting data, e-mail me off-list for some TLD goodness.
On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan <philfagan@gmail.com> wrote:
Agree'd in these "smaller" scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide "services" are effected. Perhaps the result of a State led campaign ....topic for another day.
On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson <fergdawgster@gmail.com
wrote:
I am betting that Netsol doesn't need any more "coordination" at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS:
; <<>> DiG 9.7.3 <<>> @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;parsonstech.com. IN NS
;; ANSWER SECTION: parsonstech.com. 172800 IN NS ns2617.ztomy.com. parsonstech.com. 172800 IN NS ns1617.ztomy.com.
;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81
- ferg
On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan <philfagan@gmail.com> wrote:
I should caveat.....coordinate the "recovery" of.
On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth <brandon@rd.bbc.co.uk>wrote:
Is there an organization that coordinates outages like this amongst the industry?
No, usually they are surprise outages though Anonymous have tried coordinating a few
brandon
-- Phil Fagan Denver, CO 970-480-7618
-- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
-- Phil Fagan Denver, CO 970-480-7618
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of "localhost.". Is this some sort of tactic I'm unaware of? On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch <jared@puck.nether.net> wrote:
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime.
I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones.
Is this of value? Does it need to be automated?
- Jared
On Jun 20, 2013, at 3:53 PM, jamie rishaw <j@arpa.com> wrote:
This is most definitely a coordinated and planned attack.
And by 'attack' I mean hijacking of domain names.
I show as of this morning nearly fifty thousand domain names that appear suspicious.
I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET).
Anyone credentialed (credentialed /n/., "I know you or know of you,") wanting data, e-mail me off-list for some TLD goodness.
On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan <philfagan@gmail.com> wrote:
Agree'd in these "smaller" scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide "services" are effected. Perhaps the result of a State led campaign ....topic for another day.
On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson <fergdawgster@gmail.com
wrote:
I am betting that Netsol doesn't need any more "coordination" at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS:
; <<>> DiG 9.7.3 <<>> @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;parsonstech.com. IN NS
;; ANSWER SECTION: parsonstech.com. 172800 IN NS ns2617.ztomy.com. parsonstech.com. 172800 IN NS ns1617.ztomy.com.
;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81
- ferg
On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan <philfagan@gmail.com> wrote:
I should caveat.....coordinate the "recovery" of.
On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth <brandon@rd.bbc.co.uk>wrote:
> Is there an organization that coordinates outages like this amongst the > industry?
No, usually they are surprise outages though Anonymous have tried coordinating a few
brandon
-- Phil Fagan Denver, CO 970-480-7618
-- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
-- Phil Fagan Denver, CO 970-480-7618
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Poisoning a domain's NS records with localhost will most certainly DOS the domain, yes. I have not yet seen the source of this; if anyone has a clue where the updates are coming from please post the info. Is there anything about ztomy.com that has been seen that's supicious as in they might be the origin? This could be them, or could be a joe-job against them. I do not want to point a finger lacking any sort of actual data dump of the poisoning activity... On Thu, Jun 20, 2013 at 1:02 PM, jamie rishaw <j@arpa.com> wrote:
I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of "localhost.".
Is this some sort of tactic I'm unaware of?
On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch <jared@puck.nether.net> wrote:
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime.
I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones.
Is this of value? Does it need to be automated?
- Jared
On Jun 20, 2013, at 3:53 PM, jamie rishaw <j@arpa.com> wrote:
This is most definitely a coordinated and planned attack.
And by 'attack' I mean hijacking of domain names.
I show as of this morning nearly fifty thousand domain names that appear suspicious.
I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET).
Anyone credentialed (credentialed /n/., "I know you or know of you,") wanting data, e-mail me off-list for some TLD goodness.
On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan <philfagan@gmail.com> wrote:
Agree'd in these "smaller" scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide "services" are effected. Perhaps the result of a State led campaign ....topic for another day.
On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson < fergdawgster@gmail.com
wrote:
I am betting that Netsol doesn't need any more "coordination" at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS:
; <<>> DiG 9.7.3 <<>> @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;parsonstech.com. IN NS
;; ANSWER SECTION: parsonstech.com. 172800 IN NS ns2617.ztomy.com. parsonstech.com. 172800 IN NS ns1617.ztomy.com.
;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81
- ferg
On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan <philfagan@gmail.com> wrote:
I should caveat.....coordinate the "recovery" of.
On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth <brandon@rd.bbc.co.uk>wrote:
>> Is there an organization that coordinates outages like this amongst the >> industry? > > No, usually they are surprise outages though Anonymous have tried > coordinating a few > > brandon >
-- Phil Fagan Denver, CO 970-480-7618
-- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
-- Phil Fagan Denver, CO 970-480-7618
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
-- -george william herbert george.herbert@gmail.com
It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. As far as the logic of the DNS, it is functioning as designed (What's up, Vix!) - There's another aspect of this that caused this situation. Any Alexa or similar people on this list (Goog PR, etc)? I'd love to bulk submit a domain list for some analytics. Contact me off list. On Thu, Jun 20, 2013 at 3:14 PM, George Herbert <george.herbert@gmail.com>wrote:
Poisoning a domain's NS records with localhost will most certainly DOS the domain, yes.
I have not yet seen the source of this; if anyone has a clue where the updates are coming from please post the info.
Is there anything about ztomy.com that has been seen that's supicious as in they might be the origin? This could be them, or could be a joe-job against them. I do not want to point a finger lacking any sort of actual data dump of the poisoning activity...
On Thu, Jun 20, 2013 at 1:02 PM, jamie rishaw <j@arpa.com> wrote:
I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of "localhost.".
Is this some sort of tactic I'm unaware of?
On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch <jared@puck.nether.net> wrote:
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime.
I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones.
Is this of value? Does it need to be automated?
- Jared
On Jun 20, 2013, at 3:53 PM, jamie rishaw <j@arpa.com> wrote:
This is most definitely a coordinated and planned attack.
And by 'attack' I mean hijacking of domain names.
I show as of this morning nearly fifty thousand domain names that appear suspicious.
I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET).
Anyone credentialed (credentialed /n/., "I know you or know of you,") wanting data, e-mail me off-list for some TLD goodness.
On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan <philfagan@gmail.com> wrote:
Agree'd in these "smaller" scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide "services" are effected. Perhaps the result of a State led campaign ....topic for another day.
On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson < fergdawgster@gmail.com
wrote:
I am betting that Netsol doesn't need any more "coordination" at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS:
; <<>> DiG 9.7.3 <<>> @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;parsonstech.com. IN NS
;; ANSWER SECTION: parsonstech.com. 172800 IN NS ns2617.ztomy.com. parsonstech.com. 172800 IN NS ns1617.ztomy.com.
;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81
- ferg
On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan <philfagan@gmail.com> wrote:
> I should caveat.....coordinate the "recovery" of. > > > On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth > <brandon@rd.bbc.co.uk>wrote: > >>> Is there an organization that coordinates outages like this amongst the >>> industry? >> >> No, usually they are surprise outages though Anonymous have tried >> coordinating a few >> >> brandon >> > > > > -- > Phil Fagan > Denver, CO > 970-480-7618
-- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
-- Phil Fagan Denver, CO 970-480-7618
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
-- -george william herbert george.herbert@gmail.com
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
Wait, wait. whois doesnt jive with dns. .. Conspiracy Theory Hat On : - Did someone gain access to the COM dispersion zone, or parts thereof? - Did someone figure out how to [ insert theory here ] ? I'm looking at domains that were solidly pointing at ztomy at 2:30AM (that are 'recovered' to other nameservers) that show no "updates" in `whois` records. Curiouser and curiouser. Paul? ---------- Forwarded message ---------- From: jamie rishaw <j@arpa.com> Date: Thu, Jun 20, 2013 at 3:21 PM Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) To: George Herbert <george.herbert@gmail.com> Cc: Jared Mauch <jared@puck.nether.net>, NANOG <nanog@nanog.org> It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s. As far as the logic of the DNS, it is functioning as designed (What's up, Vix!) - There's another aspect of this that caused this situation. Any Alexa or similar people on this list (Goog PR, etc)? I'd love to bulk submit a domain list for some analytics. Contact me off list. On Thu, Jun 20, 2013 at 3:14 PM, George Herbert <george.herbert@gmail.com>wrote:
Poisoning a domain's NS records with localhost will most certainly DOS the domain, yes.
I have not yet seen the source of this; if anyone has a clue where the updates are coming from please post the info.
Is there anything about ztomy.com that has been seen that's supicious as in they might be the origin? This could be them, or could be a joe-job against them. I do not want to point a finger lacking any sort of actual data dump of the poisoning activity...
On Thu, Jun 20, 2013 at 1:02 PM, jamie rishaw <j@arpa.com> wrote:
I'm rechecking realtime ns1620/2620 DNS right now and, looking at the output, I see an odd number of domains (that have changed) with a listed nameserver of "localhost.".
Is this some sort of tactic I'm unaware of?
On Thu, Jun 20, 2013 at 2:57 PM, Jared Mauch <jared@puck.nether.net> wrote:
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime.
I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones.
Is this of value? Does it need to be automated?
- Jared
On Jun 20, 2013, at 3:53 PM, jamie rishaw <j@arpa.com> wrote:
This is most definitely a coordinated and planned attack.
And by 'attack' I mean hijacking of domain names.
I show as of this morning nearly fifty thousand domain names that appear suspicious.
I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET).
Anyone credentialed (credentialed /n/., "I know you or know of you,") wanting data, e-mail me off-list for some TLD goodness.
On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan <philfagan@gmail.com> wrote:
Agree'd in these "smaller" scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide "services" are effected. Perhaps the result of a State led campaign ....topic for another day.
On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson < fergdawgster@gmail.com
wrote:
I am betting that Netsol doesn't need any more "coordination" at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS:
; <<>> DiG 9.7.3 <<>> @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;parsonstech.com. IN NS
;; ANSWER SECTION: parsonstech.com. 172800 IN NS ns2617.ztomy.com. parsonstech.com. 172800 IN NS ns1617.ztomy.com.
;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81
- ferg
On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan <philfagan@gmail.com> wrote:
> I should caveat.....coordinate the "recovery" of. > > > On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth > <brandon@rd.bbc.co.uk>wrote: > >>> Is there an organization that coordinates outages like this amongst the >>> industry? >> >> No, usually they are surprise outages though Anonymous have tried >> coordinating a few >> >> brandon >> > > > > -- > Phil Fagan > Denver, CO > 970-480-7618
-- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
-- Phil Fagan Denver, CO 970-480-7618
-- -george william herbert george.herbert@gmail.com
On 6/20/13, jamie rishaw <j@arpa.com> wrote:
It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s.
https://www.networksolutions.com/blog/2013/06/important-update-for-network-s... -- -JH
On 6/20/2013 1:46 PM, Jimmy Hess wrote:
On 6/20/13, jamie rishaw <j@arpa.com> wrote:
It's not poisoning. They somehow were able to modify the NS records; one would presume, at the registrar/s.
https://www.networksolutions.com/blog/2013/06/important-update-for-network-s...
-- -JH
"small number of Network Solutions customers" They must be staffed with physicists, astronomers, or economists.... I don't know anyone else that would consider "nearly fifty thousand" (from a previous post by Phil Fagan) to be a small number. -- Jeff Shultz
On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said:
"small number of Network Solutions customers"
They must be staffed with physicists, astronomers, or economists.... I don't know anyone else that would consider "nearly fifty thousand" (from a previous post by Phil Fagan) to be a small number.
It's relatively small when you consider there's something like 140M .com's
On 20 June 2013 14:28, <Valdis.Kletnieks@vt.edu> wrote:
On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said:
"small number of Network Solutions customers"
They must be staffed with physicists, astronomers, or economists.... I don't know anyone else that would consider "nearly fifty thousand" (from a previous post by Phil Fagan) to be a small number.
It's relatively small when you consider there's something like 140M .com's
So it's okay to screw over "nearly fifty thousand" customer domains because there are 140M .com's? When talking about inadvertently effecting that many folks I don't think it is appropriate to trivialize the customer impact by calling it small when you're talking about a handful of large websites that aren't somehow magically shared over those 140M .coms. Also it is untrue to limit it to only "the websites" given how many other things folks are likely to be using DNS for... .r'
On Thu, Jun 20, 2013 at 2:49 PM, Randy Bush <randy@psg.com> wrote:
So it's okay to screw over "nearly fifty thousand" customer domains because there are 140M .com's?
luckily, none of the rest of us make mistakes
Ages ago I responded on a Cisco list where the topic was biggest screwup you've made. I posted that I once forgot the implicit deny in an ACL and accidentally blocked all traffic between 4 locations in 2 states for a company I was working for. Downtime was a very brutal 60 seconds. Someone very insightful responded with "anyone who hasn't done similar is lying about the 10 years on their resume". So the real question would be, why wasn't there someone who has already done this in the past working on this zone? ;) -B
I don't think he was saying that at all. Just stating that from a pure numbers standpoint 50k/140mil is a small percentage. OTOH, I agree to your point - Network Solutions definitely downplayed this in their release. Curiously so. Sent from my iPhone On Jun 20, 2013, at 5:42 PM, RijilV <rijilv@riji.lv> wrote:
On 20 June 2013 14:28, <Valdis.Kletnieks@vt.edu> wrote:
On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said:
"small number of Network Solutions customers"
They must be staffed with physicists, astronomers, or economists.... I don't know anyone else that would consider "nearly fifty thousand" (from a previous post by Phil Fagan) to be a small number.
It's relatively small when you consider there's something like 140M .com's So it's okay to screw over "nearly fifty thousand" customer domains because there are 140M .com's? When talking about inadvertently effecting that many folks I don't think it is appropriate to trivialize the customer impact by calling it small when you're talking about a handful of large websites that aren't somehow magically shared over those 140M .coms. Also it is untrue to limit it to only "the websites" given how many other things folks are likely to be using DNS for...
.r'
On Thu, 2013-06-20 at 14:42 -0700, RijilV wrote:
On 20 June 2013 14:28, <Valdis.Kletnieks@vt.edu> wrote:
On Thu, 20 Jun 2013 14:08:18 -0700, Jeff Shultz said:
"small number of Network Solutions customers"
They must be staffed with physicists, astronomers, or economists.... I don't know anyone else that would consider "nearly fifty thousand" (from a previous post by Phil Fagan) to be a small number.
It's relatively small when you consider there's something like 140M .com's
So it's okay to screw over "nearly fifty thousand" customer domains because there are 140M .com's? When talking about inadvertently effecting that many folks I don't think it is appropriate to trivialize the customer impact by calling it small when you're talking about a handful of large websites that aren't somehow magically shared over those 140M .coms. Also it is untrue to limit it to only "the websites" given how many other things folks are likely to be using DNS for...
.r'
I think you are reading it the wrong way. Mr.Kletnieks never said it was okay. He just stated that the numbers were trivial when compared to the rest of potential customers being affected. Be cool, Richard Golodner
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have? randy
At the DNS Servers or service provider level, one can (and I often do) have redundant providers. At the registrar level? ... Not with our current infrastructure, as far as I know how. The Internet: Discovering new SPOF since 1969! George William Herbert Sent from my iPhone On Jun 20, 2013, at 3:28 PM, Randy Bush <randy@psg.com> wrote:
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department.
but none of this is surprising.
and dnssec did not save us. is there anything which could have?
randy
....at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it should....who's the clearing house here. On Thu, Jun 20, 2013 at 4:28 PM, Randy Bush <randy@psg.com> wrote:
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department.
but none of this is surprising.
and dnssec did not save us. is there anything which could have?
randy
-- Phil Fagan Denver, CO 970-480-7618
I, for one, would not be in favor of an authoritarian rule over DNS, or any other Internet system, to "ensure that the state of [the] service[s] is running as it should." I suppose one could view such an authoritarian rule over (sub) systems to be a good thing, as in there is someone to complain to when things don't work, but recent events show that it is also easily abused. I much rather prefer the current cooperative administration of the Internet. Thanks, Fred Reimer On 6/20/13 6:39 PM, "Phil Fagan" <philfagan@gmail.com> wrote:
....at what point is the Internet a piece of infrastructure whereby we actually need a way to watch this thing holistically as it is one system and not just a bunch of inter-jointed systems? Who's job is it to do nothing but ensure that the state of DNS and other services is running as it should....who's the clearing house here.
On Thu, Jun 20, 2013 at 4:28 PM, Randy Bush <randy@psg.com> wrote:
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department.
but none of this is surprising.
and dnssec did not save us. is there anything which could have?
randy
-- Phil Fagan Denver, CO 970-480-7618
On 6/20/13, Randy Bush <randy@psg.com> wrote:
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department. but none of this is surprising. and dnssec did not save us. is there anything which could have?
What's puzzling is the "How the heck did they do that?" The registrar doesn't maintain the .COM database that contains the list of nameservers.... they had to submit changes to all those records. So, why weren't there security controls to make sure that the registrar could not submit changes without appropriate authorization from the Administrative/Tech contact?
randy -- -JH
At 07:28 21/06/2013 +0900, Randy Bush wrote:
netsol screwed up. they screwed up bigtime. they are shoveling kitty litter over it as fast as they can, and they have a professional kitty litter, aka pr, department.
They are too busy adding new revenue: http://www.streetinsider.com/Corporate+News/NetSol+%28NTWK%29+Enters+$10M+Ag... -Hank
At 17:12 20/06/2013 -0500, Richard Golodner wrote:
I think you are reading it the wrong way. Mr.Kletnieks never said it was okay. He just stated that the numbers were trivial when compared to the rest of potential customers being affected. Be cool, Richard Golodner
<sarcasm> and Netsol agrees with you: http://www.networksolutions.com/blog/2013/06/important-update-for-network-so... "a small number of Network Solutions customers were inadvertently affected for up to several hours." </sarcasm> -Hank
I remember when I used to own a small ISP and NetSOL "lost" 1/3 of the domains. Just lost them. And it wasn't a DDOS, it was their screw up. It went on for days -----Original Message----- From: Hank Nussbacher [mailto:hank@efes.iucc.ac.il] Sent: Thursday, June 20, 2013 11:10 PM To: Richard Golodner Cc: nanog@nanog.org Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) At 17:12 20/06/2013 -0500, Richard Golodner wrote:
I think you are reading it the wrong way. Mr.Kletnieks never said it was okay. He just stated that the numbers were trivial when compared to the rest of potential customers being affected. Be cool, Richard Golodner
<sarcasm> and Netsol agrees with you: http://www.networksolutions.com/blog/2013/06/important-update-for-network-so... "a small number of Network Solutions customers were inadvertently affected for up to several hours." </sarcasm> -Hank
On 6/20/13, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote:
It's relatively small when you consider there's something like 140M .com's
Yeah... I'm in agreement about that's probably what is going on... It's relatively small, but absolutely large, and absolute numbers matter. 5 domains is small, 50k is not, even if Netsol has a 100 billion domains. If I had 50,000 fingers; I might think differently. But the definition of a large number doesn't change to people, just because you also have a massive number of that thing. The phrase "a small number" means an absolutely small number, so it seems like a really really misleading if not possibly dishonest PR spin; they could have said "a small proportion" or "a relatively small number", in that case. -- -JH
I think we need a better measure than number of domains (in this case .COM), particularly vs total domains. If it was 100 domains it might seem small, unless that list began with facebook.com, amazon.com, google.com and g*d forbid theworld.com. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
Data on June 20 : .COM. : 108,985,894 unique domains + the tld. -> 234,479 NSEC3/RRSIG records, -> 2,253,400 nameserver entries on 831,088 unique IP addresses. .. ish. -jamie On Fri, Jun 21, 2013 at 5:23 PM, Barry Shein <bzs@world.std.com> wrote:
I think we need a better measure than number of domains (in this case .COM), particularly vs total domains.
If it was 100 domains it might seem small, unless that list began with facebook.com, amazon.com, google.com and g*d forbid theworld.com.
-- -Barry Shein
The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
It's 120M if you add the .COM and the .NET's together, both of which NetSol is responsible for. http://www.verisigninc.com/en_US/products-and-services/domain-name-services/ registry-products/tld-zone-access/index.xhtml Frank -----Original Message----- From: Nicolai [mailto:nicolai-nanog@chocolatine.org] Sent: Friday, June 21, 2013 11:16 AM To: nanog@nanog.org Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) On Thu, Jun 20, 2013 at 05:28:17PM -0400, Valdis.Kletnieks@vt.edu wrote:
It's relatively small when you consider there's something like 140M .com's
Just FWIW, the current size of .com is roughly 109M domains. Someday it will reach 140M but not today. Nicolai
In article <001a01ce6ef9$bf74d4a0$3e5e7de0$@iname.com> you write:
It's 120M if you add the .COM and the .NET's together, both of which NetSol is responsible for. http://www.verisigninc.com/en_US/products-and-services/domain-name-services/ registry-products/tld-zone-access/index.xhtml
In late breaking news, Verisign spun off Network Solutions in 2003, and the two companies have been unrelated for the past decade. These days NetSol is just another registrar. Since 2011 it has been part of web hosting company web.com. R's, John
I know how we got here, but perhaps we can take corporate parentage and how big .com is now to -discuss? What happened with the registry data that caused the outage and what can / should be done about it / to prevent it happening again still seem to me to be operational topics. George William Herbert Sent from my iPhone
Wild speculation: netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service providers fat-fingered their DDOS-averting routing in such a way that netsol DNS traffic arrived at ztomy.com instead of a netsol server. The ztomy.com server would know how to answer the queries... I have no data to base this speculation on. Grüße, Carsten
Hello everyone, I'm new here. +1 to this theory. I've been watching what's happening since 3am Eastern, because a domain of mine (of the many at NetSol) was a victim of this event. -Gabor -----Original Message----- From: Carsten Bormann [mailto:cabo@tzi.org] Sent: Thursday, June 20, 2013 5:11 PM To: NANOG list Subject: Re: This is a coordinated hacking. (Was Re: Need help in flushing DNS) Wild speculation: netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service providers fat-fingered their DDOS-averting routing in such a way that netsol DNS traffic arrived at ztomy.com instead of a netsol server. The ztomy.com server would know how to answer the queries... I have no data to base this speculation on. Grüße, Carsten
No. The ztomy nameservers appeared in this morning's master .COM zonefile as /authoritative/ for the number of domains I mentioned. It is a clear change from just a couple of days ago, when the listed nameservers were nowhere to be seen. I have solid data to back this up, straight from Verisign GRS (Verisign), the authoritative registry for .COM, .NET and others. j On Thu, Jun 20, 2013 at 4:10 PM, Carsten Bormann <cabo@tzi.org> wrote:
Wild speculation:
netsol says this is a human error incurred during DDOS mitigation. ztomy.com is a wild-card DNS provider that seems to use prolexic. Now imagine someone at netsol or its DDOS service providers fat-fingered their DDOS-averting routing in such a way that netsol DNS traffic arrived at ztomy.com instead of a netsol server. The ztomy.com server would know how to answer the queries...
I have no data to base this speculation on.
Grüße, Carsten
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
https://www.networksolutions.com/blog/2013/06/important-update-for-network-s...
Why are they infinitely looping a script on their web server to check for a cookie? Are these people insane?
Not so easy and straightforward to do. You'll find that a lot of the big names out there frequently tweak DNS, which will result in a non-stop stream of "alerts". Andy Andrew Fried andrew.fried@gmail.com On 6/20/13 3:57 PM, Jared Mauch wrote:
It seems there may be a need for some sort of 'dns-health' check out there that can be done in semi-realtime.
I ran a report for someone earlier today on a domain doing an xref against open resolver data searching for valid responses vs invalid ones.
Is this of value? Does it need to be automated?
- Jared
On Jun 20, 2013, at 3:53 PM, jamie rishaw <j@arpa.com> wrote:
This is most definitely a coordinated and planned attack.
And by 'attack' I mean hijacking of domain names.
I show as of this morning nearly fifty thousand domain names that appear suspicious.
I'm tempted to call uscentcom and/or related agencies (which agencies, who the hell knows, as ICE seems to have some sort of authority over domains (nearly two hundred fifty of them as I type this in COM alone and another thirty-some in NET).
Anyone credentialed (credentialed /n/., "I know you or know of you,") wanting data, e-mail me off-list for some TLD goodness.
On Thu, Jun 20, 2013 at 12:29 PM, Phil Fagan <philfagan@gmail.com> wrote:
Agree'd in these "smaller" scenario's I just wonder if in a larger scale scenario, whatever that might look like, if its necessary. Whereby many organizations who provide "services" are effected. Perhaps the result of a State led campaign ....topic for another day.
On Thu, Jun 20, 2013 at 11:25 AM, Paul Ferguson <fergdawgster@gmail.com
wrote:
I am betting that Netsol doesn't need any more "coordination" at the moment -- their phones are probably ringing off-the-hook. There are still ~400 domains still pointing to the ztomy NS:
; <<>> DiG 9.7.3 <<>> @foohost parsonstech.com NS ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49064 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;parsonstech.com. IN NS
;; ANSWER SECTION: parsonstech.com. 172800 IN NS ns2617.ztomy.com. parsonstech.com. 172800 IN NS ns1617.ztomy.com.
;; Query time: 286 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Jun 20 19:16:25 2013 ;; MSG SIZE rcvd: 81
- ferg
On Thu, Jun 20, 2013 at 10:13 AM, Phil Fagan <philfagan@gmail.com> wrote:
I should caveat.....coordinate the "recovery" of.
On Thu, Jun 20, 2013 at 11:10 AM, Brandon Butterworth <brandon@rd.bbc.co.uk>wrote:
> Is there an organization that coordinates outages like this amongst the > industry?
No, usually they are surprise outages though Anonymous have tried coordinating a few
brandon
-- Phil Fagan Denver, CO 970-480-7618
-- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
-- Phil Fagan Denver, CO 970-480-7618
-- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs
participants (23)
-
Andrew Fried -
Barry Shein -
Bryan Irvine -
Carsten Bormann -
David Walker -
Frank Bulk -
Fred Reimer -
Gabor Tokaji -
George Herbert -
Hank Nussbacher -
jamie rishaw -
Jared Mauch -
Jeff Shultz -
Jimmy Hess -
John Levine -
Kain, Rebecca (.) -
Nicolai -
Phil Fagan -
Randy Bush -
Richard Golodner -
RijilV -
Ryan - Lists -
Valdis.Kletnieks＠vt.edu