First, the good news: so far, the NANOG conference has been very valuable and content-rich, covering a lot of issues that need to be discussed. For that, I am grateful. But now, the bad news(?): Maybe it's just me & my paranoia, but do I detect an inkling of "murk spam" going on with some presentations? Because there seems to be a fundamental misunderstanding, either on my part, or the part of certain vendors: I'm hear to discuss ideas & freely share them, and they are here to discuss (it would seem) their products. Sometimes both goals coincide, and that is fine...but... When a vendor at the security BOF starts showing documents that are "company confidential", and trying to whip up a climate of fear, that we should all deploy their product in front of our recursive name servers, i get this funny feeling that I am being "murk spammed". Perhaps that is my own perspective (& paranoia?), but I found the CERT gentleman's call to monitor icmp backscatter on our authoritative nameservers far more informative -- and open. But I was disappointed with two vendors and their presentations: the first had the tactic of saying "DNSSEC is the actual solution" when asked about why their product would be necessary...completely ignoring the fact that their proprietary "interim solution" was by no means the only way to prevent cache poisoning attacks. Indeed, I would daresay it isn't the best, either by a BCP perspective, or a cost analysis perspective. To put a finer point on this, i should say that i found myself discomforted by a presentation suggesting that I should put their proprietary appliances between my recursive name servers & the Net, and I am grateful that Mr. Vixie stood up and said that there are other ways of dealing with the problem. Then there was the gentleman with the DDOS detection/mitigation appliance, who flipped through several graphs, which were intended to show the number of each type of attack. It's unfortunate that there wasn't more time for questions, because I really wanted to ask why "http GET" and "spidering" attacks weren't listen on their graphs...more on that in a second. Fortunately, said vendor had a table at "beer and gear", so I was able to talk with one of their representatives -- and learned that they have just as much trouble with automatic detection of attacks designed to look like a "slashdotting"...which cleared up the mystery as to why it wasn't on the graphs. Because this is a real problem: anybody, with sufficient knowledge & preparation can vandalize _anybody's_ network. Showing me a graph that ping floods happen all the time doesn't impress me -- what would impress me is going over the actual methods, algorithms (and heuristics?) used in these attack mitigation appliances. Because, the "best" attack mitigation appliance vendor would seem to have 100% of their market, and thus, charge exhorbant prices for their product(s). When I brought this up with Mr. Vendor, his first reaction was to point out that the cost was less than a home-grown solution. When I raised the question of open source software to do the same thing, his reaction was to ask: "oh? who's going to write it?" And that right there would seem to be a bit of bravado, perhaps fueled by a misunderstanding of the role that FOSS has played on the Net. Fortunately -- and again, I am grateful for this -- the ISC was represented in the security BOF, presenting the SIE concept...as well as what applications _already exist_ to detect and mitigate various attacks. One demonstration that blew me away: detecting a botnet being set up for a phishing attack...and preventing the attack before it even started. So in conclusion, I'll say this: the last NANOG I attended was NANOG 9 -- and i remember that being a more challenging environment for vendors. Probably the biggest problem discussed back then was head-of-line blocking on a vendor's switches. _That_ is the kind of content that i have found valuable, both on this list, and at a conference. And so: If I weren't so knock-kneed in public venues, I would probably be doing what i would like to call on conference participants to do: if someone gives a presentation that includes their own proprietary black-box "solution", I think the best benefit for NANOG would be to point out alternatives. -Scott p.s. sorry for the long post.
Let me avoid being long winded and just put on my Captain Obvious cape. Avoid magic DDoS appliances, particularly those that require some type of relationship or deposit to be made in advance no matter how "risk free." There is a reason why these vendor presentations aren't meeting your expectations. You're also dead on concerning one's ability to develop and deploy OSS. Human capital is generally your best resource. My two cents, Jeff On Tue, Oct 14, 2008 at 7:08 PM, Scott Doty <scott@sonic.net> wrote:
First, the good news: so far, the NANOG conference has been very valuable and content-rich, covering a lot of issues that need to be discussed. For that, I am grateful.
But now, the bad news(?): Maybe it's just me & my paranoia, but do I detect an inkling of "murk spam" going on with some presentations?
Because there seems to be a fundamental misunderstanding, either on my part, or the part of certain vendors: I'm hear to discuss ideas & freely share them, and they are here to discuss (it would seem) their products. Sometimes both goals coincide, and that is fine...but...
When a vendor at the security BOF starts showing documents that are "company confidential", and trying to whip up a climate of fear, that we should all deploy their product in front of our recursive name servers, i get this funny feeling that I am being "murk spammed".
Perhaps that is my own perspective (& paranoia?), but I found the CERT gentleman's call to monitor icmp backscatter on our authoritative nameservers far more informative -- and open.
But I was disappointed with two vendors and their presentations: the first had the tactic of saying "DNSSEC is the actual solution" when asked about why their product would be necessary...completely ignoring the fact that their proprietary "interim solution" was by no means the only way to prevent cache poisoning attacks. Indeed, I would daresay it isn't the best, either by a BCP perspective, or a cost analysis perspective.
To put a finer point on this, i should say that i found myself discomforted by a presentation suggesting that I should put their proprietary appliances between my recursive name servers & the Net, and I am grateful that Mr. Vixie stood up and said that there are other ways of dealing with the problem.
Then there was the gentleman with the DDOS detection/mitigation appliance, who flipped through several graphs, which were intended to show the number of each type of attack. It's unfortunate that there wasn't more time for questions, because I really wanted to ask why "http GET" and "spidering" attacks weren't listen on their graphs...more on that in a second.
Fortunately, said vendor had a table at "beer and gear", so I was able to talk with one of their representatives -- and learned that they have just as much trouble with automatic detection of attacks designed to look like a "slashdotting"...which cleared up the mystery as to why it wasn't on the graphs.
Because this is a real problem: anybody, with sufficient knowledge & preparation can vandalize _anybody's_ network. Showing me a graph that ping floods happen all the time doesn't impress me -- what would impress me is going over the actual methods, algorithms (and heuristics?) used in these attack mitigation appliances.
Because, the "best" attack mitigation appliance vendor would seem to have 100% of their market, and thus, charge exhorbant prices for their product(s). When I brought this up with Mr. Vendor, his first reaction was to point out that the cost was less than a home-grown solution. When I raised the question of open source software to do the same thing, his reaction was to ask: "oh? who's going to write it?"
And that right there would seem to be a bit of bravado, perhaps fueled by a misunderstanding of the role that FOSS has played on the Net.
Fortunately -- and again, I am grateful for this -- the ISC was represented in the security BOF, presenting the SIE concept...as well as what applications _already exist_ to detect and mitigate various attacks. One demonstration that blew me away: detecting a botnet being set up for a phishing attack...and preventing the attack before it even started.
So in conclusion, I'll say this: the last NANOG I attended was NANOG 9 -- and i remember that being a more challenging environment for vendors. Probably the biggest problem discussed back then was head-of-line blocking on a vendor's switches. _That_ is the kind of content that i have found valuable, both on this list, and at a conference.
And so: If I weren't so knock-kneed in public venues, I would probably be doing what i would like to call on conference participants to do: if someone gives a presentation that includes their own proprietary black-box "solution", I think the best benefit for NANOG would be to point out alternatives.
-Scott p.s. sorry for the long post.
-- Jeffrey Lyon, President Level III Information Systems Technician jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Talk for 4h 45m from the U.S. to Latin America for $10.00: http://www.defensecalling.com
Scott, Given that I both co-moderated the ISP security BOF AND gave a ~9 minute presentation covering *empirical* data and stats of observed attack vectors across 100 ISP networks over 640 days, and shared a slide or two with stats from an infrastructure security survey we've been doing and sharing with the operations community for 4 years now, I take a bit of offense to your comments below. I make a concerted effort to decouple vendor pitches from both the data sets presented and believe I did so effectively. There was open microphone time and you were welcome to share your thoughts. There has been context set with both the data I presented and the survey in previous meetings and NANOGs, it's unfortunate you're unfamiliar with this. Rodney's presentation was one vendor's approach to a very real problem, one that has consumed a significant amount of ISP operations resources over the past 6 months, and you were certainly welcome to comment on that as well - as you note Vixie and others did - and that's a large part of the point of the BOF, IMO. You're welcome to contribute positively in some manner to the next BOF - proactively - or co-moderate if you'd like, but to address the question in the subject line directly - "Am I mistaken", I believe yes. Also, please don't confuse discussion of what happened at beer n gear with what happened at the BOF. -danny
On Oct 14, 2008, at 12:08 PM, Scott Doty wrote:
First, the good news: so far, the NANOG conference has been very valuable and content-rich, covering a lot of issues that need to be discussed. For that, I am grateful.
But now, the bad news(?): Maybe it's just me & my paranoia, but do I detect an inkling of "murk spam" going on with some presentations?
I fully agree with you -- some talks are thinly (or not so thinly) veiled attempts to convince you to buy a vendor's shiny, new solution. There are a large number of reasons for this, and the Program Committee works hard (and I think is doing a great job) to limit the amount of sales pitch but A: there are a limited number of talks and B: many vendors are unable to resist trying to spin their product. I suggest that if you have a topic that you would like to present (and will keep it sales free) you resent it to the PC. I *do* however disagree with you that this happened in the talks to which you are referring...
Because there seems to be a fundamental misunderstanding, either on my part, or the part of certain vendors: I'm hear to discuss ideas & freely share them, and they are here to discuss (it would seem) their products.
Once again, great -- please submit a talk to the PC and they will review it. The PC is always looking for good talks...
Sometimes both goals coincide, and that is fine...but...
When a vendor at the security BOF starts showing documents that are "company confidential", and trying to whip up a climate of fear, that we should all deploy their product in front of our recursive name servers, i get this funny feeling that I am being "murk spammed".
Hmmm... The vendor that you are referring to provides authoritative DNS for many domains (and, at least some of them I view as "important", meaning that I would prefer a correct response!). Yes, I am sure that he would be happy to have you as a customer and, yes, this is feature that differentiates his company, but I did not get the impression AT ALL that he was trying to sell his service, but rather provide better service to his existing customers, even going so far as to provide free devices to people who run large recursive resolvers. This helps both his existing customers (who, yes, will be more likely to continue using him), but, more importantly helps me as an end user feel a little comfortable that the page that I am getting is the correct page...
Perhaps that is my own perspective (& paranoia?), but I found the CERT gentleman's call to monitor icmp backscatter on our authoritative nameservers far more informative -- and open.
But I was disappointed with two vendors and their presentations: the first had the tactic of saying "DNSSEC is the actual solution" when asked about why their product would be necessary...completely ignoring the fact that their proprietary "interim solution" was by no means the only way to prevent cache poisoning attacks.
I may be mistaken, but I didn't get the impression that he believed that his solution was the only one -- he repeatedly pointed out that DNSSEC is the correct solution and this his solution does not solve all of the problems that DNSSEC would -- however, DNSSEC is FAR from being fully deployed.
Indeed, I would daresay it isn't the best, either by a BCP perspective, or a cost analysis perspective.
To put a finer point on this, i should say that i found myself discomforted by a presentation suggesting that I should put their proprietary appliances between my recursive name servers & the Net, and I am grateful that Mr. Vixie stood up and said that there are other ways of dealing with the problem.
Hmmm.. We must have VERY different recollections -- I don't remember him mentioning how much this would cost, other than that he would be give away some to the biggest wins first. Without knowing how much these widgets will be, it is not possibly to do a cost comparison, but don't discount just how expensive engineering time is, and just how hard it is to find competent DNS folks able to deploy something else. I have chatted with many people about the state of their DNS infrastructure -- many people don't care, many people DO care but just don't have the cycles to properly maintain it, many have weird internal politics around them, and many just don't have the knowledge. Some of these are hard to solve, the lack of knowledge is probably the easiest, so I would welcome any how0-to, etc guides that would feel like writing....
Then there was the gentleman with the DDOS detection/mitigation appliance, who flipped through several graphs, which were intended to show the number of each type of attack. It's unfortunate that there wasn't more time for questions, because I really wanted to ask why "http GET" and "spidering" attacks weren't listen on their graphs...more on that in a second.
Hmmm, probably some of this is my fault, I am largely responsible for the agenda -- this was my first tie doing this an I suspect that I tried to fit too many talks into too little time. If there had been more time Danny might have covered their collection methodology (but, I need to warn you that that would probably have involved some information that *could* be construed as "This is what differentiates us" and would have been construed as sales, but whatever...). The information that was presented is part of a very well know report that gets published (but in a more executive format) and he (apparently incorrectly) assumed that the BOF audience would already be aware of how the information is collected and some of the benefits and short comings of their collection methodology. Once agin, probably my fault that he didn't have enough time to go though how the data is collected, but if he had, most of the audience would have bored out of their minds and they already know this and the rest would have felt like they were being sold to...
Fortunately, said vendor had a table at "beer and gear", so I was able to talk with one of their representatives -- and learned that they have just as much trouble with automatic detection of attacks designed to look like a "slashdotting"...which cleared up the mystery as to why it wasn't on the graphs.
Because this is a real problem: anybody, with sufficient knowledge & preparation can vandalize _anybody's_ network. Showing me a graph that ping floods happen all the time doesn't impress me -- what would impress me is going over the actual methods, algorithms (and heuristics?) used in these attack mitigation appliances.
Ok, now I am confused --- you would like the vendor to stand up (in a NANOG presentation) and say: "Here is our widget, look how shiny it is.. Our device is better than $COMPETITOR because we do X, Y, Z, etc. We use the following heuristics <cough> and other vendors don't </ cough>"? To me this sound WAY more like a sales ploy (and, some of the other talks were much closer to this....).
Because, the "best" attack mitigation appliance vendor would seem to have 100% of their market, and thus, charge exhorbant prices for their product(s). When I brought this up with Mr. Vendor, his first reaction was to point out that the cost was less than a home-grown solution.
Yup... Said vendor does have a large market share -- by explaining how they collect the information they would have had to explain just how much of the Internet they instrument, which to me would have felt very salesey...
When I raised the question of open source software to do the same thing, his reaction was to ask: "oh? who's going to write it?" And that right there would seem to be a bit of bravado, perhaps fueled by a misunderstanding of the role that FOSS has played on the Net.
Yes, you can build your own attack mitigation solution (either based on OSS and / or from scratch), but there are limitations. Just saying "use OSS" doesn't make a fully formed solution spring into being, there are *large* investments needed in terms of time, effort, resource, scaling, training, lack of support, etc. While you *can* build a router using just OSS tools[0] there is a reason that most don't...
Fortunately -- and again, I am grateful for this -- the ISC was represented in the security BOF, presenting the SIE concept...as well as what applications _already exist_ to detect and mitigate various attacks. One demonstration that blew me away: detecting a botnet being set up for a phishing attack...and preventing the attack before it even started.
Great, I'm glad you liked that...
So in conclusion, I'll say this: the last NANOG I attended was NANOG 9 -- and i remember that being a more challenging environment for vendors. Probably the biggest problem discussed back then was head-of-line blocking on a vendor's switches. _That_ is the kind of content that i have found valuable, both on this list, and at a conference.
Hmmm, I remember some of these -- and I remember the "Our box does this way better than $OTHER_VENDOR" spin that was always put on this...
And so: If I weren't so knock-kneed in public venues, I would probably be doing what i would like to call on conference participants to do: if someone gives a presentation that includes their own proprietary black-box "solution", I think the best benefit for NANOG would be to point out alternatives.
Next time, please try and overcome your fear (although, I will happily point out that I haven't -- even saying "sorry, only time for 1 more question" gives me sweaty palms, makes me feel queasy, etc. What helps is to remember just how badly most of the other people here speak and that no-one cares) -- other (sane and realistic) solutions are always welcomed...
-Scott p.s. sorry for the long post.
W [0]: OMG, have I just kicked off the "Liinux / BSD as your core router" discussion again?!
On Wed, Oct 15, 2008 at 4:05 PM, Warren Kumari <warren@kumari.net> wrote:
On Oct 14, 2008, at 12:08 PM, Scott Doty wrote:
When a vendor at the security BOF starts showing documents that are "company confidential", and trying to whip up a climate of fear, that we should all deploy their product in front of our recursive name servers, i get this funny feeling that I am being "murk spammed".
Hmmm... The vendor that you are referring to provides authoritative DNS for many domains (and, at least some of them I view as "important", meaning that I would prefer a correct response!). Yes, I am sure that he would be happy to have you as a customer and, yes, this is feature that differentiates his company, but I did not get the impression AT ALL that he was trying to sell his service, but rather provide better service to his existing customers, even going so far as to provide free devices to people who run large recursive resolvers. This helps both his existing customers (who, yes, will be more likely to continue using him), but, more importantly helps me as an end user feel a little comfortable that the page that I am getting is the correct page...
it's probably also worth noting that the person in question has a history of giving away this sort of protection (in other forms) for the DNS system... and innovating as a DNS service provider, both for free (howdy: 4.2.2.1) and for a price.... I'm not sure I'd classify anything he does as a sales pitch in the venue in question. -Chris
"Christopher Morrow" <morrowc.lists@gmail.com> writes:
On Wed, Oct 15, 2008 at 4:05 PM, Warren Kumari <warren@kumari.net> wrote:
On Oct 14, 2008, at 12:08 PM, Scott Doty wrote:
When a vendor at the security BOF starts showing documents that are "company confidential", and trying to whip up a climate of fear, that we should all deploy their product in front of our recursive name servers, i get this funny feeling that I am being "murk spammed".
... I did not get the impression AT ALL that he was trying to sell his service, but rather provide better service to his existing customers, even going so far as to provide free devices to people who run large recursive resolvers. ...
i've heard the following concerns about this free device expressed to me. first, its value-add is its proprietary relationship to one dns authority (ultradns), so if neustar deploys a lot of them it will create third party incentive among domainholders to move their authority service to neustar. so while other commercial authority dns vendors (such as nominum or microsoft) might be willing to license this proprietary technology from neustar and we can all assume that there are commercial terms under which neustar would do this, we can also expect that domainholders who prefer to self-host using f/l/oss (bind, nsd, tinydns, powerdns, etc) won't have that option. rodney said it was necessary that neustar not have to wait for the standards community before deploying this service, but noone asked him why he hasn't open-sourced his solution so that other dns authority suppliers can also benefit from the recursive-dns frontend boxes he's giving away. i know that neustar is in the business of selling outsourced authority dns, so i understood scott doty's comments as referring to the pressure a large deployment of free recursive-dns frontend boxes will put on anyone who isn't a neustar customer to please become a neustar customer so that their zones will be safer. second, there's no real possibility that someone who deploys a free neustar box inline/upstream of their recursive dns server would also deploy a second one if anyone else with a proprietary solution wanted to follow neustar's example. rodney did not say whether the front-end boxes were user programmable or whether he planned to make it possible for competitors of neustar to embed their solutions in this free box. rodney also did not say how many boxes would be available for free before neustar would have to start charging for them, nor whether the price at that point would represent cost recovery or also be a profit center for neustar. these questions also appear (to me) to be implied by scott doty's original question. now for my own concerns.
it's probably also worth noting that the person in question has a history of giving away this sort of protection (in other forms) for the DNS system... and innovating as a DNS service provider, both for free (howdy: 4.2.2.1) and for a price.... I'm not sure I'd classify anything he does as a sales pitch in the venue in question.
in spite of my great admiration for rodney's lifetime of contribution, i do not see any natural consequence toward dnssec from this dns frontend giveaway. i have total confidence that the solution will work, and reasonable confidence that it will indirectly improve neustar's revenue outlook, but no confidence that anyone who wasn't planning to deploy dnssec in their product or network will, as a result of rodney's work, decide to deploy dnssec. far better in my opinion would be for rodney to sign all the zone he carries (keeping the keys he has to generate in escrow to be surrendered to the domainholders upon demand with a reasonable escrow and transfer fee), and to either start his own DLV registry or to offer free secondary service to ISC's DLV registry, and to submit all his customer keys to whichever DLV registry he decided upon. anyone running BIND 9.3.0 (not 9.6.0 as was mentioned -- we're talking about old and somewhat stable code here) can just speak DLV directly. anyone who can and wants to upgrade to BIND with its DLV support can do that. anyone else could install a free recursive dns frontend box from neustar that would do inline DLV. but there's a pure software-only solution that would work. (noting that in rodney's preso he spoke of the many folks who have never upgraded their nameservers, are still running BIND4, etc, but for the larger recursive dns operators this isn't how they work and they can deploy new code, and it would be very easy for nominum-ans and nlnetlabs-unbound to implement DLV, which is unencumbered even though never subject to IETF delays.) it's easy to assume that my worry about this is as someone in the authority dns business whose customers (the vast majority of whom pay nothing), who stands to lose market share when rodney starts pushing his boxes into the field. but since i've been giving away free shovels to people who mostly want to buy holes, and rodney sells holes, i think that ship has already sailed. the baser knee-jerk reaction underlying my discomfort is that isc's mission statement (front and center at www.isc.org) values the autonomy of the internet's participants. dnssec does that. a dnssec-based solution, or a dnssec-leveraging solution, does that. rodney's plan doesn't do that. i'd welcome raw data about dns poisonining events, too. we're scanning the hell out of all the open recursives, and we're not finding much poison, in spite of all the "please stop querying our nameserver!" complaints we incite. so while i want dnssec, i'm pretty comfortable with 16-bit port randomization as a stopgap. rodney's free inline recursive dns frontend could just do 16-bit port randomization if all we want is an until-there-is-dnssec stopgap. -- Paul Vixie
Scott, On Oct 14, 2008, at 9:08 AM, Scott Doty wrote:
First, the good news: so far, the NANOG conference has been very valuable and content-rich, covering a lot of issues that need to be discussed. For that, I am grateful.
Thank you. We worked hard to make it valuable.
But now, the bad news(?): Maybe it's just me & my paranoia, but do I detect an inkling of "murk spam" going on with some presentations?
Not sure what you mean by "murk spam". Thats a term that died years ago. And it really related to people claiming that spam was "in compliance with federal laws". But I think I can guess your intentions from the tone of your email, so let me try and respond.
Because there seems to be a fundamental misunderstanding, either on my part, or the part of certain vendors: I'm hear to discuss ideas & freely share them, and they are here to discuss (it would seem) their products. Sometimes both goals coincide, and that is fine...but...
When a vendor at the security BOF starts showing documents that are "company confidential", and trying to whip up a climate of fear, that we should all deploy their product in front of our recursive name servers, i get this funny feeling that I am being "murk spammed".
Well, that's interesting. I see your last NANOG was 9, in February of 1997. So "Welcome back!". We're glad to have you here in person. Things have changed slightly since then. NSP-SEC never existed in 1997. It really came about in the early 2000's where it was developed as a forum for actual operators to share views and thoughts, generally in real time, to help the 'net in general survive disruption, malicious or otherwise. It has really worked pretty well, so if you qualify, I'd encourage you to get involved. See http://puck.nether.net/mailman/listinfo/nsp-security for info. The NSP-SEC bof at NANOG is not quite the same environment as the NSP- SEC mailing list, but it generally includes the same people, plus others from the operations community who take the effort to attend NANOG, and so are sort of self-selected as being "one of the operators" with an already working amount of clue about the subjects that are being discussed. Additionally, the concept of a "trusted environment" still sorta applies. You may not have realized it, but unlike all other sessions at NANOG, the slides are not published, they are not available online, and the session is not broadcast. So "Confidential" was there to remind folks in the BoF that this was a non-public (for a skewed version of public) presentation. Having explained that bit of history which gives you a general background, let me deal with some specifics.
Perhaps that is my own perspective (& paranoia?), but I found the CERT gentleman's call to monitor icmp backscatter on our authoritative nameservers far more informative -- and open.
I don't think anyone from CERT presented. Perhaps you meant Barry Green from Juniper's CERT team? Another "vendor"? Well, as you'll see further on, not really. In this context, like everyone else who presented, he was there as an operator, sharing knowledge and experience. But I digress...
But I was disappointed with two vendors and their presentations: the first had the tactic of saying "DNSSEC is the actual solution" when asked about why their product would be necessary...completely ignoring the fact that their proprietary "interim solution" was by no means the only way to prevent cache poisoning attacks. Indeed, I would daresay it isn't the best, either by a BCP perspective, or a cost analysis perspective.
While we may disagree on your last claim (and I actually have a few years of experience to help me argue my point), I specifically said there were a) solutions that solved part of the problem (switching to TCP, detecting and blocking cache poisoning attacks) and b) the right solutions like DLV and DNSSEC that will take some time to be deployed. And I then made sure everyone heard me when I said that we need to find an interim solution that can be deployed *now*, until DNSSEC exists in a useful footprint. I ignore *nothing*. If you have another solution that solves the same problems that has running code now, please share it with all of us. Remember, it has to scale, it has to solve all of the problems, and it has to be implementable across a range of levels of clue.
To put a finer point on this, i should say that i found myself discomforted by a presentation suggesting that I should put their proprietary appliances between my recursive name servers & the Net, and I am grateful that Mr. Vixie stood up and said that there are other ways of dealing with the problem.
Indeed. Read further.
Fortunately, said vendor had a table at "beer and gear", so I was able to talk with one of their representatives -- and learned that they have just as much trouble with automatic detection of attacks designed to look like a "slashdotting"...which cleared up the mystery as to why it wasn't on the graphs.
Because this is a real problem: anybody, with sufficient knowledge & preparation can vandalize _anybody's_ network. Showing me a graph that ping floods happen all the time doesn't impress me -- what would impress me is going over the actual methods, algorithms (and heuristics?) used in these attack mitigation appliances.
Because, the "best" attack mitigation appliance vendor would seem to have 100% of their market, and thus, charge exhorbant prices for their product(s). When I brought this up with Mr. Vendor, his first reaction was to point out that the cost was less than a home-grown solution. When I raised the question of open source software to do the same thing, his reaction was to ask: "oh? who's going to write it?"
And that right there would seem to be a bit of bravado, perhaps fueled by a misunderstanding of the role that FOSS has played on the Net.
Fortunately -- and again, I am grateful for this -- the ISC was represented in the security BOF, presenting the SIE concept...as well as what applications _already exist_ to detect and mitigate various attacks. One demonstration that blew me away: detecting a botnet being set up for a phishing attack...and preventing the attack before it even started.
Cool. I'm glad you saw value from that "vendor". Seriously. SIE is good stuff.
So in conclusion, I'll say this: the last NANOG I attended was NANOG 9 -- and i remember that being a more challenging environment for vendors. Probably the biggest problem discussed back then was head-of-line blocking on a vendor's switches. _That_ is the kind of content that i have found valuable, both on this list, and at a conference.
And so: If I weren't so knock-kneed in public venues, I would probably be doing what i would like to call on conference participants to do: if someone gives a presentation that includes their own proprietary black-box "solution", I think the best benefit for NANOG would be to point out alternatives.
*I* was the "vendor" at the security BOF you took aim at. Except I am not a vendor in this environment. I am an operator. Just like ISC (Vixie) and McPherson (Arbor) and Greene (Juniper) etc. We are there as operators and *none* of us was selling *anything. We were describing issues that we currently are facing as operators, and solutions we have developed. You're not alone amongst "newcomers" in missing the point, so don't be hard on yourself ;-). In my case, *nothing* was being sold, other than *a* solution, which I am actually *giving* away to networks that matter in solving the probelm, and picking up the costs myself. I assume you missed that. And the reason I was doing that with a *proprietary* solution was because the open source solution is *not yet ready* for prime time, mainly because it (they) have not solved the wide implementation challenge. And *we* need to find a solution today while the open source (and best solution) gets rolled out effectively. Paul (also a "vendor" in the same vein, but an operator in the BoF forum) answered the question of whether there was another solution by saying "there is in Bind 9.6" - his product, which was released a couple of weeks ago. I referred to it in my presentation, as a solution, along with DNSSEC. It's called DLV. Unfortunately, and Paul admits it, there are challenges to widespread adoption. It works, but there is no business case that makes it easy to roll out. And therein lies the challenge. My customers need it today. And if it isn't out there in wide use, *it doesn't solve the problem*. So I am solving that by picking up the tab myself, and being reimbursed by the people I am a vendor to, my customers. And they're happy to pay for it. None of them were at the bof. Well, not strictly true, but not in numbers to matter. But hopefully you get the point. And you now understand that in the BoF we are all working to try and *solve* problems, not sell products. I'm sorry you failed to grok that difference. Finally, despite your knocking knees, you should have stood up and questioned anything you heard, or misunderstood. Then you would have had a better experience of the bof. As a member of the Program Committee and coincidentally the host of this NANOG, I'm sorry we didn't do a better job. We're trying to get better. I think that this was one of the best NANOGs we've ever had. But I'm biased, especially this time ;-). As an aside, since you were last at a NANOG, we now have Beer 'n Gear, where Vendors have the opportunity to show off their wares, and in exchange they support and underwrite some of the costs of what is a pretty slick conference. I'm not sure why you believe that the vendor pitching his/her products at Beer 'n Gear is in some way violating the sacred rule against talking about a product. The B&G specifically provides the controlled environment and tradeoff. And *most* operators appreciate it, and make really good use of the opportunity to learn about new products that actually matter in such a useful environment. In one place we get to talk to actual engineers, about their products, together with 500 fellow operators who ask questions we may not even know we should ask. If you have any other questions about my presentation, or the program, please feel free to ask directly.
-Scott p.s. sorry for the long post.
Ditto for the response. But I have to assume you were not the only one who may have missed key points. Thanks for coming back. Hopefully we'll see you in the Dominican Republic next January.
I do seem to have put my foot in my mouth. I apologize for any offense my comments made, as well as any misunderstanding on my part. I see the note to take this discussion to nanog-futures, so I'll reply further there. And the Security BOF was very good, I was thankful to have been there and hear the discussion. Next time I'll use the microphone. Thank you, -Scott
On Tue, 14 Oct 2008, Scott Doty wrote:
First, the good news: so far, the NANOG conference has been very valuable and content-rich, covering a lot of issues that need to be discussed. For that, I am grateful.
But now, the bad news(?): Maybe it's just me & my paranoia, but do I detect an inkling of "murk spam" going on with some presentations?
Judging by the email after this, the 'vendor' involves Rodney Joffe and probably UltraDNS. My opinion: Yes, you are being 'murk spammed' Joffe and company represent what Professor Dan Bernstein (DJBDNS) calls the 'Bind Company'. I think a better term is the 'BIND Cartel', since it is a collection of companies and individuals. Joffe, Vixie, John Levine et al own or direct Whitehat.com, a spammer. Remember Sanford Wallace? Wallace sent spam and offered anti-spam services; that was a non-starter for many. Vixie, Joffe, Levine et al just stole Wallace's business plan and false-teamed themselves as anti-spammers. What they were really doing was sending spam, and using the MAPS blacklist to detect and interfere with their competitors, and using the credentials with the anti-spam commun and inside information to avoid spam-traps. See http://www.iadl.org/maps/maps-story.html Joffe/Centergate/Bill Manning was the founder of UltraDNS. Manning is also connected to Vixie through PAIX, and to ISC employee Susan Woolf through ISI. Vixie, Conrad, Manning, Woodcock, Curran, Plzak, Ed Lewis, etc all worked together at ARIN, and have had 22 ARIN employees attend NANOG, including the ARIN executive secretary. ARIN is giving NANOG $50,000 checks, even though the Board members have undisclosed conflicts of interest. ARIN resource analysts have (and probably are now) attending NANOG. The resource analysts are the guys who make allocation decisions, so getting chummy with NANOG people is a conflict of interest in the making. So far, I've discovered two cases where ARIN has made allocations in 2 hours. Have they done this before? The answer is yes. The previous scam was AXFR-clarify draft. The draft was presented by the BIND Cartel as not changing the DNS protocol, but in fact did change the protocol. When Dr. Bernstein discovered this, and reported it, Bernstein's email was disrupted and censored. There are other scams that I'm writing up, but this gives you some inkling of what's going on now and what's gone on before. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
Vixie, Conrad, Manning, Woodcock, Curran, Plzak, Ed Lewis, etc all worked together at ARIN, and have had 22 ARIN employees attend NANOG, including the ARIN executive secretary. ARIN is giving NANOG $50,000 checks, even though the Board members have undisclosed conflicts of interest. ARIN resource analysts have (and probably are now) attending NANOG. The resource analysts are the guys who make allocation decisions, so getting chummy with NANOG people is a conflict of interest in the making. So far, I've discovered two cases where ARIN has made allocations in 2 hours.
Didn't you get banned temporarily from this list, then banned for life + 5 years, your children and grandchildren also banned for their lives + 5 years once before for all this? Tuc/TBOH
[snip] http://www.gweep.net/~crimson/Don't_Feed_The_Trolls.mp3 -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
Vixie, Conrad, Manning, Woodcock, Curran, Plzak, Ed Lewis, etc all worked together at ARIN, and have had 22 ARIN employees attend NANOG, including the ARIN executive secretary. ARIN is giving NANOG $50,000 checks, even though the Board members have undisclosed conflicts of interest. ARIN resource analysts have (and probably are now) attending NANOG. The resource analysts are the guys who make allocation decisions, so getting chummy with NANOG people is a conflict of interest in the making. So far, I've discovered two cases where ARIN has made allocations in 2 hours.
Didn't you get banned temporarily from this list, then banned for life + 5 years, your children and grandchildren also banned for their lives + 5 years once before for all this?
I was never temporarilly banned. I was banned in 2000 so that I couldn't gloat that the CFAA applied to ISPs. See http://www.iadl.org/nanog/nanog-story.html Looks like someone messed up. ;-) --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
Vixie, Conrad, Manning, Woodcock, Curran, Plzak, Ed Lewis, etc all worked together at ARIN, and have had 22 ARIN employees attend NANOG, including the ARIN executive secretary. ARIN is giving NANOG $50,000 checks, even though the Board members have undisclosed conflicts of interest. ARIN resource analysts have (and probably are now) attending NANOG. The resource analysts are the guys who make allocation decisions, so getting chummy with NANOG people is a conflict of interest in the making. So far, I've discovered two cases where ARIN has made allocations in 2 hours.
Didn't you get banned temporarily from this list, then banned for life + 5 years, your children and grandchildren also banned for their lives + 5 years once before for all this?
I was never temporarilly banned. I was banned in 2000 so that I couldn't gloat that the CFAA applied to ISPs. See http://www.iadl.org/nanog/nanog-story.html
Looks like someone messed up. ;-)
Well, yes and no........... I actually was thinking of the ARIN list that you had the temporary ban on : http://lists.arin.net/pipermail/arin-discuss/2008-February/000897.html and then the permanent ban : http://lists.arin.net/pipermail/arin-discuss/2008-June/001058.html as for banning from NANOG, there is a message, purportedly from you : http://lists.arin.net/pipermail/arin-discuss/2008-February/000890.html contains "So Harris banned me from NANOG." . Not sure if thats the meeting, the NANOG list, or one of the NANOG/Merit other lists. Also, in : http://www.iadl.org/nanog/nanog-story.html I see "So, effective May 4 2005, Harris again banned Anderson. Although the new "reformed" rules require a limit of 6 months, Anderson remains banned as of April 16th, 2006. It seems permanent." but I think that refers to another NANOG group, dnsop. Tuc/TBOH
Since you so many facts wrong, a response is necessary. On Wed, 15 Oct 2008, Tuc at T-B-O-H.NET wrote:
I actually was thinking of the ARIN list that you had the temporary ban on :
http://lists.arin.net/pipermail/arin-discuss/2008-February/000897.html
I don't have a page on this because it is currently the subject of litigation. However, since you brought it up, I have to defend myself. As my attorney pointed out to ARIN, this ban was based on a fabrication by ARIN. Among other things, ARIN also threatened to make false claims that I sent spam to the ARIN lists. ARIN has also published the communications between my lawyer and ARIN's lawyer, which is very irregular. This particular ban disrupted and ended a discussion about the lack of quorum in elections that brought Bradner, Curran, Howard, Manning, Vixie and Woodcock to the ARIN Board of Directors. Notices were recently sent (certified mail) to all six ARIN Board Members informing them of the lack of quorum in their election and that they are not authorized to act as Board Members. The ban has prevented other voting members from learning these facts. (Manning and Woodcock have so far refused to accept the certified letters) In the meantime, the Board (Vixie et al) have tried to alter the ARIN bylaws to change the quorum requirements. But because the Board members voting on these changes weren't validly elected, their modifications are also invalid. Board members (e.g. Ray Plzak) also have a duty to object to improper acts; such as allowing invalidly elected board members to act as board members. This might be a tad legal, but NANOG has had seminars on internet law, so some basic business law is just a part of any operator's skillset. Everyone should know that membership rights to democratic participation are intangible property, and should know that taking property (including membership rights to democratic participation) on false pretenses is fraud. Threatening to take such property by force or fear is extortion. I encourage everyone to read http://www.usdoj.gov/usao/eousa/foia_reading_room/usam/title9/crm02403.htm particularly United States v. Teamsters Local 560.
and then the permanent ban :
http://lists.arin.net/pipermail/arin-discuss/2008-June/001058.html
Also based on fabrication, by a non-neutral body of Vixie/NANOG cronies. My attorney is preparing a response to this, so I can't comment very much about it.
as for banning from NANOG, there is a message, purportedly from you :
http://lists.arin.net/pipermail/arin-discuss/2008-February/000890.html
contains "So Harris banned me from NANOG." . Not sure if thats the meeting, the NANOG list, or one of the NANOG/Merit other lists.
The list, I don't know if this applies to meetings. However, Jeremy Porter threatened 'Dead Anderson' with "Maybe with any luck he'll show up at the next nanog meeting and be suprised in a dark alley." So I haven't attended any meetings where there will be NANOG people present without much security. It is interesing to note however that the NANOG-affiliated ARIN AUP committee claimed in June 2008 that this threat wasn't made.
Also, in : http://www.iadl.org/nanog/nanog-story.html
I see "So, effective May 4 2005, Harris again banned Anderson. Although the new "reformed" rules require a limit of 6 months, Anderson remains banned as of April 16th, 2006. It seems permanent."
This refers to the NANOG reform movement in 2005. If that's really not clear from the page, I'll edit the page for clarity.
but I think that refers to another NANOG group, dnsop.
DNSOP isn't a NANOG group. Its an IETF group. http://www.av8.net/IETF-watch/IESG/IESG-PR-discussion.html In fact, you might find everything on http://www.av8.net/IETF-watch interesting. Hope that clears up the facts. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
On Thu, 16 Oct 2008, Dean Anderson wrote:
contains "So Harris banned me from NANOG." . Not sure if thats the meeting, the NANOG list, or one of the NANOG/Merit other lists.
The list, I don't know if this applies to meetings.
The Jan 2000 ban also stopped my participation in RADB. Mail to all of merit.edu was affected. I think this was to prevent me from emailing Harris' boss---she hung up on me in a phone call when I asked who her boss was. --Dean -- Av8 Internet Prepared to pay a premium for better service? www.av8.net faster, more reliable, better service 617 344 9000
On Thu, 16 Oct 2008, Dean Anderson wrote:
contains "So Harris banned me from NANOG." . Not sure if thats the meeting, the NANOG list, or one of the NANOG/Merit other lists.
The list, I don't know if this applies to meetings.
The Jan 2000 ban also stopped my participation in RADB. Mail to all of merit.edu was affected. I think this was to prevent me from emailing Harris' boss---she hung up on me in a phone call when I asked who her boss was.
--Dean Well Dean, considering that you are a pathological liar (as well as an IP theif), I can certainly understand people being defensive around you. As to your non-participation in NANOG and related flora, remember that the Innernets are a *cooperative* being. Encouraging
On Thu, Oct 16, 2008 at 10:32 PM, Dean Anderson <dean@av8.com> wrote: the participation of loose screws, cerebral derelicts, and other assorted net.trash such as yourself can only *improve* the quality of the venture. //Alif -- "Never belong to any party, always oppose privileged classes and public plunderers, never lack sympathy with the poor, always remain devoted to the public welfare, never be satisfied with merely printing news, always be drastically independent, never be afraid to attack wrong, whether by predatory plutocracy or predatory poverty." Joseph Pulitzer 1907 Speech
On Thu, Oct 16, 2008 at 10:31:21PM -0400, Dean Anderson wrote:
(Manning and Woodcock have so far refused to accept the certified letters)
and then sometime in the past 5 days, you posted a comment to DoC here; http://www.ntia.doc.gov/dns/dnssec.html that states: " Bill Manning refused to accept certified mail" If I may... I am in possesion of your certified letter -AND- the signed acknowledgement that you received notice that I have taken posession of said certified mail. please get your facts straight, esp. when making formal replies to government inqueries. it can only strengthen your case if you tell the truth. --bill
On Tue, 25 Nov 2008 bmanning@vacation.karoshi.com wrote: > If I may... I am in possesion of your certified letter > -AND- the signed acknowledgement that you received notice > that I have taken posession of said certified mail. > > please get your facts straight, esp. when making formal > replies to government inqueries. it can only strengthen > your case if you tell the truth. Equally but differently untruthful in my case. Myself, I don't sit around my house all day, breathlessly anticipating a new missive from Dean. -Bill
On Tue, Nov 25, 2008 at 08:56:43AM -0800, Bill Woodcock wrote:
On Tue, 25 Nov 2008 bmanning@vacation.karoshi.com wrote: > If I may... I am in possesion of your certified letter > -AND- the signed acknowledgement that you received notice > that I have taken posession of said certified mail. > > please get your facts straight, esp. when making formal > replies to government inqueries. it can only strengthen > your case if you tell the truth.
Equally but differently untruthful in my case. Myself, I don't sit around my house all day, breathlessly anticipating a new missive from Dean.
-Bill
then what, pray tell, do you do to while away the hours? knit? route IP datagrams? carve elaborate totems from ancient redwoods? myself, I have taken up Portugese... such an expressive language. my instruction in the email above was derived from reading the submitted comments to the DoC/NOI on securing the DNS. telling us lies is one thing, factual mis-statements to the goverment is something else and i fail to see how doing so helps ones cause. in the event that anyone has doubts, I will be glad to scan and post the evidence. Dean, care to amend your statements? --bill
Can anyone explain why we are being exposed to this? From either side? bmanning@vacation.karoshi.com wrote:
On Tue, Nov 25, 2008 at 08:56:43AM -0800, Bill Woodcock wrote:
On Tue, 25 Nov 2008 bmanning@vacation.karoshi.com wrote: > If I may... I am in possesion of your certified letter > -AND- the signed acknowledgement that you received notice > that I have taken posession of said certified mail. > > please get your facts straight, esp. when making formal > replies to government inqueries. it can only strengthen > your case if you tell the truth.
Equally but differently untruthful in my case. Myself, I don't sit around my house all day, breathlessly anticipating a new missive from Dean.
-Bill
then what, pray tell, do you do to while away the hours? knit? route IP datagrams? carve elaborate totems from ancient redwoods? myself, I have taken up Portugese... such an expressive language.
my instruction in the email above was derived from reading the submitted comments to the DoC/NOI on securing the DNS. telling us lies is one thing, factual mis-statements to the goverment is something else and i fail to see how doing so helps ones cause.
in the event that anyone has doubts, I will be glad to scan and post the evidence. Dean, care to amend your statements?
--bill
-- Jeff Shultz
the bills having a war with dean. how droll. can you maybe take it elsewhere? randy
A reminder to all list members that: 1. DNS related questions should usually be sent to more specific lists such as DNS operations: http://lists.oarci.net/mailman/listinfo/dns-operations 2. Discussion regarding the NANOG organisation and political issues surrounding it are off-topic for the main list and must only occur on the nanog-futures list http://mailman.nanog.org/mailman/listinfo/nanog-futures Simon Lyall NANOG Mailing List Committee -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
participants (17)
-
Andrew D Kirch -
Bill Woodcock -
bmanning＠vacation.karoshi.com -
Christopher Morrow -
Danny McPherson -
Dean Anderson -
J.A. Terranson -
Jeff Shultz -
Jeffrey Lyon -
Joe Provo -
Paul Vixie -
Randy Bush -
Rodney Joffe -
Scott Doty -
Simon Lyall -
Tuc at T-B-O-H.NET -
Warren Kumari