Domain Name System protection
Hi, We are trying to extend our DNS service system in near future. In current stage, it consist of 2 SUN FIRE Server with Solaris8 and BIND9 installed. Each server is configured with a IP address which is known to our customers. The DNS server is set up as Cache Server because it only servers our customers to lookup domain names. We noticed there is continous name resolution requests from IP address outside of our address pool and also there is requests not conforming to DNS documents ( like those from 10/8, 192.168/16 or something for microsoft proxy server name). We think these request waste our resource and we don't want these system stable, secure and high performance. The amount of DNS requests processed in past week is about 0.8Billion. What I'm not sure with designing new Cache Server farm is : 1. Is that really required to protect DNS server by firewall? How does those ISPs, e.g. AT&T, Sprint,mae their DNS system highly available? Could we do that by filtering traffic besides port destinated to port 53? 2. How could we extend our server farm by adding new servers while announcing the same IP addresses to our customers? 3. Is there any evaluation result of DNS server software? e.g. performance, resource required, stability, security etc.? 4. Which hardware/OS platform is better for DNS service? 5. Is that possible to filter those requests not conforming to DNS documents? Each word will be highly appreciated! Joe __________________________________________________ Do You Yahoo!? Download the latest ringtones, games, and more! http://sg.mobile.yahoo.com
> 1. Is that really required to protect DNS server by > firewall? Yes, it's a very, very good idea to do so. > How does those ISPs, e.g. AT&T, Sprint,mae > their DNS system highly available? By protecting it with a firewall. :-) > Could we do that > by filtering traffic besides port destinated to port > 53? Yes, exactly. And possibly also by creating two different pools of DNS servers: one pool which is accessible from everywhere, and which is authoritative for your and your customers' domains; the other which is accessible only to your customers, and which performs recursive resolution on their behalf. > 2. How could we extend our server farm by adding new > servers while announcing the same IP addresses to our > customers? By doing exactly that. Sharing one IP address across many servers is called "anycast" and is standard practice for DNS service provision. > 4. Which hardware/OS platform is better for DNS > service? The combination you've got, Solaris 8 and BIND 9, is fine. Some people would use other DNS server software, and some people would use FreeBSD or NetBSD, but you've got a very mainstream combination. We run Solaris 9 and Bind 9 on about forty DNS servers, for instance. > 5. Is that possible to filter those requests not > conforming to DNS documents? That's a lot tougher. Are you asking whether it's possible to have an application-layer firewall screen out mal-formed requests _before they get to your DNS server_? That's theoretically possible, but I don't know of anyone who does it. Once the queries have arrived at the DNS server, the DNS server application may be able to filter them in different ways, and discard different classes of queries with different kinds of logging or notification. -Bill
On 16 Aug 2004, at 01:18, Bill Woodcock wrote:
2. How could we extend our server farm by adding new servers while announcing the same IP addresses to our customers?
By doing exactly that. Sharing one IP address across many servers is called "anycast" and is standard practice for DNS service provision.
See, for example: http://www.isc.org/pubs/tn/isc-tn-2004-1.html http://www.isc.org/pubs/tn/isc-tn-2004-1.txt http://www.isc.org/pubs/pres/USENIX/2004/usenix-isc-dns-dist.pdf http://www.isc.org/pubs/pres/USENIX/2004/usenix-dns-dist-pres.pdf Joe
Hi, thanks for your help on my question. After reading carefully those comments, I reach the following conclusion: 1. ISPs use firewall to protect their DNS server; 2. ACL on router may be a good solution for protecting DNS servers, the policy could be "only pass those packets, whose originate from incustomers' IP address blocks and destinate to UDP port 53 of DNS server"; 3. Currently, it maybe a little difficult for firewall to filter DNS requests not conforming to DNS document; but, Nominum's product could; 4. Anycast is the most scalable and standard solution for dispersed DNS server farm, while layer-4 switch could deal could do with centralized server farm; 5. 'bogon'in BIND configuration could be used to filter requests from RFC1918 address; 6. Firewall may become bottleneck of DNS server farm in situation of DoS attack or situation of high session rate; 7. It's good solution to divide DNS servers into two groups, one for recursive lookup the other for no-recuresive; 8. BIND should be configured carefully and there is BIND secure template to follow Have I missed something? And, I got another two questions: a) If firewall is used to protect DNS server farm, could it do more than router's ACL while reaching the same performance-cost ratio ? which one is usually chosen by those ISPs having big customer numbers? (we noticed DNS requests from our customers keep increase in past months) b) Is there any public available performance evaluation on Nominum's product? Any of your words will be highly appreciated. Joe __________________________________________________ Do You Yahoo!? Download the latest ringtones, games, and more! http://sg.mobile.yahoo.com
On Aug 16, 2004, at 11:03 PM, Joe Shen wrote:
3. Currently, it maybe a little difficult for firewall to filter DNS requests not conforming to DNS document; but, Nominum's product could;
If you are looking for firewall-esque devices to protected name servers, look into Riverhead (purchased by cisco recently). -- TTFN, patrick
1. ISPs use firewall to protect their DNS server;
some do, some don't
4. Anycast is the most scalable and standard solution for dispersed DNS server farm, while layer-4 switch could deal could do with centralized server farm;
its not a standard.
5. 'bogon'in BIND configuration could be used to filter requests from RFC1918 address;
this should be pushed to the router. don't waste CPU cycles on the Nameserver.
6. Firewall may become bottleneck of DNS server farm in situation of DoS attack or situation of high session rate;
yes
7. It's good solution to divide DNS servers into two groups, one for recursive lookup the other for no-recuresive;
yes
8. BIND should be configured carefully and there is BIND secure template to follow
altho the template will not meet every case.
a) If firewall is used to protect DNS server farm, could it do more than router's ACL while reaching the same performance-cost ratio ? which one is usually chosen by those ISPs having big customer numbers? (we noticed DNS requests from our customers keep increase in past months)
general rule - drop undesired traffic as far upstream as possible.
b) Is there any public available performance evaluation on Nominum's product?
you should check w/ the Nominum staff on any performance evaluations.
Any of your words will be highly appreciated.
Joe
__________________________________________________ Do You Yahoo!? Download the latest ringtones, games, and more! http://sg.mobile.yahoo.com
On Aug 16, 2004, at 11:57 PM, bmanning@vacation.karoshi.com wrote:
4. Anycast is the most scalable and standard solution for dispersed DNS server farm, while layer-4 switch could deal could do with centralized server farm;
its not a standard.
<ftp://ftp.rfc-editor.org/in-notes/rfc1546.txt>, aka _Host Anycasting Service_ Looks pretty standard to me. Not to mention being used in production on several major networks for over half a decade (read: "forever") on the Internet is pretty well tested technology. -- TTFN, patrick P.S. That would be "it's". =)
4. Anycast is the most scalable and standard solution for dispersed DNS server farm, while layer-4 switch could deal could do with centralized server farm;
its not a standard.
<ftp://ftp.rfc-editor.org/in-notes/rfc1546.txt>, aka _Host Anycasting Service_
Looks pretty standard to me.
Nope. Its -INFORMATIONAL- e.g. Not a Standard.
Not to mention being used in production on several major networks for over half a decade (read: "forever") on the Internet is pretty well tested technology.
true enough. but not "the most scaleable and standard solution"
-- TTFN, patrick
P.S. That would be "it's". =)
On Aug 17, 2004, at 12:31 AM, bmanning@vacation.karoshi.com wrote:
4. Anycast is the most scalable and standard solution for dispersed DNS server farm, while layer-4 switch could deal could do with centralized server farm;
its not a standard.
<ftp://ftp.rfc-editor.org/in-notes/rfc1546.txt>, aka _Host Anycasting Service_
Looks pretty standard to me.
Nope. Its -INFORMATIONAL- e.g. Not a Standard.
Ahhh, I see.
Not to mention being used in production on several major networks for over half a decade (read: "forever") on the Internet is pretty well tested technology.
true enough. but not "the most scaleable and standard solution"
Yeah, well, neither is HTTP for that matter. But most people would consider both of them pretty standard(ized?). I'm perfectly happy for someone to quote even an _informational_ RFC as a "standard". Guess I'm weird. Sorry if everyone thinks differently. -- TTFN, patrick P.S. That would be "i.e.". If you are going to argue semantic points, you should get your grammar right. =)
On 17 Aug 2004, at 00:46, Patrick W Gilmore wrote:
Nope. Its -INFORMATIONAL- e.g. Not a Standard.
P.S. That would be "i.e.". If you are going to argue semantic points, you should get your grammar right. =)
I think "a Standard" was just an example of one of the things it is not. It is also not a pressure washer, a small rodent native of south-east Asia or the cornerstone of a successful marketing campaign for hand soap. Joe
Nope. Its -INFORMATIONAL- e.g. Not a Standard.
P.S. That would be "i.e.". If you are going to argue semantic points,
you should get your grammar right. =)
I think "a Standard" was just an example of one of the things it is not. It is also not a pressure washer, a small rodent native of south-east Asia or the cornerstone of a successful marketing campaign for hand soap.
e.g. exemplia gratia In English this means "free example" but we more commonly say "for example" because most of us don't know Latin any more. i.e. id est In English this means "it is" but we more commonly say "that is <pause>" because it seems more natural somehow. In any case, try these two possibilities on for size and choose your favourite. Nope. It's -INFORMATIONAL- that is, Not a Standard. Nope. It's -INFORMATIONAL- for example, Not a Standard. My money is on the former. More punch. More pizzaz! P.S. A farmer is a man, outstanding in his field.
On Tue, Aug 17, 2004 at 03:57:17AM +0000, bmanning@vacation.karoshi.com wrote:
5. 'bogon'in BIND configuration could be used to filter requests from RFC1918 address;
this should be pushed to the router. don't waste CPU cycles on the Nameserver.
Hosts tend to be a faster writeoff cycle than routers in companies I've worked at, therefore getting the benefit of moores law about 25% faster than the routers. Turn on firewalling in the host. That said, I do filter 1918 at my edge. /vijay
That said, I do filter 1918 at my edge.
/vijay
ok everybody, vijay says the snapshot below didn't come from him. who wants to claim it, then? # tcpdump -n -c 25 net 10 or net 192.168 or net 172.16.0.0/12 tcpdump: listening on fxp0 19:52:53.787244 10.9.10.250.53 > 192.5.5.241.53: 29644 MX? rogers.com. (29) 19:52:53.789098 10.9.10.250.53 > 192.5.5.241.53: 29643 A? tock.usno.navy.mil. (36) 19:52:53.790367 10.9.10.250.53 > 192.5.5.241.53: 29642 MX? nygh.on.ca. (29) 19:52:53.791023 10.9.10.250.53 > 192.5.5.241.53: 29641 MX? sympatico.ca. (31) 19:52:54.000576 10.6.166.16.35067 > 192.5.5.241.53: 51520 PTR? 23.180.243.65.in-addr.arpa. (44) (DF) 19:52:54.000591 10.6.166.16.35067 > 192.5.5.241.53: 39692 MX? wedweb.cc. (27) (DF) 19:52:54.049835 10.21.13.50.32769 > 192.5.5.241.53: 19542 NS? . (17) (DF) 19:52:54.167651 10.1.10.8.53 > 192.5.5.241.53: 17611 PTR? 1.18.32.10.in-addr.arpa. (41) 19:52:54.227294 172.22.26.5.53 > 192.5.5.241.53: 5298 A? autodesk.com. (30) 19:52:54.327460 10.48.10.250.53 > 192.5.5.241.53: 29477 MX? unco.edu. (27) 19:52:54.328475 10.48.10.250.53 > 192.5.5.241.53: 29476 MX? unco.edu. (27) 19:52:54.329118 10.48.10.250.53 > 192.5.5.241.53: 29475 MX? icella.com. (29) 19:52:54.329736 10.48.10.250.53 > 192.5.5.241.53: 29474 MX? att.net. (26) 19:52:54.487335 10.40.1.29.53 > 192.5.5.241.53: 10970 [b2&3=0x400] A? czdm01.bauholding.com. (39) 19:52:54.490662 10.40.1.29.53 > 192.5.5.241.53: 10971 A? IBM-4406B6DF58E.bauholding.com. (48) 19:52:54.491791 192.168.0.2.1033 > 192.5.5.241.53: 4574 A? velu.neuro6.com. (33) 19:52:54.493123 192.168.0.2.1033 > 192.5.5.241.53: 4580 A? velu.neuro6.com. (33) 19:52:54.495051 192.168.0.2.1033 > 192.5.5.241.53: 12777 A? velu.neuro6.com. (33) 19:52:54.508596 172.23.3.39.1057 > 192.5.5.241.53: 2240 A? download.windowsupdate.com. (44) 19:52:54.511223 172.23.3.39.1057 > 192.5.5.241.53: 14538 A? download.windowsupdate.com. (44) 19:52:54.513568 172.23.3.39.1057 > 192.5.5.241.53: 6358 A? download.windowsupdate.com. (44) 19:52:54.527938 10.26.0.10.32769 > 192.5.5.241.53: 53702 A? nuyoo.utm.mx. (30) (DF) [tos 0x4] 19:52:54.553784 192.168.192.49.47768 > 192.5.5.241.53: 34671 PTR? 36.7.7.4.in-addr.arpa. (39) (DF) 19:52:54.605368 10.26.0.10.32769 > 192.5.5.241.53: 60698 A? uumail.unt.edu.ar. (35) (DF) [tos 0x4] 19:52:54.634115 10.26.0.10.32769 > 192.5.5.241.53: 30349[|domain] (DF) [tos 0x4] 2410 packets received by filter 0 packets dropped by kernel note: in 106 days of uptime, this particular host inside the f-root cluster has discarded the following: rule# packets --octets-- -------------rule-------------------- 00400 6446004 428112547 deny ip from 10.0.0.0/8 to any in 00500 5874604 369667080 deny ip from 172.16.0.0/12 to any in 00600 8367728 546972348 deny ip from 192.168.0.0/16 to any in this seems excessive, and so i've been assuming that it was all vijay's fault. but apparently it's not him. so which one of you isn't filtering 1918 at your edge? (oops, it's all of you, isn't it?) -- Paul Vixie
On Wed, Aug 18, 2004 at 07:57:53PM +0000, Paul Vixie wrote:
this seems excessive, and so i've been assuming that it was all vijay's fault. but apparently it's not him. so which one of you isn't filtering 1918 at your edge? (oops, it's all of you, isn't it?)
Is it really enough traffic that you, as a root server operator, can't just suck it up and deal? Sure there are going to be a few folks who are misconfigured, but I can't imagine that it is enough to cause operational issues. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
<quote who="Richard A Steenbergen">
Is it really enough traffic that you, as a root server operator, can't just suck it up and deal? Sure there are going to be a few folks who are misconfigured, but I can't imagine that it is enough to cause operational issues.
No, no operational issues at all from RFC1918 space.... http://www.as112.net/ (just to drop the most well documented example...) -davidu ---------------------------------------------------- David A. Ulevitch - Founder, EveryDNS.Net http://david.ulevitch.com -- http://everydns.net ----------------------------------------------------
On Wed, Aug 18, 2004 at 02:18:32PM -0700, David A. Ulevitch wrote:
<quote who="Richard A Steenbergen">
Is it really enough traffic that you, as a root server operator, can't just suck it up and deal? Sure there are going to be a few folks who are misconfigured, but I can't imagine that it is enough to cause operational issues.
No, no operational issues at all from RFC1918 space....
http://www.as112.net/ (just to drop the most well documented example...)
That looks like a 1918 issue to me... Lets be clear about the difference between a DNS query for 1918 space and a DNS query sources from 1918 space which can never be returned too. Yes I'm sure it is annoying, but the questions are: How much EXTRA load does it really place on the rootservers? Is it really so much load that you can't just chalk it up to a normal part of the service being provided? Or to put it another way: How much computing power would I need to buy you so that I never have to hear complaints about queries from 1918 space on a mailing list again? :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Wed, Aug 18, 2004 at 05:31:47PM -0400, Richard A Steenbergen wrote:
On Wed, Aug 18, 2004 at 02:18:32PM -0700, David A. Ulevitch wrote:
<quote who="Richard A Steenbergen">
Is it really enough traffic that you, as a root server operator, can't just suck it up and deal? Sure there are going to be a few folks who are misconfigured, but I can't imagine that it is enough to cause operational issues.
No, no operational issues at all from RFC1918 space....
http://www.as112.net/ (just to drop the most well documented example...)
That looks like a 1918 issue to me... Lets be clear about the difference between a DNS query for 1918 space and a DNS query sources from 1918 space which can never be returned too.
Yes I'm sure it is annoying, but the questions are:
How much EXTRA load does it really place on the rootservers? Is it really so much load that you can't just chalk it up to a normal part of the service being provided?
Or to put it another way:
How much computing power would I need to buy you so that I never have to hear complaints about queries from 1918 space on a mailing list again? :)
Let me put it the ultimate way: How many routers, linecards, configs, etc.. need to be upgraded to insure that there is source address validation. I want to insure that every packet I deliver to my end-customers is from a real host on the other side. Even if it's 0wned, i want to pass that packet until such time as our security team is notified and works to mitigate it. We (AS2914) attempt to insure that packets our customers pass to our network are from address space they are registered/authorized to pass. I know that AT&T (AS7018) does this as well with their customers. Anyone that isn't working on this (even slowly) is helping contribute to part of the problem/mess of rfc1918 sourced packets leaking to the internet. While there is a cost on operators of services (eg: Paul/ISC in f.root ops), it's not just the 1918 sourced packets you should be worried about, it's the people spoofing others ips... While enabling u-rpf in one of our pops, i was watching what sources were coming in on the links to insure that we were not dropping the wrong packets, or the customers didn't need to really source packets from those ranges.. a lot of machines were spewing packets from random ips on the other side of the world (europe, asia) that should not have been coming from machines in the US behind some random T1 customer.. Router#deb ip cef drops ? rpf Packets dropped by CEF Unicast RPF - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Wed, Aug 18, 2004 at 06:12:38PM -0400, Jared Mauch wrote:
Anyone that isn't working on this (even slowly) is helping contribute to part of the problem/mess of rfc1918 sourced packets leaking to the internet.
Tell it to the unfortunate number of people manufacturing customer edge aggregation boxes which still don't support uRPF. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Aug 18, 2004, at 6:46 PM, Richard A Steenbergen wrote:
On Wed, Aug 18, 2004 at 06:12:38PM -0400, Jared Mauch wrote:
Anyone that isn't working on this (even slowly) is helping contribute to part of the problem/mess of rfc1918 sourced packets leaking to the internet.
Tell it to the unfortunate number of people manufacturing customer edge aggregation boxes which still don't support uRPF. :)
I think he just did. And, perhaps more importantly, he is telling the people who are considering buying such hardware. And, perhaps most importantly, people should tell transit providers they are considering purchasing transit from that this is unacceptable. (You can consider this my personal notice to all transit providers.) -- TTFN, patrick
Is it really enough traffic that you, as a root server operator, can't just suck it up and deal? Sure there are going to be a few folks who are misconfigured, but I can't imagine that it is enough to cause operational issues.
a few folks? no. if it was a few packets now and then i'd say no problem. in the example i posted earlier, i included some numbers from one member of the "f troop", which showed ~21M packets from rfc1918 space over the course of ~106 days. that's 241 queries per second. on only one host of many. granted it's not much as a percentage of the total, but it's not "a few". furthermore, leaking rfc1918 is evidence that a network would also allow ip spoofing, and probably is being used as a spoofed-source attack vector. if we clean up the problems we can prove we're having, then it will make the remaining problems stand out a little better against an uncluttered background. (but i'm sure that a community as robustly steeped in operations philosophy as NANOG doesn't need me to tell them something so elementary-- sorry to "preach to the choir" as it were.)
Let me put it the ultimate way: ... We (AS2914) attempt to insure that packets our customers pass to our network are from address space they are registered/authorized to pass.
thank you!
I know that AT&T (AS7018) does this as well with their customers.
thank you at&t!
Anyone that isn't working on this (even slowly) is helping contribute to part of the problem/mess of rfc1918 sourced packets leaking to the internet.
yes.
While there is a cost on operators of services (eg: Paul/ISC in f.root ops), it's not just the 1918 sourced packets you should be worried about, it's the people spoofing others ips... While enabling u-rpf in one of our pops, i was watching what sources were coming in on the links to insure that we were not dropping the wrong packets, or the customers didn't need to really source packets from those ranges.. a lot of machines were spewing packets from random ips on the other side of the world (europe, asia) that should not have been coming from machines in the US behind some random T1 customer..
encore, encore! if BCP38 is too long and complicated for your management to understand when you ask for additional staff or equipment to turn on u-rpf, there's a shorter (4 pages) executive-compatible document that you should print out and staple to your requisition, at <http://www.icann.org/committees/security/sac004.txt>, from which i shall hum a few bars since the "icann" might be a turnoff: SECSAC Paul Vixie, ISC SAC 004 October 17, 2002 Securing the Edge Abstract At every edge of the global Internet are the hosts who generate and consume the packet flows which, together, form the overall Internet traffic load. By number, most of these hosts are not secure, leading to dangerous, untraceable traffic flows which can be used to attack other hosts. This memo describes some of the security problems "at the edge" and makes some recommendations for improvement. ... yes, this really was published four days before a widely publicized global DDoS against the root name server system, which was documented at <http://d.root-servers.org/october21.txt>. this was just a coincidence, but as long as i'm humming songs for y'all, here's the top of this one: ISC/UMD/Cogent Paul Vixie, ISC OCTOBER21.TXT Gerry Sneeringer, UMD November 24, 2002 Mark Schleifer, Cogent Events of 21-Oct-2002 Abstract On October 21, 2002, the Internet Domain Name System's root name servers sustained a denial of service attack. This report explains the nature and impact of the attack, based on previously and publically available information. ... happy homework! and please keep those rfc1918-related / u-rpf related cards and letters coming. -- Paul Vixie
vixie@vix.com (Paul Vixie) writes:
in the example i posted earlier, i included some numbers from one member of the "f troop", which showed ~21M packets from rfc1918 space over the course of ~106 days. that's 241 queries per second. on only one host of many. granted it's not much as a percentage of the total, but it's not "a few".
sorry, i mixed up my numbers. it's only a trickle of queries per second at any given f-root node. but it sure adds up when you look at all f-root nodes together, or when you look at all root servers together. -- Paul Vixie
Joe Shen wrote:
We noticed there is continous name resolution requests from IP address outside of our address pool and also there is requests not conforming to DNS documents ( like those from 10/8, 192.168/16 or something for microsoft proxy server name). We think these request waste our resource and we don't want these system stable, secure and high performance.
If the resolver caches are only supposed to be accessed from your IP space, I am sure you can easily throw in a router ACL to accept connections on port 53 only from these IPs. Oh, and filter out bogons at your borders while you are at it (like for example rfc1918 source addresses from outside your network) srs
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Suresh Ramasubramanian wrote: | | Joe Shen wrote: | |> We noticed there is continous name resolution requests |> from IP address outside of our address pool and also |> there is requests not conforming to DNS documents ( |> like those from 10/8, 192.168/16 or something for |> microsoft proxy server name). We think these request |> waste our resource and we don't want these system |> stable, secure and high performance. | | | If the resolver caches are only supposed to be accessed from your IP | space, I am sure you can easily throw in a router ACL to accept | connections on port 53 only from these IPs. | | Oh, and filter out bogons at your borders while you are at it (like for | example rfc1918 source addresses from outside your network) | And check out the CYMRU Secure Bind template at http://www.cymru.com/Documents/secure-bind-template.html - -- ========= bep -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (MingW32) iD8DBQFBIQ3HE1XcgMgrtyYRAuAXAJ4z6GI+X7nPL3wZZ2kvB30YGQ+B/QCeIagA mqIz2gcRVeY+g2LVBjLc6dQ= =iAkf -----END PGP SIGNATURE-----
participants (13)
-
Bill Woodcock -
bmanning＠vacation.karoshi.com -
Bruce Pinsky -
David A. Ulevitch -
Jared Mauch -
Joe Abley -
Joe Shen -
Michael.Dillon＠radianz.com -
Patrick W Gilmore -
Paul Vixie -
Richard A Steenbergen -
Suresh Ramasubramanian -
vijay gill