- NANOG - lists.nanog.org

Re: How to get better security people
by Kelly J. Cooper 29 Mar '02

29 Mar '02

On Mar 29, 2:22pm, Sean Donelan wrote: > Subject: Re: How to get better security people * *On Tue, 26 Mar 2002, Kelly J. Cooper wrote: *> I also had a short list of other questions that I used to try and get *> a feel for the person's "security minded-ness" (my term, I invented it *> a'ight?). Because when it comes to ISP security, there's a very *> limited pool of talent so candidates are unlikely to come in with the *> right skillset native. * *What is the right mindset for ISP security. It seems to be a little *different from the traditional security mindset found in the corporate *or military security world. A lot of sharp people with that background *try to move into ISP security, but they often have a difficult time *making the transition. Hmm. Incredibly biased opinion follows... A basic security mindset is a combination of paranoia, a talent for contingency planning, and an understanding of business need. However, the paranoia must not be so extensive as to be crippling, the contingency planning must not be so obsessive as to be paralysing, and the understanding of business need should not interfere with the periodic difficult and unpopular decisions that must be made to protect the greater good. Specific skill-sets that are useful for ISP Operational Security (pick one or mix-n-match for the overachieving): - Incident Response/handling capability - Deep understanding of TCP/IP - Deep understanding of the design of big WANs - Deep understanding of the design of switched LANs (hosting ISPs) - Unix adminstration and forensics - Microsoft administration and forensics - Firewall administration and forensics (NOTE that I'm not covering Engineering Security, at least not in this post.) I would say the most important skill for a dedicated ISP Security person to have is that of incident handling. Then again, it happens to be the one skill out of this set that I have, so extra bias hold the sauce. But hear me out... When a customer gets hit, it can be a break-in, a DoS attack, a DDoS attack, an insider betrayal, some accidentally free porn, a really dumb move by the marketing department, a political sit-in, bad press, an attempted break-in, a misconfiguration, a /. overload or something else entirely. Whatever it is, once the intial triage is done and the Security person is brought in, she's got to have a broad base of knowledge about the possibilities as well as knowledge of her own organization to know how to engage experts to assist. Plus she's got to document the whole thing and keep track of the contributions by the customer, by the experts, by management, etc. A really great security team has someone with each of those skills who can be brought into an event to help, each utilising her particular expertise. They've sat in with other teams so that they understand how the network works, how triage and trouble-shooting are done, how teams hand things off to one another, etc. And they have just enough cross-training to know when they should hand off to another security team-member. Best case scenario, most of the team has incident handling skills so that no particular handler is always getting paged. So the mindset is jack-of-all-trades rather than specifically focused on one task. The work is interrupt-driven rather than project or patch/upgrade driven. The mindset is to share information (judiciously) and bring people in, rather than keeping it a secret and doing it yourself. Those differences might explain the difficult transitions. *The government is about to spend a lot of *money training students in "cybersecurity." Congressional aides have *been coming to Internet conferences asking people what should Congress *spend money on. * *http://www.washingtonpost.com/wp-dyn/articles/A33471-2002Mar28.html * *But are the students really getting the right training for working in *a public network such as an ISP? If they're being taught about security in general, like policy and procedure writing and management, what we mean by access controls, how to handle disaster recovery, crypto basics, perimeter management, incident response, then that's fantastic. Even if they go to an ISP, they'll have the right skillset to start and they can learn the rest on the job. If they are ALSO being taught network design (LAN and WAN), firewall basics, the value of the heterogeneous network, how packets get put together and pulled apart, routing, end-to-end troubleshooting, DNS infrastructure, and maybe the specific configuration details for some of the top router vendors, then they are absolutely golden to go into ISP Security. But since I have no idea what they're learning, I can't comment on that specific article. There's some indication in the article that students are learning system hardening. That's usually a good skill. There's no indication that students are focusing on ISP skills or on ISP jobs. So, just out of curiousity, why are you asking this question? On the NSF's website, I found the Education & Human Resources (EHR) pages - www.ehr.nsf.gov. It includes the Division of Undergraduate Education (DUE) - www.ehr.nsf.gov/due/ - which includes the Federal Cyber Service: Scholarship for Service (SFS) page, which appears to be the program referenced in the article. http://www.ehr.nsf.gov/ehr/due/programs/sfs/ But it's hard to tell without digging through each of the awards that has been made what they're focusing on, although the general sense I get is that they're trying to increase the number of clueful IT Security personnel. So is your point idle speculation? Or that we should be designing curricula to increase the number of ISP Security folks? Or are we bemoaning the government's possibly misguided focus? Regards, Kelly J. -- Kelly J. Cooper - Security Engineer, CISSP GENUITY - Main # - 800-632-7638 3 Van de Graaff Drive - Fax - 781-262-2744 Burlington, MA 01803 - http://www.genuity.net

2 1

Re: How to get better security people
by Kelly J. Cooper 29 Mar '02

29 Mar '02

On Mar 26, 2:15pm, Sean Donelan wrote: > Subject: Re: How to get better security people * *On Tue, 26 Mar 2002, Tony Wasson wrote: *> >> If I was looking for top security talent, what would I ask for whether *> >> I was hiring directly or outsourcing? *> *> I agree with Steve Wilcox, incidents are important. I would ask for a *> description of the 3 most interesting incidents they've ever worked on, and *> what they contributed. * *I'm sorry, but that's confidential information and I can't disclose it. * *Would you hire a "security" person, who will likely be involved in the *most embarrassing slip ups your company makes, if he tells people about *"interesting" incidents at previous employers. * *Maybe, it depends on what he says. Long ago and downstairs, when I used to interview people for Operations Security, I asked each candidate whether s/he had ever handled a Denial of Service attack or an intrusion, and if so, could they describe in general terms how they handled it? I would specifically ask them to NOT provide any identifying info, just the process (and an explication of the attack) so I could gauge their understanding of the situation. I also had a short list of other questions that I used to try and get a feel for the person's "security minded-ness" (my term, I invented it a'ight?). Because when it comes to ISP security, there's a very limited pool of talent so candidates are unlikely to come in with the right skillset native. But if the person comes in and s/he is someone who thinks about scenarios and contingency plans and has a working knowledge of networking/computing, then I can teach him/her everything else. Kelly J. -- Kelly J. Cooper - Security Engineer, CISSP GENUITY - Main # - 800-632-7638 3 Van de Graaff Drive - Fax - 781-262-2744 Burlington, MA 01803 - http://www.genuity.net

3 2

RADB renewal deadline April 2
by Larry J. Blunk 29 Mar '02

29 Mar '02

This is a reminder that effective April 2, 2002, Merit will begin the process of removing RADB maintainer objects that have not been renewed and paid for 2002. The clean-up project will take several weeks to complete. The estimated completion date for removal of all unpaid maintainers is May 31. Maintainers scheduled for removal are listed at http://www.radb.net/docs/deletelist.html. You may also query our payment database at the same URL to determine the status of your maintainer. About 1,000 maintainers with valid route objects that are currently being announced to the Internet are scheduled to be removed. These maintainers will be removed gradually -- but steadily -- over the next several weeks. It is not too late to contact Merit to make arrangements to continue your RADB service. If you wish to renew service, please send e-mail to invoice(a)radb.net or or go to our online payment page: https://www.merit.edu/cgi-bin/radb/pay/alive All deleted RADB data will be archived by Merit. We will be able to restore deleted maintainers even if payment is made after April 2. If your data is removed as part of the clean-up project and you wish to renew service, please contact invoice(a)radb.net. URLs: To renew service and pay online: https://www.merit.edu/cgi-bin/radb/pay/alive To view or query the list of maintainer IDs scheduled for removal: http://www.radb.net/docs/deletelist.html To review the RADB maintainer object agreement: http://www.radb.net/docs/agreement.html For general information: http://www.radb.net ------ Larry J. Blunk Merit Network, Inc. Ann Arbor, Michigan

1 0

RE: Help with bad announcement from UUnet
by James Smith 29 Mar '02

29 Mar '02

I would generally agree that non-paying customers should not get top-shelf service, but when it is someone with clue calling (your people should be able to tell, they should be clueful to a degree) about an issue that is being sourced from your network, or transits your network, is it not an issue that you should be involved in anyway? Why wait for the call from your upstream when you can get a jump on the problem? James H. Smith II NNCDS NNCSE Systems Engineer The Presidio Corporation Yeah, I know, top-posting is frowned upon. I have other bad habits... -----Original Message----- From: fingers [mailto:fingers@fingers.co.za] Sent: Friday, March 29, 2002 11:51 AM To: Leo Bicknell Cc: nanog(a)merit.edu Subject: Re: Help with bad announcement from UUnet Hi > Note that in both cases, b0rken-noc takes a single call, so their > load is unchanged. The second case adds a call to both my-upstream-noc, > and b0rken-noc-upstream-noc. > > It would seem going direct would put a lower load on NOC's in general, > which presumably would let them spend more time on problems and provide > better service. surely a noc's first responsability is to direct customers? even if the other network experiencing the problem may affect said customer, the service is not just about connectivity, but also about trying to deal with calls in the best possible manner. if more time were spent on non-customers, a paying customer would end up losing out on that warm fuzzy feeling when his call is answered promptly, the person he speaks to actually listens, and his general experience interacting with the noc is something he doesn't walk away from feeling cheated. Regards --Rob

1 0

Call for Presentations
by Susan Harris 29 Mar '02

29 Mar '02

* * * * * * * * * * * * * * * * * CALL FOR PRESENTATIONS * * NANOG 25 * * June 9 -12, 2002 * * * * * * * * * * * * * * * * * The North American Network Operators' Group (NANOG) will hold its 25th meeting June 9-11, 2002, in Richmond Hill, Ontario, Canada (just outside Toronto). The meeting will be hosted by Group Telecom. Registration begins April 22. NANOG conferences provide a forum for the coordination and dissemination of technical information related to large-scale (i.e., national/international) Internet backbone networking technologies and operational practices. Meetings are held three times each year, and include two days of short presentations, plus afternoon/evening tutorial sessions. The meetings are informal, with an emphasis on relevance to current backbone engineering practices. NANOG conferences draw over 500 participants, mainly consisting of engineering staff from national service providers, and members of the research and education community. <p> The meeting will be held at the the Sheraton Parkway Toronto North. For more information about NANOG meetings, schedules, and logistics, see: http://www.nanog.org ------------------------------------------------------------------------------ CALL FOR PRESENTATIONS NANOG invites presentations on backbone engineering, coordination, and research topics. Presentations should highlight issues relating to technology already deployed or soon to be deployed in core Internet backbones and exchange points. Previous meetings have included presentations on: - Backbone traffic engineering - Inter-provider security and routing protocol authentication - Routing scalability in backbone infrastructures - Security issues for the Internet core - Routing policy specification and backbone router configuration - Building large-scale measurement infrastructure - Cooperative inter-provider caching - Alternatives to hot-potato routing - Recommendations on queue management and congestion avoidance - Experience with differentiated services - Inter-domain multicast deployment - Backbone network failure analysis Tutorials have covered topics such as: - Inter-domain traffic engineering: principles, applications, and case studies - BGP multihoming guide - ISP security: real world techniques - IP multicast technologies ------------------------------------------------------------------------------ HOW TO PRESENT Submit a detailed abstract or outline describing the presentation in email to nanog-support(a)nanog.org. The deadline for proposals is April 29, 2002. While the majority of speaking slots will be filled by April 29, a limited number of slots will be available after that date for topics that are exceptionally timely and important. Submissions will be reviewed by the NANOG Program Committee, and presenters will be notified of acceptance by May 13, 2002. NANOG also welcomes suggestions/recommendations for tutorials, panels and other presentation topics. ---------------------------------------------------------------------------

1 0

Not sure what I'm seeing... (Looking Glass Query)
by Aaron Mahler 29 Mar '02

29 Mar '02

Hello! Hope nobody minds a quick question from a small, non-transit but BGP aware AS... Since the day we started using multiple circuits and BGP I've always seen multiple paths through each of our providers to our network (AS 19138). At least a few C&W (3561) and at least a few Network Virginia (7066) via Sprint... For weeks or more now, every single looking glass save one shows - for the most part - a single advertised path through C&W. No sign of Sprint & Network Virginia. Even the Sprint looking glass shows no sign of a Sprint path to us. The one exception is route-views.oregon-ix.net which shows about 50... almost all of which are C&W. If I pull down the C&W circuit, things shift and Sprint starts showing up with multiple paths in all the looking glasses but will remain that way after the C&W circuit comes back online until I bring down the Sprint circuit. It seems to be one or the other... but never both. Is this not a big deal and I'm misreading the situation, or has something changed for the worse somewhere in our advertising and relationship with our providers? I can't seem to track down a cause. I've certainly made no changes to my BGP tables and announcements upstream. For anyone interested in glancing at their routes for me, check 198.28.32.0/19 and AS 19138. Thanks! - Aaron Mahler Sweet Briar College P.S. Feel free to yell at me if this post shouldn't be here. I've been lurking for ages and enjoy following the discussions of the very knowledgeable crew out there. Not sure what other forum to ask such a question at this point...

2 1

Route filters, IRRs, and route objects
by Przemyslaw Karwasiecki 29 Mar '02

29 Mar '02

Hello, I would like to ask you for an advice in regards to "proxy registering" of customer route objects in IRR. What is the best current practice in a situation, when your customers want to advertise to you several /18 or /19 but they also have a requirement to be able to advertise some deaggregated routes on top of aggregates. It is very common that they are unable to predict exactly which deaggregated routes they will need to advertise, as they use those to achieve some traffic engineering objectives which change over time. And "over time" does NOT occur once per 30 minutes or so, so they DON'T generate any major BGP fluctuations. Forgive my ignorance, but is my understanding of RPSL correct, that it should be possible to specify routes in a way which will allow cover aggregate plus whole set of possible more specific routes upto certain netmask length. Something like: 10.0.0.0/18^18-24 So why this is uncommon to use such notation to describe routing policy, and use it to generate filters? Why it is required by some providers to generate explicit, exact route objects, in order to allow routes through their filters? Is it really necessary to "explode" route-sets like those 10.0.0.0/m^m-n into 2^(n-m+1) separate route objects to meet requirements of some providers? I believe that this is very common problem, so if there are any places on the web with some "best practice" documents, please point me to them. Thank you, Przemek

4 4

RE: Let's talk about Distance Sniffing/Remote Visibility
by Gironda, Andre 29 Mar '02

29 Mar '02

On Thu, Mar 28, 2002 at 9:20AM -0700, Richard A Steenbergen wrote: > > I'm imagining that even with a relatively speedy box, if you were > > trying to do analysis from multiple interfaces you'd at least choke > > the disk I/O. There's always stringent filters, I guess. > > Disk I/O on a sniffer box? Sounds like you've been sniffing something > other than packets my friend. :) Why do you say that? In the 10/100 range, yes, no problems. But at the Gigabit range (say with two GbE cards or a single OC-48 card) on an x86 box with IDE disks (or even SCSI RAID0), doesn't disk I/O become a severe problem? Under Solaris or Linux, scaling disk seems relatively easy with Veritas Foundation Suite on Solaris or GFS under Linux http://www.sistina.com/products_gfs.htm However, I don't think Linux or Solaris can handle the packet capture capabilities like FreeBSD and BPF can. I've heard things about the new LPF capabilities and turbopacket, but it's just hard to believe coming from such a joke/toy operating system. Whether you are passively tapping a gigabit ethernet or SONET fiber, or even spanning an entire VLAN or mirroring a gigabit ethernet or SONET port on a router/switch -- you've got a lot of packets/frames to deal with, especially if you want to keep all of them for analyzing later. Sounds like a disk I/O problem to me. Are you doing packet capture at these rates and ran into no disk problems? How did you deal with that? We are doing so right now, but only with the IP headers and some "top N" information. Getting full packets and keeping them for awhile (a day or two even) is going to take a lot of I/O and disk space. I don't think it's worth it, really. > You can build your own box like that easily enough. If you're going > for FastE sniffing I highly recommend the Adaptec Quartet 4-port > cards. If you're going for GigE sniffing, I STILL highly recommend > anything Alteon Tigon 2 based (NetGear GA620's were the cheapest > if you can still find them, not the 621/622). You can accomplish almost anything with the Tigon2's and FreeBSD, agreed. Another vendor I'm sort of looking at now is Endance (DAG cards): http://www.endace.com/ http://www.endace.com/products/dag42ge.html This only does IP headers, but that's the fun stuff anyways ;> > You don't even have to do anything fancy with the card firmware, > there is a native command for receiving only part of the frame. > Check out the programming manuals at > http://people.freebsd.org/~wpaul/Alteon/, and I recommend you use > FreeBSD for this of course. Just add in a PARTIAL_RX_CNT command, > and the card will only DMA part of the packet (say 64 bytes for > full headers) across the PCI bus. Combined with interrupt coalescing > (or luigi's device polling and tuning the card to allocate all > memory to RX and remove the TX functionality completely), you can > sniff quite a few "gigabits" of traffic on a single cheap PC server. > You can dump it through the BPF mechanism and still maintain support > for all your favorite sniffer programs. Or if you're comfortable > writing kernel code, I recommend you make a character device for > sniffer device control, and use it to pass page-aligned malloc'd > memory pointers from userland into the nic driver, which you then > pass to the card as the RX ring buffers. This will let you DMA your > packets directly into userland. If not, at least unhook ether_input(). > :) Can you post more details or catch up with me offline about this? I'm really very interested in your implementations and results. > Or you can buy these things commercially. My favorite was from a > company called Tekelec, who sold a VERY expensive box which turned > out to be a pentium 200ish box running solaris x86 and completely > useless sniffing software, with a bunch of ISA ethernet cards hooked > up by proprietary (and VERY expensive) cables, all in a box made > out of what I swear was some kind of lead/neutron star material > alloy. Of course that was a couple years ago, maybe they've upgraded > to the current market's $50 processor. > :) We've been looking at NetVCR from Niksun which sounds similar except that is actually is FreeBSD-based. Somebody needs to put together a list of all these companies and do some comparisons of the product offerings. Like you, I'd rather just build my own box and run with it ;> -dre > -- > Richard A Steenbergen <ras(a)e-gerbil.net> > http://www.e-gerbil.net/ras > PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 > 14 36 FE B6) >

2 1

Re: BGP without an IGP
by Jake Khuon 28 Mar '02

28 Mar '02

### On Thu, 28 Mar 2002 13:40:51 -0500, "Abarbanel, Benjamin" ### <Benjamin.Abarbanel(a)Marconi.com> casually decided to expound upon ### "'Randy Bush'" <randy(a)psg.com>, Russ White <ruwhite(a)cisco.com> the ### following thoughts about "RE: BGP without an IGP": BA> into the AS as IBGP routes. But from what I understood Ken original topology BA> he was only talking about reachability within the AS. Reachability between IBGP BA> peers that are more than 1 hop away. Unless memory and past email messages serve me wrong, I believe Ken's topology called for full-mesh. BTW, we ran iBGP full mesh without an IGP quite fine. Okay.. so there's a twist... We did it for IPv6 (before Cisco had IPv6 IS-IS) but I see no reason why it wouldn't also work for IPv4. -- /*===================[ Jake Khuon <khuon(a)NEEBU.Net> ]======================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | --------------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| N E T W O R K S | +=========================================================================*/

1 1

Let's talk about Distance Sniffing/Remote Visibility
by CARL.P.HIRSCH＠sargentlundy.com 28 Mar '02

28 Mar '02

I'd like to hear from the list as to what your preferred means of determining what the hell is going on at a packet level at the other side of a WAN/MAN/frame/etc link. It seems to me that the means available are A) a very expensive distributed NAI Sniffer installation B) standard RMON probes and the NMS of your choice and C) A linux box with a ton of interfaces running Ethereal accessed via Xwindows/VNC/whatever. Personally, I'm inclined to explore option C. The bells & whistles of a commercial package like NAI's Distributed Sniffer are not really a compelling value added to me. It's a shame there's no Open Source RMON agent available. I'm asking NANOG because I figure this is something pretty much every network engineer bumps into at one time or another. thanks, -carl

7 6