On Jun 5, 2012, at 9:52 AM, Peter Kristolaitis wrote:
As far as horror stories... yeah. My most memorable experience was a guy (with a CISSP designation, working for a company who came highly recommended) who: - Spent a day trying to get his Backtrack CD to "work properly". When I looked at it, it was just a color depth issue in X that took about 45 seconds from "why is this broken?" to "hey look, I fixed it!". - Completely missed the honeypot machine I set up for the test. I had logs from the machine showing that his scanning had hit the machine and had found several of the vulnerabilities, but the entire machine was absent from the report. - Called us complaining that a certain behavior that "he'd never seen before" was happening when he tried to nmap our network. The "certain behavior" was a firewall with some IPS functionality, along with him not knowing how to read nmap output. - Completely messed up the report -- three times. His report had the wrong ports & vulnerabilities listed on the wrong IPs, so according to the report, we apparently had FreeBSD boxes running IOS or MS SQL... - Stopped taking our calls when we asked why the honeypot machine was completely missing from the report.
In general, my experience with most "pen testers" is a severe disappointment, and isn't anything that couldn't be done in-house by taking the person in your department who has the most ingrained hacker/geek personality, giving them Nessus/Metasploit/nmap/etc, pizza and a big ass pot of coffee, and saying "Find stuff we don't know about. Go.". There is the occasional pen tester who is absolutely phenomenal and does the job properly (i.e. the guys who actually write their own shellcode, etc), but the vast majority of "pen testers" just use automated tools and call it a day. Like everything else in IT, security has been "commercialized" to the point where finding really good vendors/people is hard, because everyone and their mom has CEH, CISSP, and whatever other alphabet soup certifications you can imagine.
I agree with a lot of what you've said, but there are absolutely good security guys (pen tester, vulnerability assessors, etc) that use both open source and commercial automated tools, but still do a fantastic job because they understand the underlying technologies and protocols. I used to do a lot of this in the past, had lots of automated tools, and only occasionally wrote some assessment modules or exploit code if necessary. But again, a person in that position has to understand technology holistically (network, systems, software, protocols, etc). -b