Re: Let's talk about Distance Sniffing/Remote Visibility
Yeah, the device I've got in my head is a 1U server with 4 (or more?) interfaces... not so much to simultaneously pull 400Mbps of bandwidth for analysis but rather to just have a interface going to each switch I might want to monitor and then span traffic to the Ethereal box. Given that I'm trying to attain remote visibility, it might be nice not to need remote hands to be swapping patch cords back and forth. I'm imagining that even with a relatively speedy box, if you were trying to do analysis from multiple interfaces you'd at least choke the disk I/O. There's always stringent filters, I guess. thanks for the input, -carl "E.B. Dreger" <eddy+public+spam@noc.ever To: CARL.P.HIRSCH@sargentlundy.com quick.net> cc: nanog@merit.edu Sent by: Subject: Re: Let's talk about Distance Sniffing/Remote owner-nanog@merit.edu Visibility 03/28/02 09:02 AM "C" is close. Not sure what you mean by "a ton of interfaces". Most (all?) good managed switches have a "monitor port" or "mirror port" where they can blind copy traffic from other ports to the one that's set aside for snooping. Four-port ethernet cards are readily available. How many switches do you wish to monitor simultaneously? Even with only four ports (more in one box is certainly possible), you can have a fair amount of traffic to digest. -- Eddy Brotsman & Dreger, Inc. - EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence -- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
On Thu, Mar 28, 2002 at 09:23:41AM -0600, CARL.P.HIRSCH@sargentlundy.com wrote:
Yeah, the device I've got in my head is a 1U server with 4 (or more?) interfaces... not so much to simultaneously pull 400Mbps of bandwidth for analysis but rather to just have a interface going to each switch I might want to monitor and then span traffic to the Ethereal box. Given that I'm trying to attain remote visibility, it might be nice not to need remote hands to be swapping patch cords back and forth.
I'm imagining that even with a relatively speedy box, if you were trying to do analysis from multiple interfaces you'd at least choke the disk I/O. There's always stringent filters, I guess.
Disk I/O on a sniffer box? Sounds like you've been sniffing something other than packets my friend. :) You can build your own box like that easily enough. If you're going for FastE sniffing I highly recommend the Adaptec Quartet 4-port cards. If you're going for GigE sniffing, I STILL highly recommend anything Alteon Tigon 2 based (NetGear GA620's were the cheapest if you can still find them, not the 621/622). I've had great luck making the Tigon 2's into dedicated sniffers. You don't even have to do anything fancy with the card firmware, there is a native command for receiving only part of the frame. Check out the programming manuals at http://people.freebsd.org/~wpaul/Alteon/, and I recommend you use FreeBSD for this of course. Just add in a PARTIAL_RX_CNT command, and the card will only DMA part of the packet (say 64 bytes for full headers) across the PCI bus. Combined with interrupt coalescing (or luigi's device polling and tuning the card to allocate all memory to RX and remove the TX functionality completely), you can sniff quite a few "gigabits" of traffic on a single cheap PC server. You can dump it through the BPF mechanism and still maintain support for all your favorite sniffer programs. Or if you're comfortable writing kernel code, I recommend you make a character device for sniffer device control, and use it to pass page-aligned malloc'd memory pointers from userland into the nic driver, which you then pass to the card as the RX ring buffers. This will let you DMA your packets directly into userland. If not, at least unhook ether_input(). :) Or you can buy these things commercially. My favorite was from a company called Tekelec, who sold a VERY expensive box which turned out to be a pentium 200ish box running solaris x86 and completely useless sniffing software, with a bunch of ISA ethernet cards hooked up by proprietary (and VERY expensive) cables, all in a box made out of what I swear was some kind of lead/neutron star material alloy. Of course that was a couple years ago, maybe they've upgraded to the current market's $50 processor. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Date: Thu, 28 Mar 2002 12:19:55 -0500 From: Richard A Steenbergen <ras@e-gerbil.net>
(snipping throughout)
Disk I/O on a sniffer box? Sounds like you've been sniffing something other than packets my friend. :)
I like to log interesting packets; I agree with Carl.
You can build your own box like that easily enough. If you're going for FastE sniffing I highly recommend the Adaptec Quartet 4-port cards. If
D-Link DFE-570TX are _very_ cheap if you're happy with 32-bit / 33 MHz PCI. [ snip FreeBSD + Alteon ] I did not know about the partial-packet DMA transfers. Mmmmm....
Or if you're comfortable writing kernel code, I recommend you make a character device for sniffer device control, and use it to pass page-aligned malloc'd memory pointers from userland into the nic driver, which you then pass to the card as the RX ring buffers. This will let you DMA your packets directly into userland. If not, at least unhook ether_input(). :)
Never done this. About how much "capacity" does the zero-copy approach add? -- Eddy Brotsman & Dreger, Inc. - EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence -- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
On Thu, Mar 28, 2002 at 05:59:20PM +0000, E.B. Dreger wrote:
I like to log interesting packets; I agree with Carl.
Logging interesting packets is easy enough, its logging ALL packets that would be a problem. At any rate, you'd run out of harddrive space pretty quick if you were pushing max performance at any length of time. I can write a linerate FastE's worth of data to a $100 IDE disk on a $100 processor easily enough, so as long as you're buffering it intelligently it shouldn't be an issue.
Or if you're comfortable writing kernel code, I recommend you make a character device for sniffer device control, and use it to pass page-aligned malloc'd memory pointers from userland into the nic driver, which you then pass to the card as the RX ring buffers. This will let you DMA your packets directly into userland. If not, at least unhook ether_input(). :)
Never done this. About how much "capacity" does the zero-copy approach add?
Eliminating the bulk "data" being DMA's across the PCI bus is what adds most of your "capacity". Doing zero copy just lets you spend all your CPU time doing actual analysis instead of copying stuff around unnecessarily. I never did get the opportunity to benchmark it at 1.4million packets/sec, (I spent more time trying to get the 20ft of fiber to reach the lab at the previous employeer than I did writing the code to do this in the first place) but I don't see any reason it shouldn't work, with proper interrupt coalescing of course. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Date: Thu, 28 Mar 2002 13:14:17 -0500 From: Richard A Steenbergen <ras@e-gerbil.net>
Logging interesting packets is easy enough, its logging ALL packets that would be a problem. At any rate, you'd run out of harddrive space pretty quick if you were pushing max performance at any length of time. I can write a linerate FastE's worth of data to a $100 IDE disk on a $100 processor easily enough, so as long as you're buffering it intelligently it shouldn't be an issue.
This is true. Logging interesting packets, efficient buffering, and selective parsing make the big difference. I guess it also depends on log format: Raw packet( fragment)s work great. Human-readable -- a la, say, Linux kernel verbose IP logs -- make things get ugly in a hurry. With fixed-size packet captures, it would be trivial to use a disk slice as one big scratchpad, much like a swap partition. No real need for fs overhead, and one could reserve blocks for indices or other conveniences.
Eliminating the bulk "data" being DMA's across the PCI bus is what adds most of your "capacity". Doing zero copy just lets you spend all your CPU time doing actual analysis instead of copying stuff around unnecessarily.
Hmmmm. Looking back at Agner Fog's Pentium optimization guide, it does appear that the memblk cp would be less of an issue than the DMA transfers.
I never did get the opportunity to benchmark it at 1.4million packets/sec, (I spent more time trying to get the 20ft of fiber to reach the lab at the previous employeer than I did writing the code to do this in the first place) but I don't see any reason it shouldn't work, with proper interrupt coalescing of course.
It would be an interesting test. But ~100 MB/sec of traffic would choke most any single spool drive... and, assuming that all the data were of interest, it would probably take people awhile to review all the data. -- Eddy Brotsman & Dreger, Inc. - EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence -- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
participants (3)
-
CARL.P.HIRSCH＠sargentlundy.com -
E.B. Dreger -
Richard A Steenbergen