using sniffer on high-bandwidth pipes
does anyone have expirience using a sniffer on a hi-capacity network segment, that might know if there are limitations I need to worry about? example: customers doing EMC database replication across a mpls link, and when the capacity reaches aprox. 250 Mbp/s packets are arriving out of sequence etc. So we need to put sniffers on both sides to capture some data to see whats happeneing when the capacity reaches 250mbps. what kind of system requirements would be needed to be able to be able to capture that amount of data. For some reason, I dont think that the Dolch Pac 65 sniffers we have (running nt4 and sniffer pro2) would be able to handle that kind of data? If they cant, we can probbaly use a sun box. what kind of specs would the box need? tia, tr
It probably depends more on pps than bandwidth. At a prior job, I used FreeBSD 4.x machines to capture over 400,000 pps, I think, on gigabit links. You need a nic that is supported with one of the device polling drivers to keep CPU manageable. (Intel, not yet broadcom.) FreeBSD far surpassed Solaris in packet capture performance. Linux 2.6 machines may do OK, using NAPI - but I've no experience with that. todd romero wrote:
does anyone have expirience using a sniffer on a hi-capacity network segment, that might know if there are limitations I need to worry about?
example: customers doing EMC database replication across a mpls link, and when the capacity reaches aprox. 250 Mbp/s packets are arriving out of sequence etc. So we need to put sniffers on both sides to capture some data to see whats happeneing when the capacity reaches 250mbps.
what kind of system requirements would be needed to be able to be able to capture that amount of data. For some reason, I dont think that the Dolch Pac 65 sniffers we have (running nt4 and sniffer pro2) would be able to handle that kind of data? If they cant, we can probbaly use a sun box. what kind of specs would the box need?
tia, tr
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steve Francis wrote: | | It probably depends more on pps than bandwidth. | At a prior job, I used FreeBSD 4.x machines to capture over 400,000 pps, | I think, on gigabit links. | You need a nic that is supported with one of the device polling drivers | to keep CPU manageable. (Intel, not yet broadcom.) | | FreeBSD far surpassed Solaris in packet capture performance. | | Linux 2.6 machines may do OK, using NAPI - but I've no experience with | that. | Eric Weigle and Wu-Chen Feng presented a paper at PAM2002 entitled "TICKETing High-Speed Traffic with Commodity Hardware and Software" where they showed collecting traffic at greater than 600Mbps and to 1Gbps in some configurations. See http://public.lanl.gov/radiant/pubs.html#TICKET - -- ========= bep -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (MingW32) iD8DBQFBsJS1E1XcgMgrtyYRAsHiAKCRHj6cIEuxut3vcNMHZf+RIA3/QACg2txD 4fEavciBOTN4TwuigotN14c= =VEBQ -----END PGP SIGNATURE-----
On 3-dec-04, at 17:08, Steve Francis wrote:
It probably depends more on pps than bandwidth.
Although if you have very high bandwidth you may run into trouble with the PCI bus. 33 MHz 32 bit PCI can barely manage 1 Gbps, and that's withough taking overhead into account.
At a prior job, I used FreeBSD 4.x machines to capture over 400,000 pps, I think, on gigabit links.
I managed to do 600k with 32% CPU on a non-too-high-end machine two years ago. (Just taking the packets off the wire and running them through BPF, no processing, though.) If you use BPF or pcap, don't forget to increase the capture buffer or you'll have overruns, and don't capture more of the packet than you need.
We are using FreeBSD 4.x on 1Gbit Ethernet (for snifferring). Never had a problems (but I should not garantee 100% snifferring on 400,000pps). In reality, correct, pps is important, bandwidth is not important. If traffic is VoIP, it's a problem; if it is 90% WEB, it's an easy task. ----- Original Message ----- From: "Steve Francis" <sfrancis@fastclick.com> To: "todd romero" <todd@routeflap.net> Cc: <nanog@nanog.org> Sent: Friday, December 03, 2004 8:08 AM Subject: Re: using sniffer on high-bandwidth pipes
It probably depends more on pps than bandwidth. At a prior job, I used FreeBSD 4.x machines to capture over 400,000 pps, I think, on gigabit links. You need a nic that is supported with one of the device polling drivers to keep CPU manageable. (Intel, not yet broadcom.)
FreeBSD far surpassed Solaris in packet capture performance.
Linux 2.6 machines may do OK, using NAPI - but I've no experience with
that.
todd romero wrote:
does anyone have expirience using a sniffer on a hi-capacity network segment, that might know if there are limitations I need to worry about?
example: customers doing EMC database replication across a mpls link, and when the capacity reaches aprox. 250 Mbp/s packets are arriving out of sequence etc. So we need to put sniffers on both sides to capture some data to see whats happeneing when the capacity reaches 250mbps.
what kind of system requirements would be needed to be able to be able to capture that amount of data. For some reason, I dont think that the Dolch Pac 65 sniffers we have (running nt4 and sniffer pro2) would be able to handle that kind of data? If they cant, we can probbaly use a sun box. what kind of specs would the box need?
tia, tr
Todd - first thought I have is to get a linux box with a gigE port and anything pentium III based or faster. Depending on the amount of analysis you want to do, just running tcpdump to a file and then playback after the fact. Etherman would make for a good UI to review capture in. Should be able to write 250mbps out to a fast drive...wouldn't build a box with that spec with parts from compusa, though. John On Fri, Dec 03, 2004 at 10:47:08AM -0500, todd romero wrote:
does anyone have expirience using a sniffer on a hi-capacity network segment, that might know if there are limitations I need to worry about?
example: customers doing EMC database replication across a mpls link, and when the capacity reaches aprox. 250 Mbp/s packets are arriving out of sequence etc. So we need to put sniffers on both sides to capture some data to see whats happeneing when the capacity reaches 250mbps.
what kind of system requirements would be needed to be able to be able to capture that amount of data. For some reason, I dont think that the Dolch Pac 65 sniffers we have (running nt4 and sniffer pro2) would be able to handle that kind of data? If they cant, we can probbaly use a sun box. what kind of specs would the box need?
tia, tr
Date: Fri, 3 Dec 2004 10:47:08 -0500 (EST) From: todd romero <todd@routeflap.net> To: nanog@nanog.org Subject: using sniffer on high-bandwidth pipes
does anyone have expirience using a sniffer on a hi-capacity network segment, that might know if there are limitations I need to worry about?
example: customers doing EMC database replication across a mpls link, and when the capacity reaches aprox. 250 Mbp/s packets are arriving out of sequence etc. So we need to put sniffers on both sides to capture some data to see whats happeneing when the capacity reaches 250mbps.
Well, there was a nice presentation at SANE 2004 about using Linux with some tweaks... It also compared it model and performance wise with the features available under FreeBSD (4.x IIRC): http://www.nluug.nl/events/sane2004/abstracts/ab.html?id=100 Luca is the man behind NTOP: http://www.ntop.org/ Luca showed that moderate hardware is capable of handling Gb/s speeds at above 90% capture rate if you use the right combination of logic and tools (PF_Ring). In his case a moderate P3 and I believe somewhere upwards of 600Mbps... The goal was mainly to reduce the load of the CPU to allow the machine to actually process the packets it has captured ;) The ntop website has some papers: http://www.ntop.org/documentation.html
tia, tr
Kind Regards, JP Velders
If you want to get serious, check out endace cards... www.endace.com Their cards offload much of the pcap processing to the specialized nic... It is only for sniffing. They manage to do a zero copy directly to memory... You can capture near line rate at gigabit speeds. They are expensive, but worth it for serious monitoring.. Tie this with a fast dell tower that you can put on a cart or a something like a Dell 2950 with lots of disk space and run argus http://www.qosient.com/argus/index.htm Peter todd romero wrote:
does anyone have expirience using a sniffer on a hi-capacity network segment, that might know if there are limitations I need to worry about?
example: customers doing EMC database replication across a mpls link, and when the capacity reaches aprox. 250 Mbp/s packets are arriving out of sequence etc. So we need to put sniffers on both sides to capture some data to see whats happeneing when the capacity reaches 250mbps.
what kind of system requirements would be needed to be able to be able to capture that amount of data. For some reason, I dont think that the Dolch Pac 65 sniffers we have (running nt4 and sniffer pro2) would be able to handle that kind of data? If they cant, we can probbaly use a sun box. what kind of specs would the box need?
tia, tr
participants (8)
-
Alexei Roudnev -
Bruce Pinsky -
Iljitsch van Beijnum -
John Kinsella -
JP Velders -
Peter John Hill -
Steve Francis -
todd romero