Techniques for passive traffic capturing
Hello everyone, Over the past two years, there's been a trend toward doing more and more analysis and reporting based on passive traffic analysis. We started out using SPAN sessions to produce an extra copy of all of our transit links for these purposes. But the Cisco limits of two SPAN sessions per device (on our platforms) is a major limitation. Does anyone have a better soultion for more flexible data collection? I've been thinking about a move to a system based on optical taps of each of the links. I'd aggregate these links into something like a 3750 and use remote-span VLANs to pass the traffic onto servers that sniffing on their interface on that 3750. Do products like the NetOptics Matrix Switches offer a substantial advantage? Comments or suggestions? -- Ross Vandegrift ross@kallisti.us "The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell." --St. Augustine, De Genesi ad Litteram, Book II, xviii, 37
On 24/06/2008, at 8:32 AM, Ross Vandegrift wrote:
I've been thinking about a move to a system based on optical taps of each of the links. I'd aggregate these links into something like a 3750 and use remote-span VLANs to pass the traffic onto servers that sniffing on their interface on that 3750. Do products like the NetOptics Matrix Switches offer a substantial advantage?
Comments or suggestions?
<braindump> I see little point in aggregating tapped traffic, unless you have only a small amount of it and you're doing it to save cost on monitoring network interfaces - but is that saved cost still a saving when you factor in the cost of the extra 3750s in the middle? I'd guess no. Depending on how well saturated your circuits are, get double- or quad- GE network cards (Intel make some fixed ones, there are others that take SFPs and fat GBICs) and plug them directly in to the optical taps. If you need your monitoring equipment a distance from the optical taps, use netoptic's regeneration taps, which split 70/30 and then amplify the 30 before sending to your equipment on a different floor/whatever. There are other vendors, I like netoptics because they have cute purple optical patch leads, provide per-tap specs as tested at the factory, and they all worked beautifully out of the box - another vendor had a 50% failure rate, I've forgotten who they were though. A PC with 4 GE optical ports is much simpler and probably more cost effective than doing remote span complications. Note that for a single GE link, you'd need 2GE of remote span backhaul (one GE in each direction). Matrix switches aren't useful for your case, as you're talking about monitoring for trending etc. I think. Matrix switches are good when you have lots of links, and want to be able to switch between them. Is the cost of matrix switch ports worth the saving in GE interfaces on PCs? Netoptics have taps that aggregate several links in to one monitoring feed. Not really cost effective when the cost of a single GE network interface for a PC is so low. The above is based on the assumption you're using PCs for monitoring, the economics of aggregating tap traffic may make more sense if you're using some fancy monitoring platform. If you find that you need lots of GE interfaces per PC or something, and are saturating the PCI bus, look at DAG cards from Endace. They're designed for passive monitoring, and will send you only headers and do BPF in hardware. I looked at these for a similar project, but didn't bother as it was cheaper to buy more PC chassis' and commodity GE cards. They can do 10GE monitoring, so if you need several 10GE's per chassis I'd recommend these. </braindump> -- Nathan Ward
On Tue, Jun 24, 2008 at 01:19:03PM +1200, Nathan Ward wrote:
I see little point in aggregating tapped traffic, unless you have only a small amount of it and you're doing it to save cost on monitoring network interfaces - but is that saved cost still a saving when you factor in the cost of the extra 3750s in the middle? I'd guess no.
Thanks for all the info Nathan - lots of good leads in your email. Let me include some more information. The problem is finding a way to multiplex that traffic from the optical tap to multiple things that want to peek at it. The remote-span trick solves that, as well as integrating media converters. 3750 is nice since you can stack em up and mix/match the SFP and copper ports. For example - we have an FCP box from Internap. It wants to see mirrored traffic so it can watch for TCP setup problems and try to find blackholes. It takes 10G feeds of aggregated transit links. Then, we want to do some passive IDS analysis. But snort can only really only handle 600-800Mbps before it starts saturating CPU (not multithreaded...) - so one collector per gigE transit seems logical. We'd like to generate flow data out of our forwarding plane since we use 6500s to pull in border transit links. The Netflow on those boxes is terrible. pmacct does a much better job, but it needs to see all the traffic out of band.
Note that for a single GE link, you'd need 2GE of remote span backhaul (one GE in each direction).
We're mostly a content network, very few eyeballs. Our ingress traffic is negligable compared to egress, which makes the problem easier.
Matrix switches aren't useful for your case, as you're talking about monitoring for trending etc. I think. Matrix switches are good when you have lots of links, and want to be able to switch between them. Is the cost of matrix switch ports worth the saving in GE interfaces on PCs?
I guess what made me look at them is their ability to multiplex the stream of data. Take it from an optical tap, spit the same data out of multiple ports. The remote-span trick seems to do the same thing, so I'm wondering where the gotcha is. If there's an advantage to using something like the Matrix switches, I'd love to know that now.
The above is based on the assumption you're using PCs for monitoring, the economics of aggregating tap traffic may make more sense if you're using some fancy monitoring platform.
Yea - the fact that we have both makes the aggregation method look good. The FCP takes 10G aggregated feeds. The PCs will want single gig views of the transit links.
If you find that you need lots of GE interfaces per PC or something, and are saturating the PCI bus, look at DAG cards from Endace. They're designed for passive monitoring, and will send you only headers and do BPF in hardware. I looked at these for a similar project, but didn't bother as it was cheaper to buy more PC chassis' and commodity GE cards. They can do 10GE monitoring, so if you need several 10GE's per chassis I'd recommend these.
Ah the Endace gear looks really interesting. Thanks for the pointer! -- Ross Vandegrift ross@kallisti.us "The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell." --St. Augustine, De Genesi ad Litteram, Book II, xviii, 37
Ross Vandegrift <ross <at> kallisti.us> writes:
On Tue, Jun 24, 2008 at 01:19:03PM +1200, Nathan Ward wrote:
I see little point in aggregating tapped traffic, unless you have only a small amount of it and you're doing it to save cost on monitoring network interfaces - but is that saved cost still a saving when you factor in the cost of the extra 3750s in the middle? I'd guess no.
Thanks for all the info Nathan - lots of good leads in your email. Let me include some more information.
The problem is finding a way to multiplex that traffic from the optical tap to multiple things that want to peek at it. The remote-span trick solves that, as well as integrating media converters. 3750 is nice since you can stack em up and mix/match the SFP and copper ports.
http://www.gigamon.com. Taps+MultiPlexing+Filtering+Clustering+10g. I've been using them very successfully for exactly what you describe for the last 2 years. If they are a bit too pricey, look at http://www.vssmonitoring.com. Similar capabilities to Gigamon, slightly less flexibility (fixed hardware configurations vs Gigamon's modular configuration) and possibly cheaper depending on your needs.
We started out with SPAN ports, then moved on to Netoptics taps. Lately we've been using a combination of Cisco Netflow (from remote routers), and native Argus flows (from local taps) where we need more details. Flows are useful to answer "What happened X minutes/hours/days ago?", and where you do not need/want to capture full packet bodies (though with Argus you can choose whether to include payload data). http://qosient.com/argus/
On Mon, Jun 23, 2008 at 10:00:06PM -0500, Kevin Kadow wrote:
We started out with SPAN ports, then moved on to Netoptics taps.
Lately we've been using a combination of Cisco Netflow (from remote routers), and native Argus flows (from local taps) where we need more details.
Flows are useful to answer "What happened X minutes/hours/days ago?", and where you do not need/want to capture full packet bodies (though with Argus you can choose whether to include payload data).
Cool - good to know that the Netoptics gear is good. Seems like there's a few resounding approvals of them. Netflow would be lovely to export from our border routers. Unfortunately, we are somewhat married to the 6500 platform which has absolutely awful netflow support. Very small TCAM, export is CPU expensive, and sampling makes both problems worse. So a mirrored copy of the transit link is being sent to a pmacct box for flow generation. -- Ross Vandegrift ross@kallisti.us "The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell." --St. Augustine, De Genesi ad Litteram, Book II, xviii, 37
I stumbled across these last night. http://www.dovebid.com/assets/display.asp?ItemID=cne11811 I don't know anything about them and haven't done any research. The auction description would however lead me to believe that they might be useful in this case. There are many of them listed in the main auction catalog. Justin Ross Vandegrift wrote:
Hello everyone,
Over the past two years, there's been a trend toward doing more and more analysis and reporting based on passive traffic analysis.
We started out using SPAN sessions to produce an extra copy of all of our transit links for these purposes. But the Cisco limits of two SPAN sessions per device (on our platforms) is a major limitation.
Does anyone have a better soultion for more flexible data collection?
I've been thinking about a move to a system based on optical taps of each of the links. I'd aggregate these links into something like a 3750 and use remote-span VLANs to pass the traffic onto servers that sniffing on their interface on that 3750. Do products like the NetOptics Matrix Switches offer a substantial advantage?
Comments or suggestions?
participants (5)
-
Justin Shore -
Kevin Kadow -
Matt Cable -
Nathan Ward -
Ross Vandegrift