On Tue, Jun 24, 2008 at 01:19:03PM +1200, Nathan Ward wrote:
I see little point in aggregating tapped traffic, unless you have only a small amount of it and you're doing it to save cost on monitoring network interfaces - but is that saved cost still a saving when you factor in the cost of the extra 3750s in the middle? I'd guess no.
Thanks for all the info Nathan - lots of good leads in your email. Let me include some more information. The problem is finding a way to multiplex that traffic from the optical tap to multiple things that want to peek at it. The remote-span trick solves that, as well as integrating media converters. 3750 is nice since you can stack em up and mix/match the SFP and copper ports. For example - we have an FCP box from Internap. It wants to see mirrored traffic so it can watch for TCP setup problems and try to find blackholes. It takes 10G feeds of aggregated transit links. Then, we want to do some passive IDS analysis. But snort can only really only handle 600-800Mbps before it starts saturating CPU (not multithreaded...) - so one collector per gigE transit seems logical. We'd like to generate flow data out of our forwarding plane since we use 6500s to pull in border transit links. The Netflow on those boxes is terrible. pmacct does a much better job, but it needs to see all the traffic out of band.
Note that for a single GE link, you'd need 2GE of remote span backhaul (one GE in each direction).
We're mostly a content network, very few eyeballs. Our ingress traffic is negligable compared to egress, which makes the problem easier.
Matrix switches aren't useful for your case, as you're talking about monitoring for trending etc. I think. Matrix switches are good when you have lots of links, and want to be able to switch between them. Is the cost of matrix switch ports worth the saving in GE interfaces on PCs?
I guess what made me look at them is their ability to multiplex the stream of data. Take it from an optical tap, spit the same data out of multiple ports. The remote-span trick seems to do the same thing, so I'm wondering where the gotcha is. If there's an advantage to using something like the Matrix switches, I'd love to know that now.
The above is based on the assumption you're using PCs for monitoring, the economics of aggregating tap traffic may make more sense if you're using some fancy monitoring platform.
Yea - the fact that we have both makes the aggregation method look good. The FCP takes 10G aggregated feeds. The PCs will want single gig views of the transit links.
If you find that you need lots of GE interfaces per PC or something, and are saturating the PCI bus, look at DAG cards from Endace. They're designed for passive monitoring, and will send you only headers and do BPF in hardware. I looked at these for a similar project, but didn't bother as it was cheaper to buy more PC chassis' and commodity GE cards. They can do 10GE monitoring, so if you need several 10GE's per chassis I'd recommend these.
Ah the Endace gear looks really interesting. Thanks for the pointer! -- Ross Vandegrift ross@kallisti.us "The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell." --St. Augustine, De Genesi ad Litteram, Book II, xviii, 37