In a message written on Thu, Apr 12, 2012 at 05:16:27PM -0400, Maverick wrote:
1) My goal is to store the traffic may be fore ever, and analyze it in the future for security related incidents detected by ids/ips.
Let's just assume you have enough disk space that you can write out every packet, or even just packet header. That's a hard problem, but you've received plenty of suggestions on how to go down that path. Once you have that data, how are you going to process it? Yes, disk reads are faster than disk writes, but not by that much. If it takes you 24 hours to write a day of data to disk, it might take you 12 hours just to read it all back off and process it. Processing a weeks worth of back data could take days. I'm also not even starting to count the CPU and memory necessary to build state tables and statistical analysis tables to generate useful data. There's a reason why most network traffic tools summarize early, as early as on the network device when using Netflow type collection. It's not just to save storage space on disk, but it's to make the processing of the data fast enough that it can be done in a short enough time that the data is still relevant when the processing is complete. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/