more in-line... On Thu, 2012-04-12 at 17:16 -0400, Maverick wrote:
Thank you very much for your suggestions.
1) My goal is to store the traffic may be fore ever, and analyze it in the future for security related incidents detected by ids/ips.
The poor man's way to do this is to use the space you have and use the -C and -W options in tcpdump. You have as much history as you have disk space. Maybe make 500M files, and a count of 1800 to use 900G of disk space. When you have an event, you copy off the files that are relevant to the time period of the events, to a workstation. Another option is -G for rotating the files by time instead of size.
2) I am storing just header and initial few bytes but still it gets filled up quite quickly.
You can use the -z option to gzip compress the files to save space. However, I don't know how this will affect your disk io... will it be fast enough to keep up with the writing of the raw data and doing a concurrent gzip of the last file. If you have enough hardware performance, but are limited on space, then it's worth a shot.
3) Netflow approach is nice but I also want to have traces available for reasons mentioned in 1).
4) Are there any issues having an external storage as a solution for this problem.
There is also some advice in the man page for tcpdump regarding the -z option. You can write a shell script that takes the capture file as the only argument, to do other stuff you want done... in this case, copy the file off to another drive. It could be a network location too... of course, don't forget to not capture *that* traffic (feedback!).
Best, Ali
On Thu, Apr 12, 2012 at 5:06 PM, Michael J McCafferty <mike@m5computersecurity.com> wrote:
Ali, Do you need to capture the whole packet, including the payload? You will save a lot of space by just capturing the headers. For example, tcpdump doesn't capture the whole packet by default anyway. You may not be able to capture at line rate anyway depending on what you are using to capture with (drivers, libraries, software, etc). See the -s option in tcpdump man page for info.
Good luck, Mike
On Thu, 2012-04-12 at 16:25 -0400, Maverick wrote:
Hello Everyone,
Can you please comment on what is best solution for storing network traffic. We have been graciously granted access by our network administrator to capture traffic but the one Tera byte disk space is no match with the data that we are seeing, so it fills up quickly. We can't get additional space on the server itself so I am looking for some external solutions. Can you please suggest something that would be best for Gbps speeds .
Best, Ali
-- ************************************************************ Michael J. McCafferty CEO M5 Hosting http://www.m5hosting.com
Like us on Facebook for updates and photos: https://www.facebook.com/m5hosting ************************************************************
-- ************************************************************ Michael J. McCafferty CEO M5 Hosting http://www.m5hosting.com Like us on Facebook for updates and photos: https://www.facebook.com/m5hosting ************************************************************