Hello, I am trying to collect traffic traffic from pcap file and store it in a database but really confused how to organize it. Should I organize it on connection basis/ flow basis or IP basis. It might be an effort to write a customized traffic analysis tool like wireshark with only required functionality. I would really appreciate if someone can give me direction on write way of organizing the data because right now I only see individual packets and no way of putting them in some order. Best, Ali
On 2012-02-23 21:11 , Maverick wrote:
Hello,
I am trying to collect traffic traffic from pcap file and store it in a database but really confused how to organize it. Should I organize it on connection basis/ flow basis or IP basis.
It might be an effort to write a customized traffic analysis tool like wireshark with only required functionality. I would really appreciate if someone can give me direction on write way of organizing the data because right now I only see individual packets and no way of putting them in some order.
Does this all not completely depend on what you actually want to do with it? You might want to start there instead of the other way around. Greets, Jeroen
I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it. Best, Ali On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
On 2012-02-23 21:11 , Maverick wrote:
Hello,
I am trying to collect traffic traffic from pcap file and store it in a database but really confused how to organize it. Should I organize it on connection basis/ flow basis or IP basis.
It might be an effort to write a customized traffic analysis tool like wireshark with only required functionality. I would really appreciate if someone can give me direction on write way of organizing the data because right now I only see individual packets and no way of putting them in some order.
Does this all not completely depend on what you actually want to do with it? You might want to start there instead of the other way around.
Greets, Jeroen
Netflow + netflow collector. Ken Matlock Network Analyst Systems and Technology Service Center Sisters of Charity of Leavenworth Health System 12600 W. Colfax, Suite A-500 Lakewood, CO 80215 303-467-4671 matlockk@exempla.org -----Original Message----- From: Maverick [mailto:myeaddress@gmail.com] Sent: Thursday, February 23, 2012 1:19 PM To: Jeroen Massar Cc: nanog@nanog.org Subject: Re: Network Traffic Collection I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it. Best, Ali On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
On 2012-02-23 21:11 , Maverick wrote:
Hello,
I am trying to collect traffic traffic from pcap file and store it in a database but really confused how to organize it. Should I organize it on connection basis/ flow basis or IP basis.
It might be an effort to write a customized traffic analysis tool like wireshark with only required functionality. I would really appreciate if someone can give me direction on write way of organizing the data because right now I only see individual packets and no way of putting them in some order.
Does this all not completely depend on what you actually want to do with it? You might want to start there instead of the other way around.
Greets, Jeroen
*** Exempla Confidentiality Notice *** The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any other dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify me immediately by replying to the message and deleting it from your computer. Thank you. *** Exempla Confidentiality Notice ***
On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L <MatlockK@exempla.org> wrote:
Netflow + netflow collector.
+1 This guide should give you a good start. http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf Regards -- Mukom Akong Tamon ______________ "If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath. Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one way street to IRRELEVANCE." [In Search of Excellence & Perfection] - http://perfexcellence.org [Moments of TechXcellence] - http://techexcellence.net [ICT Business Integration] - http://ibiztech.wordpress.com [About Me] - http://about.me/perfexcellence
Thanks Mukom for the wonderful guide, this is really helpful. I have few questions about ntop though. How can I get access to the log files generated by ntop and do my own parsing rather than looking for webbased results that are generated. Are there any programs available that do parsing of ntops log files. When I run ntop on pcap I don't get the throughput graphs as rrd doesn't work on pcap is there any work around for that. Thanks, Ali On Sat, Feb 25, 2012 at 2:27 AM, Mukom Akong T. <mukom.tamon@gmail.com> wrote:
On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L <MatlockK@exempla.org> wrote:
Netflow + netflow collector.
+1 This guide should give you a good start.
http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf
Regards
-- Mukom Akong Tamon ______________
"If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath. Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one way street to IRRELEVANCE."
[In Search of Excellence & Perfection] - http://perfexcellence.org [Moments of TechXcellence] - http://techexcellence.net [ICT Business Integration] - http://ibiztech.wordpress.com [About Me] - http://about.me/perfexcellence
Hi Ali On Sat, Feb 25, 2012 at 6:14 PM, Maverick <myeaddress@gmail.com> wrote:
Thanks Mukom for the wonderful guide, this is really helpful. I have few questions about ntop though.
How can I get access to the log files generated by ntop and do my own parsing rather than looking for webbased results that are generated.
It's been a while i looked under the hood of ntop. Remember that ntop itself usually needs to be 'fed' traffic to analyse. I have never done it myself but if I needed the raw data, I'd mirror a port and capture it with tcpdump into a pcap file (watch disk space!!) the use whatever analysis tool suits my needs to look at it.
Are there any programs available that do parsing of ntops log files. When I run ntop on pcap I don't get the throughput graphs as rrd doesn't work on pcap is there any work around for that.
Not to my knowledge no. I think there's a switch (-f) for reading data from a pcap file as opposed to a live feed. I have never played with that as well. There are other (possible more feature laden) commercial flow collectors and analysers out there). I also started following trisul earlier on in the project, you might want to check it out.
Thanks, Ali
On Sat, Feb 25, 2012 at 2:27 AM, Mukom Akong T. <mukom.tamon@gmail.com> wrote:
On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L <MatlockK@exempla.org> wrote:
Netflow + netflow collector.
+1 This guide should give you a good start.
http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf
Regards
-- Mukom Akong Tamon ______________
"If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath. Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one way street to IRRELEVANCE."
[In Search of Excellence & Perfection] - http://perfexcellence.org [Moments of TechXcellence] - http://techexcellence.net [ICT Business Integration] - http://ibiztech.wordpress.com [About Me] - http://about.me/perfexcellence
-- Mukom Akong [Tamon] ______________ “We don't LIVE in order to BREATH. Similarly WORKING in order to make MONEY puts us on a one way street to irrelevance.“ [In Search of Excellence & Perfection] - http://perfexcellence.org [Moments of TechXcellence] - http://techexcellence.net [ICT Business Integration] - http://ibiztech.wordpress.com [About Me] - http://about.me/perfexcellence
On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress@gmail.com> wrote:
I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.
Run argus on a span port. -Suresh
Random thought, anyone ever used Splunk for this kind of thing? -mike Sent from my iPhone On Feb 23, 2012, at 10:30, Suresh Rajagopalan <sraja97@gmail.com> wrote:
On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress@gmail.com> wrote:
I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.
Run argus on a span port.
-Suresh
Splunk is an amazing tool and did an awesome thing and introduced a free license in 4.3. I'm using it at two sites now and I'm loving it! On 2012-02-23, at 3:34 PM, Mike Lyon wrote:
Random thought, anyone ever used Splunk for this kind of thing?
-mike
Sent from my iPhone
On Feb 23, 2012, at 10:30, Suresh Rajagopalan <sraja97@gmail.com> wrote:
On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress@gmail.com> wrote:
I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.
Run argus on a span port.
-Suresh
On 2012-02-23 21:34 , Mike Lyon wrote:
Random thought, anyone ever used Splunk for this kind of thing?
Various folks have, the problem of course comes down to processing power, thus you'll need to throw a lot of hardware against it to be able to process traffic in a decent network. Check http://www.raffy.ch/ and http://pixlcloud.com/ etc for more details about this. Greets, Jeroen
Run it with hadoop in EC2? Sent from my iPhone On Feb 23, 2012, at 10:52, Jeroen Massar <jeroen@unfix.org> wrote:
On 2012-02-23 21:34 , Mike Lyon wrote:
Random thought, anyone ever used Splunk for this kind of thing?
Various folks have, the problem of course comes down to processing power, thus you'll need to throw a lot of hardware against it to be able to process traffic in a decent network.
Check http://www.raffy.ch/ and http://pixlcloud.com/ etc for more details about this.
Greets, Jeroen
On Thu, 23 Feb 2012, Maverick wrote:
I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.
Truth is that most people probably don't do it, beyond temporary, ad-hoc deployments, to solve a specific problem at a specific point in time. Traffic capture and analysis doesn't scale too well into multi-Gb/s service provider environments. Netflow tools are an option if 'reasonably accurate' is good enough for your needs. jms
On Thu, Feb 23, 2012 at 1:59 PM, Justin M. Streiner <streiner@cluebyfour.org> wrote:
On Thu, 23 Feb 2012, Maverick wrote:
I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.
Truth is that most people probably don't do it, beyond temporary, ad-hoc deployments, to solve a specific problem at a specific point in time. Traffic capture and analysis doesn't scale too well into multi-Gb/s service provider environments.
Netflow tools are an option if 'reasonably accurate' is good enough for your needs.
jms
For high speed switched Ethernet environments, consider using sFlow. You can treat sFlow as remote packet capture and use Wireshark/tcpdump for troubleshooting network traffic: http://blog.sflow.com/2011/11/wireshark.html Or use sFlow reporting tools to find IP sources, protocols etc.: http://sflow.org/products/collectors.php Which tool to choose depends on your requirements.
Netflow / Sflow with one of the fallowing software packages http://www.plixer.com/products/netflow-sflow/scrutinizer-netflow-sflow.php http://www.solarwinds.com/NetFlow http://www.arbornetworks.com/ Or the hand full of other open source options out there. Carlos Alcantar Race Communications / Race Team Member 101 Haskins Way, So. San Francisco, CA. 94080 Phone: +1 415 376 3314 / carlos@race.com / http://www.race.com -----Original Message----- From: Maverick <myeaddress@gmail.com> Date: Thu, 23 Feb 2012 15:19:24 -0500 To: Jeroen Massar <jeroen@unfix.org> Cc: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: Network Traffic Collection I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it. Best, Ali On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
On 2012-02-23 21:11 , Maverick wrote:
Hello,
I am trying to collect traffic traffic from pcap file and store it in a database but really confused how to organize it. Should I organize it on connection basis/ flow basis or IP basis.
It might be an effort to write a customized traffic analysis tool like wireshark with only required functionality. I would really appreciate if someone can give me direction on write way of organizing the data because right now I only see individual packets and no way of putting them in some order.
Does this all not completely depend on what you actually want to do with it? You might want to start there instead of the other way around.
Greets, Jeroen
PCAP is not well suited to what you describe. Most people use Sflow/Cflow/... instead. Owen On Feb 23, 2012, at 12:19 PM, Maverick wrote:
I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.
Best, Ali
On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
On 2012-02-23 21:11 , Maverick wrote:
Hello,
I am trying to collect traffic traffic from pcap file and store it in a database but really confused how to organize it. Should I organize it on connection basis/ flow basis or IP basis.
It might be an effort to write a customized traffic analysis tool like wireshark with only required functionality. I would really appreciate if someone can give me direction on write way of organizing the data because right now I only see individual packets and no way of putting them in some order.
Does this all not completely depend on what you actually want to do with it? You might want to start there instead of the other way around.
Greets, Jeroen
participants (11)
-
Carlos Alcantar -
Jason Lixfeld -
Jeroen Massar -
Justin M. Streiner -
Matlock, Kenneth L -
Maverick -
Mike Lyon -
Mukom Akong T. -
Owen DeLong -
Peter Phaal -
Suresh Rajagopalan