NANOG36-NOTES 2006.02.14 talk 2 Netflow Visualization Tools
2006.02.14 talk 2 Netflow tools Bill Yurcik byurcik at ncsa.uiuc.edu NVisionIP and VisFlowConnect-IP probably a dozen tools out there, this is just two of them. Concenses is there's something to this. They're an edge network, comes into ISP domain, their tools are used by entities with many subnet blocks. Overview Project Motifivation Netflows for Security Two visualization tools NVisionIP VisFlowConnect-IP Summary Internet Security: N-Dimensional Work Space large--already lots of data to process complex--combinatorics explode quickly time dynamics--things can change quickly! Visualizations can help! in near-realtime overview-browse-details on demand People are wired to do near-realtime processing of visual information, so that's a good way to present information for humans. HCI says use overview-browse-details paradigm. Netflows for security can identify connection-oriented stats to see things like attacks, DoS, DDoS, etc. Most people don't use the data portion of the flow field, the first 64 bytes, they just look at header info or aggregated flow records. Can spot how many users are on your system at a given time, to schedule upgrades. Who are your top talkers? How long do my users surf? What are people using the network for? Where do users go? Where did they come from? Are users following the security policy? What are the top N destination ports? Is there traffic to vulnerable hosts? Can you identify and block scanners/bad guys? This doesn't replace other systems like syslog, etc.; it integrates and works alongside them. architecture slide for NCSA. Can't really do sampled view for security, so probably need distributed flow collector farm to get all the raw data safely. Two visualization tools: NVisionIP, VisFlowConnect-IP focus on quick overview of tools security.ncsa.uiuc.edu/ 3 level hierarchical tool; galaxy view (small multiple view) ((machine view)) Galaxy is overview of the whole network. color and shape of dots is each host in a network. settable parameters for each dot. Animated toolbar and clock show changes over time in the galaxy. Lets you get high-level content quickly and easily. Domain view lets you drill in a bit more; small multiple view looks at the traffic within the block. upper histogram is lower, well known ports; lower histogram is ports over 1024 You can click on a given multiple view entry to delve into one machine. Many graphs for each machine in the most detailed view. well known ports first, then rest of ports (sorted) then source and destination traffic broken out. Designed for class Bs. http://security.ncsa.uiuc.edu/distribution/VisFlowConnectDownload.html 3 vertical lines, comes from edge network perspective; middle line is edge network to manage. You set range of networks you care about. Outside lines are people sourcing or sinking traffic to you, from outside domains. There's a time axis, traffic only shown for the slice of time currently under consideration. Uses VCR-like controls to move time forward/backward Lets you see traffic/interactivity, drill into that domain, see host level connectivity flows. Shows MS Blaster virus traffic as an example. Example 2, a scan example. Just because it looks like one IP hitting many others doesn't mean it's really a security incident, though; could be a cluster getting traffic. web crawlers hitting NCSA web servers make for a very charateristic pattern over time. Summary Netflows analysis is non-trivial, NVisionIP VisFlowConnect-IP lots of references listed in very fine blue font. http://security.ncsa.uiuc.edu/distribution/NVisionIPDownload Avi Freedman, Akamai, Argus was mentioned a lot; it lets you grab symmetric netflows, but also does TCP analysis, shows some performance data as well. not sure if people are studying the impact of correlating argus data with flow data. Roland Douta? of Cisco; many people are using netflow to track security issues. They now have ingress and egress flow data on many of their platforms. In reading paper describing it, there's data conversion that needs to happen into an internal format that nVision can understand. It reads log files at the moment, takes about 5 minutes to process files. Lets them take different file data sources, make the tool for visualization independent of the input format. They can read large files, but there is a performance hit when doing it. Are they planning on doing further work on the tool to collect TCP flags, for frags, drop traffic, etc? They've looked at it, but they leave it to IDS tools for flag activity. Might be of interest to consider for future versions of the tools. Last question came up, echoed about argus. Question about interactivity, they are working on feedback through tools. Question about alarming on patterns; but once you start alarming or putting up visual indicators, it distracts from rest of the overall pattern, you tend to miss other information.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 thanks for taking notes. comments in-line: Matthew Petach wrote:
2006.02.14 talk 2 Netflow tools
Bill Yurcik byurcik at ncsa.uiuc.edu
NVisionIP and VisFlowConnect-IP
probably a dozen tools out there, this is just two of them. Concenses is there's something to this.
They're an edge network, comes into ISP domain, their tools are used by entities with many subnet blocks.
Overview Project Motifivation Netflows for Security Two visualization tools NVisionIP VisFlowConnect-IP Summary
Internet Security: N-Dimensional Work Space
large--already lots of data to process complex--combinatorics explode quickly time dynamics--things can change quickly! Visualizations can help! in near-realtime overview-browse-details on demand
People are wired to do near-realtime processing of visual information, so that's a good way to present information for humans. HCI says use overview-browse-details paradigm.
Netflows for security can identify connection-oriented stats to see things like attacks, DoS, DDoS, etc. Most people don't use the data portion of the flow field, the first 64 bytes, they just look at header info or aggregated flow records.
Can spot how many users are on your system at a given time, to schedule upgrades.
Who are your top talkers?
How long do my users surf? What are people using the network for?
Where do users go? Where did they come from?
Are users following the security policy?
What are the top N destination ports? Is there traffic to vulnerable hosts?
Can you identify and block scanners/bad guys?
This doesn't replace other systems like syslog, etc.; it integrates and works alongside them.
architecture slide for NCSA.
Can't really do sampled view for security, so probably need distributed flow collector farm to get all the raw data safely.
Two visualization tools: NVisionIP, VisFlowConnect-IP
focus on quick overview of tools security.ncsa.uiuc.edu/
3 level hierarchical tool; galaxy view (small multiple view) ((machine view))
Galaxy is overview of the whole network. color and shape of dots is each host in a network. settable parameters for each dot.
Animated toolbar and clock show changes over time in the galaxy. Lets you get high-level content quickly and easily.
Domain view lets you drill in a bit more; small multiple view looks at the traffic within the block. upper histogram is lower, well known ports; lower histogram is ports over 1024
You can click on a given multiple view entry to delve into one machine. Many graphs for each machine in the most detailed view.
well known ports first, then rest of ports (sorted) then source and destination traffic broken out.
Designed for class Bs.
http://security.ncsa.uiuc.edu/distribution/VisFlowConnectDownload.html
3 vertical lines, comes from edge network perspective; middle line is edge network to manage. You set range of networks you care about. Outside lines are people sourcing or sinking traffic to you, from outside domains.
There's a time axis, traffic only shown for the slice of time currently under consideration. Uses VCR-like controls to move time forward/backward
Lets you see traffic/interactivity, drill into that domain, see host level connectivity flows.
Shows MS Blaster virus traffic as an example.
Example 2, a scan example. Just because it looks like one IP hitting many others doesn't mean it's really a security incident, though; could be a cluster getting traffic.
web crawlers hitting NCSA web servers make for a very charateristic pattern over time.
Summary Netflows analysis is non-trivial,
NVisionIP VisFlowConnect-IP
lots of references listed in very fine blue font.
http://security.ncsa.uiuc.edu/distribution/NVisionIPDownload
Avi Freedman, Akamai, Argus was mentioned a lot; it lets you grab symmetric netflows, but also does TCP analysis, shows some performance data as well. not sure if people are studying the impact of correlating argus data with flow data.
Roland Douta? of Cisco; many people are using netflow to track security issues. They now have ingress and egress flow data on many of their platforms. In reading paper describing it, there's data conversion that needs to happen into an internal format that nVision can understand. It reads log files at the moment, takes about 5 minutes to process files. Lets them take different file data sources, make the tool for visualization independent of the input format. They can read large files, but there is a performance hit when doing it. Are they planning on doing further work on the tool to collect TCP flags, for frags, drop traffic, etc? They've looked at it, but they leave it to IDS tools for flag activity. Might be of interest to consider for future versions of the tools.
Last question came up, echoed about argus. Question about interactivity, they are working on feedback through tools. Question about alarming on patterns; but once you start alarming or putting up visual indicators, it distracts from rest of the overall pattern, you tend to miss other information.
the last part was me, virendra rode from riverdomain. my question was mostly related to a possibility of setting priority bit(s) in order to control (rate-limit, if you will) session(s) that could lead to congestion. since argus is already integrated and performs traffic auditing (i think) setting priority bit(s) would be a nice feature to integrate down the path. then again, i understand this is a performance monitoring tool. that's all. regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFD8qpUpbZvCIJx1bcRAnzaAKCsI29SetdMSJaLr3LR01MGp87CmACgnCEf 7RDnyaGsad++GevXjt2MIQY= =/55T -----END PGP SIGNATURE-----
Roland Dobbins - that's me asking about the time intervals for the bins and the TCP flags stuff. ;> Note that 5-minute bins may not always be optimal for opsec - 5 minutes minimum to see something happening and then 5 minutes to see if your mitigation action was effective is a long time. With NetFlow- based anomaly-detection systems, the active flow timeout value is generally turned down to one minute; the operator may -choose- to suppress certain types of alarms for a set period, or configure threshold-transition delays, but being stuck at a practical minimum of 10 minutes between detection and confirmation of mitigation due to data-conversion overhead (the collected flow telemetry must be converted into another format prior to analysis) may be an issue, in some circumstances. On Feb 14, 2006, at 8:13 PM, Vicky Røde wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
thanks for taking notes.
comments in-line:
Matthew Petach wrote:
2006.02.14 talk 2 Netflow tools
Bill Yurcik byurcik at ncsa.uiuc.edu
NVisionIP and VisFlowConnect-IP
probably a dozen tools out there, this is just two of them. Concenses is there's something to this.
They're an edge network, comes into ISP domain, their tools are used by entities with many subnet blocks.
Overview Project Motifivation Netflows for Security Two visualization tools NVisionIP VisFlowConnect-IP Summary
Internet Security: N-Dimensional Work Space
large--already lots of data to process complex--combinatorics explode quickly time dynamics--things can change quickly! Visualizations can help! in near-realtime overview-browse-details on demand
People are wired to do near-realtime processing of visual information, so that's a good way to present information for humans. HCI says use overview-browse-details paradigm.
Netflows for security can identify connection-oriented stats to see things like attacks, DoS, DDoS, etc. Most people don't use the data portion of the flow field, the first 64 bytes, they just look at header info or aggregated flow records.
Can spot how many users are on your system at a given time, to schedule upgrades.
Who are your top talkers?
How long do my users surf? What are people using the network for?
Where do users go? Where did they come from?
Are users following the security policy?
What are the top N destination ports? Is there traffic to vulnerable hosts?
Can you identify and block scanners/bad guys?
This doesn't replace other systems like syslog, etc.; it integrates and works alongside them.
architecture slide for NCSA.
Can't really do sampled view for security, so probably need distributed flow collector farm to get all the raw data safely.
Two visualization tools: NVisionIP, VisFlowConnect-IP
focus on quick overview of tools security.ncsa.uiuc.edu/
3 level hierarchical tool; galaxy view (small multiple view) ((machine view))
Galaxy is overview of the whole network. color and shape of dots is each host in a network. settable parameters for each dot.
Animated toolbar and clock show changes over time in the galaxy. Lets you get high-level content quickly and easily.
Domain view lets you drill in a bit more; small multiple view looks at the traffic within the block. upper histogram is lower, well known ports; lower histogram is ports over 1024
You can click on a given multiple view entry to delve into one machine. Many graphs for each machine in the most detailed view.
well known ports first, then rest of ports (sorted) then source and destination traffic broken out.
Designed for class Bs.
http://security.ncsa.uiuc.edu/distribution/ VisFlowConnectDownload.html
3 vertical lines, comes from edge network perspective; middle line is edge network to manage. You set range of networks you care about. Outside lines are people sourcing or sinking traffic to you, from outside domains.
There's a time axis, traffic only shown for the slice of time currently under consideration. Uses VCR-like controls to move time forward/backward
Lets you see traffic/interactivity, drill into that domain, see host level connectivity flows.
Shows MS Blaster virus traffic as an example.
Example 2, a scan example. Just because it looks like one IP hitting many others doesn't mean it's really a security incident, though; could be a cluster getting traffic.
web crawlers hitting NCSA web servers make for a very charateristic pattern over time.
Summary Netflows analysis is non-trivial,
NVisionIP VisFlowConnect-IP
lots of references listed in very fine blue font.
http://security.ncsa.uiuc.edu/distribution/NVisionIPDownload
Avi Freedman, Akamai, Argus was mentioned a lot; it lets you grab symmetric netflows, but also does TCP analysis, shows some performance data as well. not sure if people are studying the impact of correlating argus data with flow data.
Roland Douta? of Cisco; many people are using netflow to track security issues. They now have ingress and egress flow data on many of their platforms. In reading paper describing it, there's data conversion that needs to happen into an internal format that nVision can understand. It reads log files at the moment, takes about 5 minutes to process files. Lets them take different file data sources, make the tool for visualization independent of the input format. They can read large files, but there is a performance hit when doing it. Are they planning on doing further work on the tool to collect TCP flags, for frags, drop traffic, etc? They've looked at it, but they leave it to IDS tools for flag activity. Might be of interest to consider for future versions of the tools.
Last question came up, echoed about argus. Question about interactivity, they are working on feedback through tools. Question about alarming on patterns; but once you start alarming or putting up visual indicators, it distracts from rest of the overall pattern, you tend to miss other information.
the last part was me, virendra rode from riverdomain. my question was mostly related to a possibility of setting priority bit(s) in order to control (rate-limit, if you will) session(s) that could lead to congestion.
since argus is already integrated and performs traffic auditing (i think) setting priority bit(s) would be a nice feature to integrate down the path. then again, i understand this is a performance monitoring tool.
that's all.
regards, /virendra
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFD8qpUpbZvCIJx1bcRAnzaAKCsI29SetdMSJaLr3LR01MGp87CmACgnCEf 7RDnyaGsad++GevXjt2MIQY= =/55T -----END PGP SIGNATURE-----
---------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice Everything has been said. But nobody listens. -- Roger Shattuck
participants (3)
-
Matthew Petach -
Roland Dobbins -
Vicky Røde