hey there any recommendations on freeware flow analysis tool which can show the flow not only per prefix basis but also show asn and/or country/region as well? Juniper only. feel free to contact on/off list. mehmet
On Sun, Nov 14, 2010 at 12:07:40PM +1000, Mehmet Akcin wrote:
hey there
any recommendations on freeware flow analysis tool which can show the flow not only per prefix basis but also show asn and/or country/region as well? Juniper only.
feel free to contact on/off list.
Juniper's flow export is just like everyone else's (*), so any tool will do the same thing. Country/region analysis would depend on third party geolocation services, which have nothing to do with netflow. :) (*) Well, except M/T/MX only support NetFlow v5/v8 in the free software based sampling mode, you need an expensive services card and software license to do v9 for some reason. Oh and the sFlow on EX is actually pretty cripled when used for routing. It's missing support for a bunch of important extended message tpes, and doesn't fully populate all of the fields of the message types it does send. For example you won't get any data on ASNs, nexthops, dest ifindexes, or even netmasks of the src/dst route the flow matched, making it pretty darn useless for a lot of tasks. It's functional if you're just analyzing L2 networks at any rate. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Sat, Nov 13, 2010 at 09:17:55PM -0600, Richard A Steenbergen wrote:
Oh and the sFlow on EX is actually pretty cripled when used for routing. It's missing support for a bunch of important extended message tpes, and doesn't fully populate all of the fields of the message types it does send. For example you won't get any data on ASNs, nexthops, dest ifindexes, or even netmasks of the src/dst route the flow matched, making it pretty darn useless for a lot of tasks. It's functional if you're just analyzing L2 networks at any rate.
Agree people spend some money and hence tend to expect something in return. But it's also true those good souls developing free collectors (to stay in topic with the OP) sometimes come to the rescue: ASNs, BGP next-hop, routes, netmasks can be all looked up at the collector at pretty no major effort. Variety of methods available depending on the collector, in place or a posteriori, file or BGP lookup - it's matter of selecting what fits better the specific job. Plus, sFlow flow samples are rather successful offsetting some partial vendor implementations by carrying portion of the sampled packet - in one go MAC addresses, VLANs, 802.1p, MPLS labels, EXP bits, BoS, etc. are at the collector doorstep. OTOH it would be nice to see one day those NetFlow v9 MAC address fields populated on higher-grade boxes, say, to facilitate analysis of public peering at internet exchanges ... Cheers, Paolo
On Nov 14, 2010, at 3:59 AM, Paolo Lucente wrote:
OTOH it would be nice to see one day those NetFlow v9 MAC address fields populated on higher-grade boxes, say, to facilitate analysis of public peering at internet exchanges ...
This can be done with mac accounting in your platform. It may require ordering the right PIC in this case, but is easy to collect that data. - Jared
On Sun, Nov 14, 2010 at 09:04:36AM -0500, Jared Mauch wrote:
On Nov 14, 2010, at 3:59 AM, Paolo Lucente wrote:
OTOH it would be nice to see one day those NetFlow v9 MAC address fields populated on higher-grade boxes, say, to facilitate analysis of public peering at internet exchanges ...
This can be done with mac accounting in your platform. It may require ordering the right PIC in this case, but is easy to collect that data.
MAC accounting is not a solution, except for basic things, for at least two reasons: 1) it is coarse-grained and 2) its segregation from flow accounting prevents having all the eggs in one basket. Cheers, Paolo
Hi again, I think many of you got stuck at the "freeware" part however if there are good suggestions you have which would help achieving what I would like to achieve by paying for a software, I am open to listen suggestions on that as well. just thought would mention mehmet On Nov 15, 2010, at 1:12 AM, Paolo Lucente wrote:
On Sun, Nov 14, 2010 at 09:04:36AM -0500, Jared Mauch wrote:
On Nov 14, 2010, at 3:59 AM, Paolo Lucente wrote:
OTOH it would be nice to see one day those NetFlow v9 MAC address fields populated on higher-grade boxes, say, to facilitate analysis of public peering at internet exchanges ...
This can be done with mac accounting in your platform. It may require ordering the right PIC in this case, but is easy to collect that data.
MAC accounting is not a solution, except for basic things, for at least two reasons: 1) it is coarse-grained and 2) its segregation from flow accounting prevents having all the eggs in one basket.
Cheers, Paolo
On Sun, Nov 14, 2010 at 08:59:33AM +0000, Paolo Lucente wrote:
On Sat, Nov 13, 2010 at 09:17:55PM -0600, Richard A Steenbergen wrote:
Oh and the sFlow on EX is actually pretty cripled when used for routing. It's missing support for a bunch of important extended message tpes, and doesn't fully populate all of the fields of the message types it does send. For example you won't get any data on ASNs, nexthops, dest ifindexes, or even netmasks of the src/dst route the flow matched, making it pretty darn useless for a lot of tasks. It's functional if you're just analyzing L2 networks at any rate.
Agree people spend some money and hence tend to expect something in return. But it's also true those good souls developing free collectors (to stay in topic with the OP) sometimes come to the rescue: ASNs, BGP next-hop, routes, netmasks can be all looked up at the collector at pretty no major effort. Variety of methods available depending on the collector, in place or a posteriori, file or BGP lookup - it's matter of selecting what fits better the specific job.
Yes you can do an offline routing lookup to try and reconstruct some missing data (or do some even more interesting analysis, as described in http://www.nanog.org/meetings/nanog35/presentations/steenbergen.pdf), but it isn't always a practical solution to missing netmask, nexthop, and dest ifindex data. Remember that every RIB in your network can and will have a unique best path selection (thanks to the EBGP > IBGP rule if nothing else), and if you have a network of any size at all you'll probably have to deal with multiple exits to the same network. Even if you were only concerned with analyzing external traffic, you'd still need to collect a RIB per edge router using an IBGP feed. In my network this would put you well over 10 million paths, and consume several gigs of ram, not to mention the load of doing the routing lookups themselves. If you wanted to do traffic analysis inside your network you'd need a feed from every router, and maybe even active participation in your IGP. It CAN be done, but it's not pretty, and I don't think any existing free software has been tested under these kinds of conditions. So when a vendor says "we support sFlow", make sure they actually support the message types and fields you need. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Sun, Nov 14, 2010 at 11:32:10AM -0600, Richard A Steenbergen wrote:
On Sun, Nov 14, 2010 at 08:59:33AM +0000, Paolo Lucente wrote:
On Sat, Nov 13, 2010 at 09:17:55PM -0600, Richard A Steenbergen wrote:
Remember that every RIB in your network can and will have a unique best path selection (thanks to the EBGP > IBGP rule if nothing else), and if you have a network of any size at all you'll probably have to deal with multiple exits to the same network. Even if you were only concerned with analyzing external traffic, you'd still need to collect a RIB per edge router using an IBGP feed.
Agree.
In my network this would put you well over 10 million paths, and consume several gigs of ram, not to mention the load of doing the routing lookups themselves.
If the collector is able to keep up with the pace of the export protocol, it will not get killed by a couple of BGP lookups per flow/sample. About the memory figure: reducing information overlap among the RIBs results in storing a full BGP table in few tens of MBs. On 10M paths that translates in less than a GB of memory.
analysis inside your network you'd need a feed from every router, and maybe even active participation in your IGP.
This is separate thing. Relying only on telemetry information for such a task is one of the possible approaches - perhaps the quicker, if enabled ad-hoc for troubleshooting, but not necessarily the smarter.
It CAN be done, but it's not pretty, and I don't think any existing free software has been tested under these kinds of conditions.
Define pretty: is it pretty to move control-plane information over and over via NetFlow or sFlow? But most importantly: why passing the concept challenging conditions is something free software should be run far away from?
So when a vendor says "we support sFlow", make sure they actually support the message types and fields you need. :)
Indeed, agree. Cheers, Paolo
Hi, On Sun, Nov 14, 2010 at 4:17 AM, Richard A Steenbergen <ras@e-gerbil.net> wrote:
On Sun, Nov 14, 2010 at 12:07:40PM +1000, Mehmet Akcin wrote:
any recommendations on freeware flow analysis tool which can show the flow not only per prefix basis but also show asn and/or country/region as well? Juniper only.
feel free to contact on/off list.
Juniper's flow export is just like everyone else's (*), so any tool will do the same thing. Country/region analysis would depend on third party geolocation services, which have nothing to do with netflow. :)
(*) Well, except M/T/MX only support NetFlow v5/v8 in the free software based sampling mode, you need an expensive services card and software license to do v9 for some reason.
Shouldn't there be a (**) (**) Also Except for MX'es with trio chipsets. These can do inline-jflow that export to IPFIX (modified netflow v9) All of the open source collector solutions I've tried that can handle v9 cannot handle IPFIX from the trio cards. Richard; Do you have something that handles IPFIX? Bas
On Tue, Nov 16, 2010 at 12:33:37AM +0100, bas wrote:
Shouldn't there be a (**)
(**) Also Except for MX'es with trio chipsets. These can do inline-jflow that export to IPFIX (modified netflow v9)
All of the open source collector solutions I've tried that can handle v9 cannot handle IPFIX from the trio cards.
Richard; Do you have something that handles IPFIX?
Yes there's that too. I haven't actually gotten around to testing the Trio specific Netflow capabilities yet, but supposedly they only support IPFIX when using the "built in" sampling capabilities. If you want v9 you'll still need a Multiservice DPC, or you can always stick to classic RE-sampled v5/v8. IPFIX is effectively "netflow v10", it's largely based off of v9, but it's just different enough to be incompatible. Of course it's close enough that it shouldn't be THAT much work if you already have an existing v9 parser, but I don't know what software actually supports it today. The only flow collector implementation which I've spent any amount of time looking at besides the stuff I've written myself is pmacct, which IMHO shows great promise, but I don't believe it supports IPFIX yet. For my purposes I'd have been just as happy if everyone had standardized on sFlow (especially since I already wrote a parser for it :P), but alas it isn't meant to be. Some differences between v9 and IPFIX that googling turned up: http://www.plixer.com/blog/netflow/what-is-ipfix-vs-netflow-v9/ -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Mon, Nov 15, 2010 at 06:45:06PM -0600, Richard A Steenbergen wrote:
today. The only flow collector implementation which I've spent any amount of time looking at besides the stuff I've written myself is pmacct, which IMHO shows great promise, but I don't believe it supports IPFIX yet.
True: IPFIX is work in progress at the moment and expected to converge to a public release soon (probably it will be in the CVS by end of the year). This is thanks to somebody on-list which is kindly offering lab access to traffic generator, Juniper MX chassis and Trio card. Cheers, Paolo
Mehmet Akcin (mehmet) writes:
hey there
any recommendations on freeware flow analysis tool which can show the flow not only per prefix basis but also show asn and/or country/region as well? Juniper only.
Hi Mehmet, As someone else answered, export v9 flows and then run something like NFsen or Netflow Dashboard. On that, Geo::IP will provide country/region info (for example): http://search.cpan.org/~borisz/Geo-IP-1.38/lib/Geo/IP.pm Cheers, Phil
participants (6)
-
bas -
Jared Mauch -
Mehmet Akcin -
Paolo Lucente -
Phil Regnauld -
Richard A Steenbergen