collectd as alternative to RTG for high-resolution polling and long term storage?
Would anyone care to share their experience using collectd as an alternative to rtg for high-resolution polling of interface traffic and long term storage? I am investigating the various options for large data set size, lossless long term traffic charting (not RRAs which lose precision over time). One possible use is precision 95th billing. https://collectd.org/
Be aware that collectd itself is a collection agent. It doesn't include (last I checked) a grapher. There are however a number of graphers out there to work with those RRD files, if you use that to store the data. I personally have been using collectd across hundreds of Linux systems, using rrdcached to a central collector and wrote my own grapher for that stuff I am interested int. Ulf. On Wed, Mar 16, 2016 at 11:45 AM, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
-- Ulf Zimmermann, 1525 Pacific Ave., Alameda, CA-94501, #: 510-396-1764 You can find my resume at: http://www.Alameda.net/~ulf/resume.html
Collectd is great, IMHO. I was using collectd+graphite to gather and display stats for a large collection of VMs, servers, routers, and switches. Collectd itself was pretty low overhead, easy to configure (I managed configs via puppet) and Just Worked. Graphite and carbon cache were a little more tricky to set up - carbon by default aggregates/averages older data, so if not setup correctly, when you go back a few months and try to drill into a graph at a 5 minute interval, you get unexpected results. I’d highly recommend looking at Graphite, as well. Once you get used to it, being able to apply functions[1] to aggregate, manipulate, and quickly find patterns in data is super useful (ex: look at all interfaces on this switch, only display graphs for the top 5 abnormal traffic). Jason Dixon has written some great blogs posts about it’s use on obfuscurity.com. John 1: https://graphite.readthedocs.org/en/latest/functions.html
On Mar 16, 2016, at 11:45 AM, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Correct me if I’m wrong, but I believe that collectd uses RRD files for the backend, which you said you don’t want. You might check out Grafana (http://grafana.org/ <http://grafana.org/>). Its based off graphite and uses something like opentsdb or influxdb for the backend. I think this is probably more what you’re looking for. -- Louis Kowolowski louisk@cryptomonkeys.org <mailto:louisk@cryptomonkeys.org> Cryptomonkeys: http://www.cryptomonkeys.com/ <http://www.cryptomonkeys.com/> Making life more interesting for people since 1977
Collectd supports a large number “write” plugins[1] that can write out to various sources. I had been eyeing Grafana and OpenTSDB, they’re probably worth a look John 1: https://collectd.org/wiki/index.php/Table_of_Plugins
On Wed, Mar 16, 2016 at 11:45 AM, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Devices that support sFlow natively implement collectd type functionality for streaming interface counters to a time series database (InfluxDB, Graphite, OpenTSB, etc.) Tools like Grafana can be used to query the database and build dashboards. Host sFlow (http://sflow.net) is very similar to collectd in the metrics it exports, but with the added ability to export flow data from host adapters, bridges, vSwitches, firewalls, routing, VMs, containers etc.
Prometheus is also worth taking a look at. http://prometheus.io/docs/introduction/comparison/ *[image: userimage]Scott Larson[image: los angeles] <https://www.google.com/maps/place/4216+Glencoe+Ave,+Marina+Del+Rey,+CA+90292/@33.9892151,-118.4421334,17z/data=!3m1!4b1!4m2!3m1!1s0x80c2ba88ffae914d:0x14e1d00084d4d09c>Lead Systems Administrator[image: wdlogo] <https://www.wiredrive.com/> [image: linkedin] <https://www.linkedin.com/company/wiredrive> [image: facebook] <https://www.twitter.com/wiredrive> [image: twitter] <https://www.facebook.com/wiredrive> [image: instagram] <https://www.instagram.com/wiredrive>T 310 823 8238 x1106 <310%20823%208238%20x1106> | M 310 904 8818 <310%20904%208818>* On Wed, Mar 16, 2016 at 5:52 PM, Peter Phaal <peter.phaal@gmail.com> wrote:
participants (7)
-
Dale W. Carder
-
Eric Kuhnke
-
John Kinsella
-
Louis Kowolowski
-
Peter Phaal
-
Scott Larson
-
Ulf Zimmermann