Hi List, Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across. (Open-Source only please) TIA, M
Hi List,
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
(Open-Source only please)
TIA, M If you're comfortable with Python, Graphite is gaining some serious
On 02/18/2011 09:13 AM, Max Pierson wrote: traction http://graphite.wikidot.com/ Paul
Thomas Boutrell's 'GD'. http://www.libgd.org/Main_Page Joe From: Max Pierson <nmaxpierson@gmail.com> To: nanog group <nanog@nanog.org> Date: 02/18/2011 02:15 PM Subject: Graph Utils (Open-Source) Hi List, Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across. (Open-Source only please) TIA, M
On Fri, 18 Feb 2011, Max Pierson wrote:
Hi List,
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the
we use both gd (in php and in perl and in c++) and google's graphing magic in various places. http://code.google.com/apis/chart/ -- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html
I've used gnuplot for several projects and found it very flexible. Gnuplot is also handy because it's easy to feed it commands over a pipe. I also recommend the "Gnuplot In Action" book - it saved me a ton of time. I have also used matplotlib within Python. For more interactive graphs I've played with the Processing environment a bit, but not enough to provide a useful comparison with the other tools. Peter On Feb 18, 2011, at 11:13 AM, Max Pierson wrote:
Hi List,
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
(Open-Source only please)
TIA, M
Mostly I've heard bad things about matplotlib under Python. Lots of good features, but buggy and a bit of a memory hog. How did you find it? On 02/18/2011 10:34 AM, Peter A. Friend wrote:
I've used gnuplot for several projects and found it very flexible. Gnuplot is also handy because it's easy to feed it commands over a pipe. I also recommend the "Gnuplot In Action" book - it saved me a ton of time. I have also used matplotlib within Python.
For more interactive graphs I've played with the Processing environment a bit, but not enough to provide a useful comparison with the other tools.
Peter
On Feb 18, 2011, at 11:13 AM, Max Pierson wrote:
Hi List,
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
(Open-Source only please)
TIA, M
On Fri, 18 Feb 2011, Max Pierson wrote:
hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
What's wrong with GNUplot? I used it to do graphing of dialup server port utilization in some CGI I did back in the mid 90s. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
What's wrong with GNUplot?
Nothing at all :) My problem is with rrdtool. It doesn't scale for this project. I was looking into GNUplot, but wanted to see what else was out there as well. Thanks for all of the on and off list replies so far. I'll follow up after test driving a few of the mentioned tools. On Fri, Feb 18, 2011 at 2:36 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Fri, 18 Feb 2011, Max Pierson wrote:
hacks of course :). So really what i'm looking for is something along the
lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
What's wrong with GNUplot? I used it to do graphing of dialup server port utilization in some CGI I did back in the mid 90s.
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On 2011-02-18 22:03, Max Pierson wrote:
Nothing at all :) My problem is with rrdtool. It doesn't scale for this project. I was looking into GNUplot, but wanted to see what else was out there as well. Is scaling of rrdtool still a problem for you with rrdcached? http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html
Cheers, René
Even with rrdcached, the I/O from many RRD files being updated often will hammer the I/O subsydtem of most hosts :) We have a host with around 50k RRD data files and rrdcached running, most are updated every 5 mins, some every minute (Nagios + PNP) - with RAID 10 and 10k rpm disks the io wait on the system hangs out at about 15-20 pct, while load hangs around 1-2. I like rrd a lot but scaling it is hard. Max On 2/19/11, Rene Skjoldmose <rene.skjoldmose@gmail.com> wrote:
On 2011-02-18 22:03, Max Pierson wrote:
Nothing at all :) My problem is with rrdtool. It doesn't scale for this project. I was looking into GNUplot, but wanted to see what else was out there as well. Is scaling of rrdtool still a problem for you with rrdcached? http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html
Cheers, René
Twitter is releasing a high volume metrics collection store based on cassandra as open source soon - if you will be scaling big, might be worth looking into. On 2/19/11, Max <perldork@webwizarddesign.com> wrote:
Even with rrdcached, the I/O from many RRD files being updated often will hammer the I/O subsydtem of most hosts :)
We have a host with around 50k RRD data files and rrdcached running, most are updated every 5 mins, some every minute (Nagios + PNP) - with RAID 10 and 10k rpm disks the io wait on the system hangs out at about 15-20 pct, while load hangs around 1-2.
I like rrd a lot but scaling it is hard.
Max
On 2/19/11, Rene Skjoldmose <rene.skjoldmose@gmail.com> wrote:
On 2011-02-18 22:03, Max Pierson wrote:
Nothing at all :) My problem is with rrdtool. It doesn't scale for this project. I was looking into GNUplot, but wanted to see what else was out there as well. Is scaling of rrdtool still a problem for you with rrdcached? http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html
Cheers, René
Is scaling of rrdtool still a problem for you with rrdcached?
This helps on some of my network/server related graphs, but this data is not exactly time based (well timestamps are recorded, but not at cyclic intervals). Plus the dataset is extremely large (100's of millions or rows already in mySQL). This isn't really network or server related metrics i'm trying to plot. Regards, Max On Sat, Feb 19, 2011 at 5:53 PM, Rene Skjoldmose <rene.skjoldmose@gmail.com>wrote:
On 2011-02-18 22:03, Max Pierson wrote:
Nothing at all :) My problem is with rrdtool. It doesn't scale for this project. I was looking into GNUplot, but wanted to see what else was out there as well.
Is scaling of rrdtool still a problem for you with rrdcached? http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html
Cheers, René
On Fri, Feb 18, 2011 at 01:13:54PM -0600, Max Pierson wrote:
Hi List,
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
If you're searching something like Gnuplot, you could also try GNU R: http://www.r-project.org/ HTH, -- Marco Delaurenti http://about.me/hiraedd {twitter, flickr, posterous}/hiraedd
On 02/18/2011 05:32 PM, Marco F. Delaurenti wrote:
On Fri, Feb 18, 2011 at 01:13:54PM -0600, Max Pierson wrote:
Hi List,
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
If you're searching something like Gnuplot, you could also try GNU R: http://www.r-project.org/
Also qtiplot an SciDAVIS. Not to mention others such as found on the exhaustive list: http://en.wikipedia.org/wiki/List_of_graphing_software
Not to mention others such as found on the exhaustive list:
wow, layer 8 issue on my part :) should have checked wiki before I posted. Thanks Jim and thanks for all the feedback. I believe a combination of Graph.pm and some Perl/PHP foo will do nicely. M On Fri, Feb 18, 2011 at 6:19 PM, Jim Gettys <jg@freedesktop.org> wrote:
On 02/18/2011 05:32 PM, Marco F. Delaurenti wrote:
On Fri, Feb 18, 2011 at 01:13:54PM -0600, Max Pierson wrote:
Hi List,
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
If you're searching something like Gnuplot, you could also try GNU R: http://www.r-project.org/
Also qtiplot an SciDAVIS.
Not to mention others such as found on the exhaustive list:
On Fri, Feb 18, 2011 at 1:13 PM, Max Pierson <nmaxpierson@gmail.com> wrote:
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share
I haven't heard of gnuplot used often with other software as a framework for graphing/visualizations. For simple visualizations, I think usually a 'native' framework/API is preferred, e.g. JGraph for java apps. I suspect one reason gnuplot is not used as widely as it could be otherwise is, its licensing is not as "friendly" as other graphics frameworks. gnuplot license is GPL incompatible and does not seem to even fully meet the open source definition, Because redistributing complete modified source code of gnuplot itself is not allowed by the license; a clear reading of gnuplot license suggests only patches, unmodified source code, can be freely redistributed, redistributed binaries based on modified source have special rules). Aside from that caveat, which most likely does not normally impair private use by a network operator: gnuplot is a really good tool. If you need to paint a bunch of arbitrary X and Y values on a graph from an input file or based on simple equations, gnuplot will happily oblige; it can handle chart types rrdtool cannot, and you have more direct control of output. If you want some 3D / surface graphs, RRDTool won't do it, anyways. Gnuplot's less expensive than Matlab / Maple. You can even set terminal type to "dumb" in gnuplot, and generate some fancy ASCII art graphs on stdout. In regards to scalability... About the millions of rows... err.. Try plotting a test dataset with 500 million datapoints. Chances are gnuplot won't necessarily scale that well either, and you need some method to be selective of which rows are provided as input to the plotting framework, in that case. If you have a million datapoints on your X axis, each X position is smaller than 1/1000 of a display pixel (on a graph that fits on a display at say 1920x1080); displaying such high resolution of all datapoints at once on the unzoomed graph is beyond the display hardware capabilitiy. there should normally be some form of averaging / smoothing / "selection of points" contemplated, if the dataset is huge
experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
(Open-Source only please)
-- -JH
On 02/20/11 23:45, Jimmy Hess wrote:
On Fri, Feb 18, 2011 at 1:13 PM, Max Pierson <nmaxpierson@gmail.com> wrote:
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share
I haven't heard of gnuplot used often with other software as a framework for graphing/visualizations. For simple visualizations, I think usually a 'native' framework/API is preferred, e.g. JGraph for java apps.
I suspect one reason gnuplot is not used as widely as it could be otherwise is, its licensing is not as "friendly" as other graphics frameworks. gnuplot license is GPL incompatible and does not seem to even fully meet the open source definition,
Because redistributing complete modified source code of gnuplot itself is not allowed by the license; a clear reading of gnuplot license suggests only patches, unmodified source code, can be freely redistributed, redistributed binaries based on modified source have special rules).
Aside from that caveat, which most likely does not normally impair private use by a network operator: gnuplot is a really good tool. If you need to paint a bunch of arbitrary X and Y values on a graph from an input file or based on simple equations, gnuplot will happily oblige; it can handle chart types rrdtool cannot, and you have more direct control of output.
If you want some 3D / surface graphs, RRDTool won't do it, anyways. Gnuplot's less expensive than Matlab / Maple.
You can even set terminal type to "dumb" in gnuplot, and generate some fancy ASCII art graphs on stdout.
In regards to scalability...
About the millions of rows... err.. Try plotting a test dataset with 500 million datapoints. Chances are gnuplot won't necessarily scale that well either, and you need some method to be selective of which rows are provided as input to the plotting framework, in that case.
If you have a million datapoints on your X axis, each X position is smaller than 1/1000 of a display pixel (on a graph that fits on a display at say 1920x1080); displaying such high resolution of all datapoints at once on the unzoomed graph is beyond the display hardware capabilitiy.
there should normally be some form of averaging / smoothing / "selection of points" contemplated, if the dataset is huge
experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
(Open-Source only please)
-- -JH
If you can use Java, JfreeChart is pretty nice. It has the ability to create many different types of charts/grpahs. I've only used it a little bit for a project I was looking into that uses it, but it seems really capable. I think it's licensed under gpl or lgpl, but the creator charges for documentation/examples if you need them. http://www.jfree.org/jfreechart/
For simple visualizations, I think usually a 'native' framework/API is
Hiya Jimmy!! How has it been? preferred, e.g. JGraph for java apps. Unfortunately, I'm not savvy with Java at all, so the really cool viz API's wont work for me (there's just something about Java ... I simply can't get into it and I see alot of Java based apps that are resource hogs). I was looking at mostly using some simple Perl + PHP (or even Python) for the graph generation. My own cacti if you will, just not as feature filled but template driven.
If you need to paint a bunch of arbitrary X and Y values on a graph from an input file or based on simple equations, gnuplot will happily oblige; it can handle chart types rrdtool cannot, and you have more direct control of output.
If you want some 3D / surface graphs, RRDTool won't do it, anyways. Gnuplot's less expensive than Matlab / Maple.
Try plotting a test dataset with 500 million datapoints. there should normally be some form of averaging / smoothing / "selection of
This is why rrdtool won't work for me. I'll be inputting data sets for charts that rrdtool doesn't understand and I would like to add a z-axis for some data that would be better understood on a 3D plot. The Graph.pm has hooks for xrt3d which looks pretty neat. I'm also looking at Grace as well using that same module (xmgrace), and it also has gnuplot I can work with. This way I can reuse code if I decide to use one over the other. The other issue is that all of this data is already stored (at least most of it) in mysql, and I don't need to waste cycles and chunks on riping data from one DB and have to create another just to display a graph. Plus some of the RRA's would be huge and therefore slow. points" contemplated, if the dataset is huge The few hundred million rows are spread out across different servers and are not all the same table types. I believe it's 280-300 million rows by first estimates, but they will be represented in different manors. I won't be injecting the volume of inserts aggregate onto one plot over time. I haven't gone that mad yet :) For measurements such as that, averaging will be used for such trends. Good talking to you again, Max On Sun, Feb 20, 2011 at 10:45 PM, Jimmy Hess <mysidia@gmail.com> wrote:
On Fri, Feb 18, 2011 at 1:13 PM, Max Pierson <nmaxpierson@gmail.com> wrote:
Anyone out there using something other than rrdtool for creating graphs?? I have a project that will need a trend taken, and unfortunately rrdtool doesn't fit the bill. All of the scripting, data collection, database archival, etc will be custom written or is already done (with some hacks of course :). So really what i'm looking for is something along the lines of GNUplot. Has anyone used it before and would like to share
I haven't heard of gnuplot used often with other software as a framework for graphing/visualizations. For simple visualizations, I think usually a 'native' framework/API is preferred, e.g. JGraph for java apps.
I suspect one reason gnuplot is not used as widely as it could be otherwise is, its licensing is not as "friendly" as other graphics frameworks. gnuplot license is GPL incompatible and does not seem to even fully meet the open source definition,
Because redistributing complete modified source code of gnuplot itself is not allowed by the license; a clear reading of gnuplot license suggests only patches, unmodified source code, can be freely redistributed, redistributed binaries based on modified source have special rules).
Aside from that caveat, which most likely does not normally impair private use by a network operator: gnuplot is a really good tool. If you need to paint a bunch of arbitrary X and Y values on a graph from an input file or based on simple equations, gnuplot will happily oblige; it can handle chart types rrdtool cannot, and you have more direct control of output.
If you want some 3D / surface graphs, RRDTool won't do it, anyways. Gnuplot's less expensive than Matlab / Maple.
You can even set terminal type to "dumb" in gnuplot, and generate some fancy ASCII art graphs on stdout.
In regards to scalability...
About the millions of rows... err.. Try plotting a test dataset with 500 million datapoints. Chances are gnuplot won't necessarily scale that well either, and you need some method to be selective of which rows are provided as input to the plotting framework, in that case.
If you have a million datapoints on your X axis, each X position is smaller than 1/1000 of a display pixel (on a graph that fits on a display at say 1920x1080); displaying such high resolution of all datapoints at once on the unzoomed graph is beyond the display hardware capabilitiy.
there should normally be some form of averaging / smoothing / "selection of points" contemplated, if the dataset is huge
experiences?? Seems like it will be able to my plot data accordingly, but wanted to see if there were any other popular tools I've yet to come across.
(Open-Source only please)
-- -JH
Max Pierson <nmaxpierson@gmail.com> wrote on 02/21/2011 04:15:46 PM:
Unfortunately, I'm not savvy with Java at all, so the really cool viz API's wont work for me (there's just something about Java ... I simply can't get into it and I see alot of Java based apps that are resource hogs). I was looking at mostly using some simple Perl + PHP (or even Python) for the graph generation. My own cacti if you will, just not as feature filled but template driven.
The GD 'C' package has great Perl interfaces called GD, and GD:Graph. Easy to work with ... GD: http://search.cpan.org/~lds/GD-2.30/GD.pm GD:Graph: http://search.cpan.org/~bwarfield/GDGraph-1.44/ Joe
The GD 'C' package has great Perl interfaces called GD, and GD:Graph
I'm test driving GD as well. Trying out a few other tools that were mentioned here also. Trying to get a feel for which I like best. Thanks for all of the replies! On Mon, Feb 21, 2011 at 3:24 PM, Joe Loiacono <jloiacon@csc.com> wrote:
Max Pierson <nmaxpierson@gmail.com> wrote on 02/21/2011 04:15:46 PM:
Unfortunately, I'm not savvy with Java at all, so the really cool viz API's wont work for me (there's just something about Java ... I simply can't get into it and I see alot of Java based apps that are resource hogs). I was looking at mostly using some simple Perl + PHP (or even Python) for the graph generation. My own cacti if you will, just not as feature filled but template driven.
The GD 'C' package has great Perl interfaces called GD, and GD:Graph. Easy to work with ...
GD: http://search.cpan.org/~lds/GD-2.30/GD.pm
GD:Graph: http://search.cpan.org/~bwarfield/GDGraph-1.44/
Joe
participants (12)
-
Christopher O'Brien -
david raistrick -
Jim Gettys -
Jimmy Hess -
Joe Loiacono -
Jon Lewis -
Marco F. Delaurenti -
Max -
Max Pierson -
Paul Graydon -
Peter A. Friend -
Rene Skjoldmose