RE: [cacti-announce] Cacti 0.8.6j Released (fwd)
how do you define your schema? how long does it take to insert/index/whatnot the data?
This is a much bigger deal than most people realize. Poor schema design will cause your system to choke bade when you try to scale it. In fact, relational databases are not the ideal way to store this kind of data so when you design your schema, you are really fighting against the database to wrestle it into something that will work.
this is a huge burden to figure it all out, implement and then monitor/operate 24x7. miss enough samples or data and you end up billing too little. this is why most folks have either cooked their own, or use some expensive suite of tools, leaving just a little bit of other stuff out there.
Personally, I doubt that it is possible to build a workable system, even with plugins, that will do the job for a significant percentage of service providers. Different companies have different needs, different hot button items, etc. This is an area where breaking the problem down into well-defined separate problems with a well-defined linkage, will go a long way. But to start with, just solving the data storage problem is a good place to start. If someone can create a specialized network monitoring database that scales, then the rest of the toolkit will be much easier to deal with. Note that people have done a lot of research on this sort of time-series database. People working in high-energy physics also have to deal with massive sets of time-series data. There is plenty of literature out there to help guide a design effort. But Open-Source developers don't usually do this kind of up-front research before starting coding. Money and manpower won't solve that kind of problem. --Michael Dillon
But to start with, just solving the data storage problem is a good place to start. If someone can create a specialized network monitoring database that scales, then the rest of the toolkit will be much easier to deal with. Note that people have done a lot of research on this sort of time-series database. People working in high-energy physics also have to deal with massive sets of time-series data. There is plenty of literature out there to help guide a design effort. But Open-Source developers don't usually do this kind of up-front research before starting coding. Money and manpower won't solve that kind of problem.
How about something like: http://www.hdfgroup.org/whatishdf5.html -- Scanned for viruses and dangerous content at http://www.oneunified.net and is believed to be clean.
* Ray Burkholder:
How about something like: http://www.hdfgroup.org/whatishdf5.html
I don't think they support transactional updates, which makes it hard to use for live data. (A simple crash, and you need to recover from backup.) -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
How about something like: http://www.hdfgroup.org/whatishdf5.html
I don't think they support transactional updates, which makes it hard to use for live data. (A simple crash, and you need to recover from backup.)
Going back to this thread, http://www.kx.com/ deals in financial transaction databases where they store millions of ticks. They appear to have a transactional based language with a solution that appears to be robust and fail resistant. I'm sure it has a price tag that goes along with the capabilities. Anyone encountered this before? -- Scanned for viruses and dangerous content at http://www.oneunified.net and is believed to be clean.
On 2/7/07, Ray Burkholder <ray@oneunified.net> wrote:
Going back to this thread, http://www.kx.com/ deals in financial transaction databases where they store millions of ticks. They appear to have a transactional based language with a solution that appears to be robust and fail resistant.
I'm sure it has a price tag that goes along with the capabilities.
Anyone encountered this before?
hmm, that is quite interesting. and apparently people out there _are_ using it for things like counter values and what not - based on their FAQ. I'd absolutely love to know more about the algorithms and math behind something like kdb+
Hey folks, I am following up to an ancient email because I'm curious if anyone has some SNMP-related resources. Basically, there's a lot of how-to or manpage sort of information, but I'm still unclear on what an MIB actually _is_, what problem ASN.1 actually solves, and more to the point how the whole shebang (I'm using net-snmpd) is typically used. I believe that what I need to do is get any/all MIBs for all "entities" (typically networking hardware devices) that I want to monitor, and import them into the net-snmp configuration somehow, and then software that calls on net-snmp can access the information from the devices. Is this accurate? Will I need to import MIBs to every net mgmt application? Should they be carefully accounted for and synchronized, or can I treat them like a typical configuration file, where it is obvious if I need it and I get them as needed? On Thu, Jan 25, 2007 at 02:43:40PM +0100, Florian Weimer wrote:
How about something like: http://www.hdfgroup.org/whatishdf5.html I don't think they support transactional updates, which makes it hard to use for live data. (A simple crash, and you need to recover from backup.)
Hmm, doesn't that depend on how they do their writes and the structure of the file/database? I can think of a number of things that write to disk which recover automagically if they crash which don't have transaction logs or rollback or anything like that... it's just inherent in how they write. -- Kill dash nine, and its no more CPU time, kill dash nine, and that process is mine. -><- <URL:http://www.subspacefield.org/~travis/> For a good time on my UBE blacklist, email john@subspacefield.org.
Oh yeah, I'm reading a book which mentions the following F/OSS config mgmt or net mgmt apps, was wondering if anyone has any opinions before I spend a lot of time forming my own: ISC router monitor (rtmon) (no known URL) router audit tool (RAT): http://www.sans.org/reading_room/whitepapers/networkdevs/238.php http://cisecurity.org/bench_cisco.html really awesome new cisco confIg differ (RANCID): http://shrubbery.net/rancid/ http://www.openxtra.co.uk/resource-center/open_source_network_management_sys... -- Kill dash nine, and its no more CPU time, kill dash nine, and that process is mine. -><- <URL:http://www.subspacefield.org/~travis/> For a good time on my UBE blacklist, email john@subspacefield.org.
[If people think this is off-topic, please let me know and I'll take it to private mail with Travis.] On Tue, May 08, 2007 at 07:32:18PM -0500, Travis H. wrote:
Hey folks, I am following up to an ancient email because I'm curious if anyone has some SNMP-related resources. Basically, there's a lot of how-to or manpage sort of information, but I'm still unclear on what an MIB actually _is_,
It's an overloaded term. Technically, I think it's the values which you can query by OID in an agent, but most people use the term to describe the textual description of the OIDs and what they mean, especially when they talk about "downloading a MIB".
what problem ASN.1 actually solves,
How to encode the queries and responses. Unless you're actually writing an agent or low-level manager library, ignore it. Seriously, you don't need the headache.
and more to the point how the whole shebang (I'm using net-snmpd) is typically used.
Agent on device provides values, management app(s) collect data by polling (and possibly via traps), sysadmin gets to go home on time for once.
I believe that what I need to do is get any/all MIBs for all "entities" (typically networking hardware devices) that I want to monitor, and import them into the net-snmp configuration somehow, and then software that calls on net-snmp can access the information from the devices.
Is this accurate?
Kinda-sorta. You don't actually need a MIB to be able to query a device -- you can, in theory, just walk it from the root and get all the OIDs (and their values) that the agent provides. However, since all you'll get are massive quantities of numbers, that'll be fairly useless, and the MIB file you refer to will help you (and your agent software) decode the OIDs into something more readable. That being said, if you only want to monitor a few OIDs, and you know the OIDs already, then the MIB is unnecessary. Where you put the MIBs to net-snmp can find them depends on where net-snmp has been told to look for them. /usr/share/snmp/mibs is where they go on my system, but $DEITY knows where they might end up on some Unices.
Will I need to import MIBs to every net mgmt application? Should they
If they use different OIDs, and you want to be able to use them easily, yes. This "using different OIDs" thing is depressingly common -- although there are RFC standards for a lot of the "common" types of networking data, a combination of "the RFCs don't define all our statistics" and NIH means that a lot of vendor equipment does it's own SNMP thing.
be carefully accounted for and synchronized, or can I treat them like a typical configuration file, where it is obvious if I need it and I get them as needed?
They're not critical to the operation of the whole thing, merely the comprehensibility, so don't get too obsessed over your MIBs. - Matt -- Just because we work at a University doesn't mean we're surrounded by smart people. -- Brian Kantor, in the monastery
and more to the point how the whole shebang (I'm using net-snmpd) is typically used.
Agent on device provides values, management app(s) collect data by polling (and possibly via traps), sysadmin gets to go home on time for once.
I have yet to see this work in practice however.
On Tue, May 08, 2007 at 08:10:56PM -0700, matthew zeier wrote:
and more to the point how the whole shebang (I'm using net-snmpd) is typically used.
Agent on device provides values, management app(s) collect data by polling (and possibly via traps), sysadmin gets to go home on time for once.
I have yet to see this work in practice however.
Yeah, I misread 'typically' as 'theoretically'. Practical experience is more like: Agent on device lies about it's values, management apps collect lies (and ignore/lose traps), and the sysadmin has yet more software to swear at. <grin> - Matt -- I'm seriously considering getting one of those bright-orange prison overalls and stencilling PASSENGER on the back. Along with the paper slippers, I ought to be able to walk right through security. Not. -- Brian Kantor, in the Monastery
Matthew Palmer wrote:
On Tue, May 08, 2007 at 08:10:56PM -0700, matthew zeier wrote:
and more to the point how the whole shebang (I'm using net-snmpd) is typically used.
Agent on device provides values, management app(s) collect data by polling (and possibly via traps), sysadmin gets to go home on time for once.
I have yet to see this work in practice however.
Yeah, I misread 'typically' as 'theoretically'. Practical experience is more like:
Agent on device lies about it's values, management apps collect lies (and ignore/lose traps), and the sysadmin has yet more software to swear at. <grin>
- Matt
Just for curiousities sake IASON is reading logs most of the time. proc2pl is reading the /proc filesystem. I did not find the time and equipment for testing so I used snmpwalk to write a file and read it just like any normal file or /proc. Processing the output of snmpwalk just got me the "normal" log file I was interested in. I tried writing back into snmp variables but I never got a HP Procurve switch to do what I wanted. When they used different MIBs for different families of their switches, I gave up. Now I see linux boxes most of the time. They all use different MIBs for different things. Reading /proc is much easier and there a fewer differences between the machines. The soho stuff I find mostly uses web interfaces sometimes a real linux with a real log but almost never snmp. Looks sad, but I am still interested as it could make things a lot easier. Cheers Peter and Karin -- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Rimbacher Strasse 16 D-69509 Moerlenbach-Bonsweiher +49(6209)795-816 (Telekom) +49(6252)750-308 (VoIP: sipgate.de) mail: peter@peter-dambier.de mail: peter@echnaton.arl.pirates http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/ http://www.cesidianroot.com/
but I'm still unclear on what an MIB actually _is_,
A MIB is the database schema for an object-oriented hierarchical database. The key words there are schema and hierarchical. Schema means that it describes how the data is organized and hierarchical means that it is *NOT* organized in tables like a relational database, but in a tree structure, like DNS gone mad.
what problem ASN.1 actually solves,
When you transmit data across a wire, you need to be able to parse the individual data elements out of the bitstream at the receiving end. ASN.1 is merely one way of encoding data so that it can be reliably parsed at the other end. If you have ever had problems loading a CSV file into a database, you will realize the importance of having a defined standard for encoding data so that it can be reliably decoded by an arbitrary recipient. Theoretically, SNMP is better than parsing IOS command output, but in practice, it may be hard or impossible to get the information that you want via SNMP.
and more to the point how the whole shebang (I'm using net-snmpd) is typically used.
Typically? I think that typically people use the names from the schemas (MIBs) to identify the key to the data that they want, then put the numeric keys directly into their application. That way you can run it on any server regardless of whether the MIB(Schema) is available or not. But if you do stick keys like .1.3.6.1.4.1.2021.4.11.0 in your code it is a good idea to include the named version of the key in the comments.
I believe that what I need to do is get any/all MIBs for all "entities" (typically networking hardware devices) that I want to monitor, and import them into the net-snmp configuration somehow, and then software that calls on net-snmp can access the information from the devices.
Is this accurate?
That would work but it can be tricky to get the RIGHT MIBs that match the data actually available in your device. Also, reading MIBs can be misleading because you will see things that look great, but don't work because they are deprecated or because the router vendor didn't implement it the way that you think.
Will I need to import MIBs to every net mgmt application? Should they be carefully accounted for and synchronized, or can I treat them like a typical configuration file, where it is obvious if I need it and I get them as needed?
Now you see where the SNMP alligator swamp lies. If you are building your own network management applications, you may be happier only putting the MIBs on the development machines, and putting the numeric keys into your application code, or better yet, into your application's config file. MIBs have lots of stuff that you probably don't need unless you are allowing users to browse through and query arbitrary data. --Michael Dillon
On 9-May-2007, at 05:25, <michael.dillon@bt.com> <michael.dillon@bt.com> wrote:
but I'm still unclear on what an MIB actually _is_,
A MIB is the database schema for an object-oriented hierarchical database.
I believe that (some?) purists would assert that there is but one MIB, and that all other MIB-like entities shipped by vendors and others are properly called MIB Extensions. This is pedantry, however. Once upon a time when I was called about to care about this stuff I digested the book "Understanding SNMP MIBs" by David Perkins and Evan McGinnis, Prentice Hall, ISBN 0-13-437708-7. Anybody looking for a dead-tree treatment of SNMP from the design perspective could do a lot worse. Joe
On Wed, 9 May 2007 michael.dillon@bt.com wrote:
keys into your application code, or better yet, into your application's config file. MIBs have lots of stuff that you probably don't need unless you are allowing users to browse through and query arbitrary data.
...for example, if you're running a tool like Cacti. (which we do at $DAYJOB, and fortunately, I've never had to screw around with MIBs or OIDs) -- Steve Sobol, Professional Geek ** Java/VB/VC/PHP/Perl ** Linux/*BSD/Windows Victorville, California PGP:0xE3AE35ED It's all fun and games until someone starts a bonfire in the living room.
On Wed, May 09, 2007 at 10:25:14AM +0100, michael.dillon@bt.com wrote:
A MIB is the database schema for an object-oriented hierarchical database. The key words there are schema and hierarchical.
A-ha! So when they say "object" as in "OID", they are referring to stuff in the MIB database? Okay, now many things are beginning to make more sense. By itself, that word gives no clue as to what it refers to. For that matter, it'd be nice if someone defined LDAP's use of the word "attribute", too. Drift: LDAP too uses ASN.1, in fact the same OIDs used by SNMP, and in the O'Reilly book it mentions that it is possible to define different matching rules for each class. Now, do they mean that somehow, this MIB syntax can actually encode an algorithm in some kind of hideous turing-machine-gone-mad, and that I've got to worry about malicious MIBs, or does it just refer to a routine implemented elsewhere?
Schema means that it describes how the data is organized
Should read: ``Schemata describe how the data are organized'' Stigma, stigmata; schema, schemata :-) Forgive me if I digress into ASN.1 very briefly; it apparently rears its ugly head in numerous places in cryptography as well as networking, and I have struggled with it a bit. Based on what I have read, this syntax is "abstract" in the sense that it says something like "class C is composed of a DATE object, TIME object, and BLARG object", without specifying how to encode or decode any of those objects into some concrete form either for the user or to put in a packet to send to another system. The encoding and decoding is done with a "transfer syntax", and interpreting it for a human (that is, figuring out a way to represent it) is yet another unsolved problem. Sounds a lot like stone soup (or XML) to me.
That would work but it can be tricky to get the RIGHT MIBs that match the data actually available in your device. Also, reading MIBs can be misleading because you will see things that look great, but don't work because they are deprecated
Those of you who use this word frequently may be amused at its definition: To pray against, as an evil; to seek to avert by prayer; to seek deliverance from; to express deep regret for; to desire the removal of. [archaic]
Now you see where the SNMP alligator swamp lies. If you are building your own network management applications, you may be happier only putting the MIBs on the development machines, and putting the numeric keys into your application code, or better yet, into your application's config file. MIBs have lots of stuff that you probably don't need unless you are allowing users to browse through and query arbitrary data.
Yeah, at this point I'm just playing around and exploring, and so want the MIBs to make sense of the numbers. -- Kill dash nine, and its no more CPU time, kill dash nine, and that process is mine. -><- <URL:http://www.subspacefield.org/~travis/> For a good time on my UBE blacklist, email john@subspacefield.org.
ASN.1 is quite concrete, and specifys several encoding methods (I prefer BER myself) :) I'm not saying everyone would consider it pretty, but it's quite concrete ... Check out http://lionet.info/asn1c/ On 5/17/07, Travis H. <travis+ml-nanog@subspacefield.org> wrote:
On Wed, May 09, 2007 at 10:25:14AM +0100, michael.dillon@bt.com wrote:
A MIB is the database schema for an object-oriented hierarchical database. The key words there are schema and hierarchical.
A-ha!
So when they say "object" as in "OID", they are referring to stuff in the MIB database? Okay, now many things are beginning to make more sense. By itself, that word gives no clue as to what it refers to. For that matter, it'd be nice if someone defined LDAP's use of the word "attribute", too.
Drift:
LDAP too uses ASN.1, in fact the same OIDs used by SNMP, and in the O'Reilly book it mentions that it is possible to define different matching rules for each class. Now, do they mean that somehow, this MIB syntax can actually encode an algorithm in some kind of hideous turing-machine-gone-mad, and that I've got to worry about malicious MIBs, or does it just refer to a routine implemented elsewhere?
Schema means that it describes how the data is organized
Should read: ``Schemata describe how the data are organized''
Stigma, stigmata; schema, schemata
:-)
Forgive me if I digress into ASN.1 very briefly; it apparently rears its ugly head in numerous places in cryptography as well as networking, and I have struggled with it a bit.
Based on what I have read, this syntax is "abstract" in the sense that it says something like "class C is composed of a DATE object, TIME object, and BLARG object", without specifying how to encode or decode any of those objects into some concrete form either for the user or to put in a packet to send to another system. The encoding and decoding is done with a "transfer syntax", and interpreting it for a human (that is, figuring out a way to represent it) is yet another unsolved problem. Sounds a lot like stone soup (or XML) to me.
That would work but it can be tricky to get the RIGHT MIBs that match the data actually available in your device. Also, reading MIBs can be misleading because you will see things that look great, but don't work because they are deprecated
Those of you who use this word frequently may be amused at its definition:
To pray against, as an evil; to seek to avert by prayer; to seek deliverance from; to express deep regret for; to desire the removal of. [archaic]
Now you see where the SNMP alligator swamp lies. If you are building your own network management applications, you may be happier only putting the MIBs on the development machines, and putting the numeric keys into your application code, or better yet, into your application's config file. MIBs have lots of stuff that you probably don't need unless you are allowing users to browse through and query arbitrary data.
Yeah, at this point I'm just playing around and exploring, and so want the MIBs to make sense of the numbers. -- Kill dash nine, and its no more CPU time, kill dash nine, and that process is mine. -><- <URL:http://www.subspacefield.org/~travis/> For a good time on my UBE blacklist, email john@subspacefield.org.
This is where dbms' designed for data warehouses might come into play, something like SybaseIQ. It is adapted for long term storage and retrieval. michael.dillon@bt.com wrote:
how do you define your schema? how long does it take to insert/index/whatnot the data?
This is a much bigger deal than most people realize. Poor schema design will cause your system to choke bade when you try to scale it. In fact, relational databases are not the ideal way to store this kind of data so when you design your schema, you are really fighting against the database to wrestle it into something that will work.
this is a huge burden to figure it all out, implement and then monitor/operate 24x7. miss enough samples or data and you end up billing too little. this is why most folks have either cooked their own, or use some expensive suite of tools, leaving just a little bit of other stuff out there.
Personally, I doubt that it is possible to build a workable system, even with plugins, that will do the job for a significant percentage of service providers. Different companies have different needs, different hot button items, etc. This is an area where breaking the problem down into well-defined separate problems with a well-defined linkage, will go a long way.
But to start with, just solving the data storage problem is a good place to start. If someone can create a specialized network monitoring database that scales, then the rest of the toolkit will be much easier to deal with. Note that people have done a lot of research on this sort of time-series database. People working in high-energy physics also have to deal with massive sets of time-series data. There is plenty of literature out there to help guide a design effort. But Open-Source developers don't usually do this kind of up-front research before starting coding. Money and manpower won't solve that kind of problem.
--Michael Dillon
participants (13)
-
Aaron Glenn -
Dorn Hetzel -
Florian Weimer -
Jason LeBlanc -
Joe Abley -
Matt Palmer -
Matthew Palmer
-
matthew zeier -
michael.dillon＠bt.com -
Peter Dambier -
Ray Burkholder -
Steve Sobol -
Travis H.