On 7/29/2014 6:42 PM, Matt Palmer wrote:
Of course, getting anything back*out* of that again in any sort of reasonable timeframe would be... optimistic. I suppose if you're storing it all in hadoop you can map/reduce your way out of trouble, but that's going to mean a lot of equipment sitting around doing nothing for 99.99% of the time. Perhaps mine litecoin between searches? The timestamp is a natural index. You shouldn't need to run a distributed query for finding information about a specific incident. You would have to write your own custom tools to access and manage the db, so that's just impractical. The timestamp as well as most of the other fields should be fairly easily compressible since most of the bits are the same. You might as well use a regular plaintext logfile and gzip it.