On Thu, 22 Dec 2005, Christopher L. Morrow wrote:
Yes, legacy blocks (with large number of smaller allocations) whenever datasize during processing exceeded certain amount. The bad data was present at 2 of 4 servers for duration of the night but dns was being
so 50+% of your system was hozed for some long period of time :( bad.
Yes, bad reaction time - when problems were reported last year, it was dealt with in < 2 hours. It is always bad when things happen during the night, but as engineers we typically plan updates for that time thinking it would have less impact on everyone - sometimes it backfires.
changes same time as well, so I don't know how much affect there was but apparently considerable; this is the most serious problem in months.
'most serious problem in months' ... this has happened in smaller chunks during the past 'months' ? yikes... is that noted on your site so users of the 'service' will know what sorts of 'problems' they might be encountering due to their reliance on this 'service'?
I knew the system was out of space and that is why the change to new system, although in last couple days I had to rush a bit to finish it. Previously the errors were confined to comparisons to what is in bgp, so mostly not operationally important except for some research (also in couple cases last months impacted ip->country data files causing some country data to not list any ip blocks). This is not typical of the service and as I already noted. New server has 8% of space used right now (enough for 5+ years if nothing seriously changes; plus access to storage server with 1.5TB space) which means the same should not repeat. However I still have days of work to finish this upgrade in other ways. -- William Leibzon Elan Networks william@elan.net