Bogon stupidity... warning... operational post.
For those of you who make use of the completewhois bogon lists (apparently there are some)... It appears that as of this morning they've decided that 204.57.128.0 - 204.255.255.255 belongs in the bogon lists. Grrrr..... Pardon the operational interruption.
I'm aware of that and rerunning the generation program right now. There is a new server being installed from yesterday to today as replacement for old and after everything is done things will work much smoother as the old server ran out of space and is causing problems. On Thu, 22 Dec 2005, Rodney Joffe wrote:
For those of you who make use of the completewhois bogon lists (apparently there are some)...
It appears that as of this morning they've decided that 204.57.128.0 - 204.255.255.255 belongs in the bogon lists.
Grrrr.....
Pardon the operational interruption.
On Dec 22, 2005, at 10:03 AM, william(at)elan.net wrote:
I'm aware of that and rerunning the generation program right now. There is a new server being installed from yesterday to today as replacement for old and after everything is done things will work much smoother as the old server ran out of space and is causing problems.
Wouldn't it then have been appropriate then to remove the service/ data until it is fixed? Or am I missing something? ---------------------------------------------- Rodney Joffe CenterGate Research Group, LLC. http://www.centergate.com "Technology so advanced, even we don't understand it!"(R)
On Thu, 22 Dec 2005, Rodney Joffe wrote:
I'm aware of that and rerunning the generation program right now. There is a new server being installed from yesterday to today as replacement for old and after everything is done things will work much smoother as the old server ran out of space and is causing problems.
Wouldn't it then have been appropriate then to remove the service/data until it is fixed? Or am I missing something?
It was fixed - some of the data was being served was correct, some was not depending on which ip address you had in dns cache. I've reloaded it all manually right now, so should be correct and in sync everywhere available to the public. Basicly because of changes in servers, entire system of how data is distributed from generation server to distribution servers and confirmation of data from another server is screwed up and I'm working on all that today. This is the first time I had to change master server ... P.S. 204/8 was not the only problem, there were problems with 128/8 and 133/8 as well so my apologies to people who may have noticed problems overnight. -- William Leibzon Elan Networks william@elan.net
At 12:56 PM 12/22/2005, you wrote: P.S. 204/8 was not the only problem, there were problems with 128/8 and
133/8 as well so my apologies to people who may have noticed problems overnight.
199.128.0.0/9 too. -Robert Tellurian Networks - The Ultimate Internet Connection http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin
On Thu, 22 Dec 2005, Robert Boyle wrote:
At 12:56 PM 12/22/2005, you wrote: P.S. 204/8 was not the only problem, there were problems with 128/8 and
133/8 as well so my apologies to people who may have noticed problems overnight.
199.128.0.0/9 too.
Yes, legacy blocks (with large number of smaller allocations) whenever datasize during processing exceeded certain amount. The bad data was present at 2 of 4 servers for duration of the night but dns was being changes same time as well, so I don't know how much affect there was but apparently considerable; this is the most serious problem in months. -- William Leibzon Elan Networks william@elan.net
On Thu, 22 Dec 2005, william(at)elan.net wrote:
On Thu, 22 Dec 2005, Robert Boyle wrote:
At 12:56 PM 12/22/2005, you wrote: P.S. 204/8 was not the only problem, there were problems with 128/8 and
133/8 as well so my apologies to people who may have noticed problems overnight.
199.128.0.0/9 too.
Yes, legacy blocks (with large number of smaller allocations) whenever datasize during processing exceeded certain amount. The bad data was present at 2 of 4 servers for duration of the night but dns was being
so 50+% of your system was hozed for some long period of time :( bad.
changes same time as well, so I don't know how much affect there was but apparently considerable; this is the most serious problem in months.
'most serious problem in months' ... this has happened in smaller chunks during the past 'months' ? yikes... is that noted on your site so users of the 'service' will know what sorts of 'problems' they might be encountering due to their reliance on this 'service'?
On 12/22/05 1:35 PM, "Christopher L. Morrow" <christopher.morrow@mci.com> wrote:
On Thu, 22 Dec 2005, william(at)elan.net wrote:
On Thu, 22 Dec 2005, Robert Boyle wrote:
At 12:56 PM 12/22/2005, you wrote: P.S. 204/8 was not the only problem, there were problems with 128/8 and
133/8 as well so my apologies to people who may have noticed problems overnight.
199.128.0.0/9 too.
Yes, legacy blocks (with large number of smaller allocations) whenever datasize during processing exceeded certain amount. The bad data was present at 2 of 4 servers for duration of the night but dns was being
so 50+% of your system was hozed for some long period of time :( bad.
changes same time as well, so I don't know how much affect there was but apparently considerable; this is the most serious problem in months.
'most serious problem in months' ... this has happened in smaller chunks during the past 'months' ? yikes... is that noted on your site so users of the 'service' will know what sorts of 'problems' they might be encountering due to their reliance on this 'service'?
I wonder how many problems cymru has had in that period? I'm guess not so many... -- Daniel Golding
On Thu, 22 Dec 2005, Daniel Golding wrote:
On 12/22/05 1:35 PM, "Christopher L. Morrow" <christopher.morrow@mci.com> wrote:
'most serious problem in months' ... this has happened in smaller chunks during the past 'months' ? yikes... is that noted on your site so users of the 'service' will know what sorts of 'problems' they might be encountering due to their reliance on this 'service'?
I wonder how many problems cymru has had in that period? I'm guess not so many...
not sure, they did announce a planned outage a bit ago, and they had their bgp meltdown about 2 months ago...
On Thu, 22 Dec 2005, Christopher L. Morrow wrote:
Yes, legacy blocks (with large number of smaller allocations) whenever datasize during processing exceeded certain amount. The bad data was present at 2 of 4 servers for duration of the night but dns was being
so 50+% of your system was hozed for some long period of time :( bad.
Yes, bad reaction time - when problems were reported last year, it was dealt with in < 2 hours. It is always bad when things happen during the night, but as engineers we typically plan updates for that time thinking it would have less impact on everyone - sometimes it backfires.
changes same time as well, so I don't know how much affect there was but apparently considerable; this is the most serious problem in months.
'most serious problem in months' ... this has happened in smaller chunks during the past 'months' ? yikes... is that noted on your site so users of the 'service' will know what sorts of 'problems' they might be encountering due to their reliance on this 'service'?
I knew the system was out of space and that is why the change to new system, although in last couple days I had to rush a bit to finish it. Previously the errors were confined to comparisons to what is in bgp, so mostly not operationally important except for some research (also in couple cases last months impacted ip->country data files causing some country data to not list any ip blocks). This is not typical of the service and as I already noted. New server has 8% of space used right now (enough for 5+ years if nothing seriously changes; plus access to storage server with 1.5TB space) which means the same should not repeat. However I still have days of work to finish this upgrade in other ways. -- William Leibzon Elan Networks william@elan.net
participants (5)
-
Christopher L. Morrow -
Daniel Golding -
Robert Boyle -
Rodney Joffe -
william(at)elan.net