Dear All, I would like to sincerely apologize to everyone everwhere who experienced problems yesterday due to the 7007 AS announcements. If anyone cares to know, here is what happened: At 11:30AM, EST, on 25 Apr 1997, our border router, stamped with AS 7007, recieved a full routing view from a downstream ISP (well, a view contacing 23,000 routes anyway). There was no distibute list imposed on the downstream since they also advertise their customer AS's to us (they were also experimenting with sending some routes out through us and some out through the MAE). We did filter out routes from them containing any of our AS numbers but since they got the view from someone at MAE-East none of our internal AS numbers showed up at all. Not having a filter imposed on the inbound side was our error. In an as yet unexplained twist of bits, the 7007 router then began to de-aggregate the 23K route view *and* strip the AS path out of it. I will emphasize that we were running no IGP at the time. Not one. Not OSPF, not RIP, nothing. Our MAE-East border router, AS 6082, then got a feed of these routes, at last count 73,000+, which set off our network monitor system which wacthes for, among other things, route views over 45k lines in size. At 11:45AM we disabled the BGP peering session with AS 1790 that was in place with the 7007 router and immediately contacted Sprint (contrary to popular belief that they called *us* first to let us know about the problem). As we were trying to determine what had happened, we began getting calls from other ISPs saying that we were announcing their routes with specificity as well as best AS path. That really alarmed us since we saw no announcements still going out. When these calls persisted, we rebooted our 7007 router (that was at 12:00PM). When the router came back up, it did begin to announce a full view to AS 1790 again, but this time as a normal BGP advertisement, i.e. with AS paths and aggregated addresses. We then imposed a distibute filter on our downstream and toward 1790, which stopped the announcement and, we thought, solved the problem. Well, the phone *kept* ringing and we then started to see the 7007 paths coming into our other routers over the MAE's. Okay, so panic ensued, and we unlugged *everything* at 12:15PM almost to the second. Then, at 12:25 the Sprint NOC called us to say the they were about to turn down the DS-3 connection to our 7007 router since they were *still* seeing the routes. We of course told them to go ahaead (since the router had no power to it at this point we were *very* confused). It seems that even after we stopped announcing the demon-view at 12:00, 1790 kept propagating the routes. We continued to field calls until about 4:45PM yesterday from ISP's all over the world. According to our conversations with the Sprint NOC at 2:14PM yesterday, they simply could not clear the 7007 routes from their BGP tables, they "just keep appearing again" as one of the techs told us. It also seems that some large, switched-based backbone provider, began distributing the routes to MSN one the west coast which lingered until about 7:00PM EST. We had engineers from the router manufacturer in until about 1:00AM this moring crawling all over the equipment making sure that we hadn't created an incorrect config set. We also now impose full distribute list filters on all peers. All I can say in our defense is that I believe we did debug the problem in the most expedient manner possible, and when it seemed that even after disabling the BGP session, we were still endagering other networks, we did completely disconnect ourselves from the Net. We did *not* perform any of this maliciously, I'm not sure that I could duplicate the event if I tried. Anyone who called and got a harsh voice on the phone, well, I sincerely apologize to them individually, but some in particular should not have tried to impersonate a company other than their own *and* should not have started cursing out the NOC tech who answered. I would also like to take this time to thank AT&T WorldNet, NASA Sciences Institute, and Net Access Corporation who called and did not just ask for an explanation, but offered asisstance. Sincerely, MANAGEMENT ANALYSIS, INCORPORATED Vincent J. Bono Director Network Services