At 10:21 PM 3/15/96 -0500, Joel Katz wrote:
IGC (Internet Gateway Connections, an ISP in Florida that was multihomed between NET99 and MCI) just had an unexpected outage lasting in
*The lie:*
excess of an hour. This outage was deliberately engineered by AGIS/NET99.
*The fact:* IGC's routes were pulled from the Net99 net within 90 minutes of their being turned off *at IGC's request*.
First, a little history:
IGC's first 'Net T1 was to NET99. Instead of connecting us directly to their backbone (as we had expected) they connected us to Netrunner (one of our competitors). Netrunner/NET99 only has a single T1 out of Miami (to Atlanta) and as a result, our throughput was awful and we obtained an additional T1 to MCI.
This was common in the Net99 network. I guess it worked OK at first, but it didn't scale as customers grew. There was a tremendous hoo-ha back when AGIS bought Net99 because some people who were in mid-install were redirected to AGIS. That hurt the pending customers, but it helped the existing customers like IGC as much as it could. *The lie:*
A few days ago, someone at Netrunner got upset with someone at NET99/AGIS and shut off _our_ service. For some reason (we strongly suspect a configuration error at NET99) our traffic failed to reroute over our MCI T1 for several hours. (They have a habit of broadcasting static routes to route around outages, thus completely screwing their multi-homed customers.)
*The truth:* The referenced disconnection did occur, and was restored. This kind of problem is a very real danger of the way Net99 colocated in certain cities. That's why AGIS only colocates in POP's and private spaces. And in the event that a customer is dually homed and not running BGP, it's incumbent upon the customer to make sure that *both providers* are aware of the fact and properly setting local prefs. IGC did not do that and bears full responsibility for any failure to route properly during an outage. Aha: a true statement:
We asked AGIS for a one month refund to compensate us for our losses due to the sabotage. Instead, they issued us a _one_day_ credit.
Ah, another true statement:
We got upset, both with the ability of our T to pass traffic, with NET99's continual routing problems, with NET99's backbone being disassembled from under us, and with NET99/AGIS's continual delays in migrating us to the AGIS network. We asked them to discontinue our service.
True, but we certainly wouldn't have left them without a circuit. We've tried mightily to install a new circuit to their premises, and ran into a b8zs facilities problem. We could have installed it AMI as a temporary measure with no problem. *Another LIE*
They did. They also deliberately and maliciously began broadcasting more specific routes for the individual class C networks in our CIDR block. (We confirmed this with MCI.)
*The truth:* Over the months various static routes have been installed by various people on Net99 to route around some problems (a practice which predates our purchase). You run into some curious situations like a static on a Chicago router pointing to Phoenix. In this case, there were statics of differing prefix lengths on different routers. The aggregate address statement got pulled, and we didn't catch the more specifics on another router. As soon as we found out what happened, we pulled them. In fact, we pulled them while we were on the phone with MCI so that they could confirm that they were gone. The statement of malicious intent is untrue and libellous and must be withdrawn immediately by Mr. Katz. [...] *Another LIE*
I contacted MCI again and they contacted AGIS/NET99 with the ultimate threat being for MCI to stop carrying AGIS' traffic if they would not cease broadcasting the bogus routes. Shortly after being contacted by MCI, AGIS/NET99 stopped announcing the bogus routes to MCI, but continued advertising these routes to their other peers at the NAPs.
Nope. Like I said, MCI didn't threaten us at all, and worked with us to see that the routing was correct. Net99 isn't announcing those routes anywhere. Both AGIS and Net99 announce all routes at all exchange points to all providers in compliance with our peering contracts. If MCI had called back and said, "we still have a problem," we'd work with them until it was taken care of.
Fortunately, MCI was able to reconfigure our access lists at their router to allow us to broadcast competing more specific routes for each individual class C in the CIDR block. Thus (more or less) restoring IGC's net connectivity.
More specifics aren't competing. They take precedence. Sometimes it's better to be silent and thought dumb... Ah, more libel:
NET99's deliberate, malicious actions are contrary to the very spirit of the Internet. They resulted in DALnet (the third largest IRC network) being split to ribbons, depriving 2,500 people of the ability to converse with each other and cutting off DALnet's security and access control system which is hosted here.
Another retraction required ASAP, Joel. AND NOW, here's what Joel left out: IGC called our NOC up, screaming, the night this occurred, and "put us on notice" that "if we didn't disconnect our circuit to them immediately [we all know that phone companies do nothing immediately] they were going to flood ping [another Miami customer] until we did." We had no reason to disbelieve them so we did turn down the port, and had the routing unwound within a couple of hours, which isn't bad given the nonexistent notice and other things going on. In other words, Joel comes to the court of public opinion with unclean hands. Joel threatened an illegal act against another customer. Joel's righteousness rings hollow. Joel must recant immediately.
Please excuse me if this is not the appropriate forum for discussing this event.
And this is weird. The sig and the FROM don't match. Is this all a joke?
David Schwartz Director of Network Services Internet Gateway Connections (954)-430-3030
_____________________________________________________________________ Peter Kline Senior Network Engineer| NOC 313-730-1130 AGIS - Internet Backbone Services | ///////////////////////////////////////////////////////////////////// "I knew exactly how he felt. Not so much banging the head against a brick wall as actively being attacked by a cliff." - Dick Francis in _Enquiry_