At 10:21 PM 3/15/96 -0500, Joel Katz wrote:
IGC (Internet Gateway Connections, an ISP in Florida that was multihomed between NET99 and MCI) just had an unexpected outage lasting in
*The lie:*
excess of an hour. This outage was deliberately engineered by AGIS/NET99.
*The fact:* IGC's routes were pulled from the Net99 net within 90 minutes of their being turned off *at IGC's request*.
First, a little history:
IGC's first 'Net T1 was to NET99. Instead of connecting us directly to their backbone (as we had expected) they connected us to Netrunner (one of our competitors). Netrunner/NET99 only has a single T1 out of Miami (to Atlanta) and as a result, our throughput was awful and we obtained an additional T1 to MCI.
This was common in the Net99 network. I guess it worked OK at first, but it didn't scale as customers grew. There was a tremendous hoo-ha back when AGIS bought Net99 because some people who were in mid-install were redirected to AGIS. That hurt the pending customers, but it helped the existing customers like IGC as much as it could. *The lie:*
A few days ago, someone at Netrunner got upset with someone at NET99/AGIS and shut off _our_ service. For some reason (we strongly suspect a configuration error at NET99) our traffic failed to reroute over our MCI T1 for several hours. (They have a habit of broadcasting static routes to route around outages, thus completely screwing their multi-homed customers.)
*The truth:* The referenced disconnection did occur, and was restored. This kind of problem is a very real danger of the way Net99 colocated in certain cities. That's why AGIS only colocates in POP's and private spaces. And in the event that a customer is dually homed and not running BGP, it's incumbent upon the customer to make sure that *both providers* are aware of the fact and properly setting local prefs. IGC did not do that and bears full responsibility for any failure to route properly during an outage. Aha: a true statement:
We asked AGIS for a one month refund to compensate us for our losses due to the sabotage. Instead, they issued us a _one_day_ credit.
Ah, another true statement:
We got upset, both with the ability of our T to pass traffic, with NET99's continual routing problems, with NET99's backbone being disassembled from under us, and with NET99/AGIS's continual delays in migrating us to the AGIS network. We asked them to discontinue our service.
True, but we certainly wouldn't have left them without a circuit. We've tried mightily to install a new circuit to their premises, and ran into a b8zs facilities problem. We could have installed it AMI as a temporary measure with no problem. *Another LIE*
They did. They also deliberately and maliciously began broadcasting more specific routes for the individual class C networks in our CIDR block. (We confirmed this with MCI.)
*The truth:* Over the months various static routes have been installed by various people on Net99 to route around some problems (a practice which predates our purchase). You run into some curious situations like a static on a Chicago router pointing to Phoenix. In this case, there were statics of differing prefix lengths on different routers. The aggregate address statement got pulled, and we didn't catch the more specifics on another router. As soon as we found out what happened, we pulled them. In fact, we pulled them while we were on the phone with MCI so that they could confirm that they were gone. The statement of malicious intent is untrue and libellous and must be withdrawn immediately by Mr. Katz. [...] *Another LIE*
I contacted MCI again and they contacted AGIS/NET99 with the ultimate threat being for MCI to stop carrying AGIS' traffic if they would not cease broadcasting the bogus routes. Shortly after being contacted by MCI, AGIS/NET99 stopped announcing the bogus routes to MCI, but continued advertising these routes to their other peers at the NAPs.
Nope. Like I said, MCI didn't threaten us at all, and worked with us to see that the routing was correct. Net99 isn't announcing those routes anywhere. Both AGIS and Net99 announce all routes at all exchange points to all providers in compliance with our peering contracts. If MCI had called back and said, "we still have a problem," we'd work with them until it was taken care of.
Fortunately, MCI was able to reconfigure our access lists at their router to allow us to broadcast competing more specific routes for each individual class C in the CIDR block. Thus (more or less) restoring IGC's net connectivity.
More specifics aren't competing. They take precedence. Sometimes it's better to be silent and thought dumb... Ah, more libel:
NET99's deliberate, malicious actions are contrary to the very spirit of the Internet. They resulted in DALnet (the third largest IRC network) being split to ribbons, depriving 2,500 people of the ability to converse with each other and cutting off DALnet's security and access control system which is hosted here.
Another retraction required ASAP, Joel. AND NOW, here's what Joel left out: IGC called our NOC up, screaming, the night this occurred, and "put us on notice" that "if we didn't disconnect our circuit to them immediately [we all know that phone companies do nothing immediately] they were going to flood ping [another Miami customer] until we did." We had no reason to disbelieve them so we did turn down the port, and had the routing unwound within a couple of hours, which isn't bad given the nonexistent notice and other things going on. In other words, Joel comes to the court of public opinion with unclean hands. Joel threatened an illegal act against another customer. Joel's righteousness rings hollow. Joel must recant immediately.
Please excuse me if this is not the appropriate forum for discussing this event.
And this is weird. The sig and the FROM don't match. Is this all a joke?
David Schwartz Director of Network Services Internet Gateway Connections (954)-430-3030
_____________________________________________________________________ Peter Kline Senior Network Engineer| NOC 313-730-1130 AGIS - Internet Backbone Services | ///////////////////////////////////////////////////////////////////// "I knew exactly how he felt. Not so much banging the head against a brick wall as actively being attacked by a cliff." - Dick Francis in _Enquiry_
On Mon, 18 Mar 1996, Peter Kline, Sr. Network Engineer wrote:
At 10:21 PM 3/15/96 -0500, Joel Katz wrote:
IGC (Internet Gateway Connections, an ISP in Florida that was multihomed between NET99 and MCI) just had an unexpected outage lasting in
*The lie:*
excess of an hour. This outage was deliberately engineered by AGIS/NET99.
*The fact:*
IGC's routes were pulled from the Net99 net within 90 minutes of their being turned off *at IGC's request*.
The outage had nothing to do with Net99 removing any routes, as I think I clearly stated.
First, a little history:
*The lie:*
A few days ago, someone at Netrunner got upset with someone at NET99/AGIS and shut off _our_ service. For some reason (we strongly suspect a configuration error at NET99) our traffic failed to reroute over our MCI T1 for several hours. (They have a habit of broadcasting static routes to route around outages, thus completely screwing their multi-homed customers.)
*The truth:*
The referenced disconnection did occur, and was restored. This kind of problem is a very real danger of the way Net99 colocated in certain cities. That's why AGIS only colocates in POP's and private spaces. And in the event that a customer is dually homed and not running BGP, it's incumbent upon the customer to make sure that *both providers* are aware of the fact and properly setting local prefs. IGC did not do that and bears full responsibility for any failure to route properly during an outage.
We did not do *WHAT*? How were we BGP peering with NET99 in the first place if we didn't notify them that we were multi-homed? What are you claiming we didn't do? And what in the excerpt are you claiming is a lie?
Aha: a true statement:
We asked AGIS for a one month refund to compensate us for our losses due to the sabotage. Instead, they issued us a _one_day_ credit.
Ah, another true statement:
We got upset, both with the ability of our T to pass traffic, with NET99's continual routing problems, with NET99's backbone being disassembled from under us, and with NET99/AGIS's continual delays in migrating us to the AGIS network. We asked them to discontinue our service.
True, but we certainly wouldn't have left them without a circuit. We've tried mightily to install a new circuit to their premises, and ran into a b8zs facilities problem. We could have installed it AMI as a temporary measure with no problem.
If this is true, it was never communicated to me.
*Another LIE*
They did. They also deliberately and maliciously began broadcasting more specific routes for the individual class C networks in our CIDR block. (We confirmed this with MCI.)
*The truth:*
Over the months various static routes have been installed by various people on Net99 to route around some problems (a practice which predates our purchase). You run into some curious situations like a static on a Chicago router pointing to Phoenix. In this case, there were statics of differing prefix lengths on different routers. The aggregate address statement got pulled, and we didn't catch the more specifics on another router. As soon as we found out what happened, we pulled them. In fact, we pulled them while we were on the phone with MCI so that they could confirm that they were gone.
The only problem with this is that it is inconsistent with recorded statements made to me be people at AGIS when I called to discuss this issue with them. When I _specifically_ asked why AGIS was broadcasting routes for individual class C's in our CIDR block, I was told to ask Randy Epstein, the IGC employee who made the decision to terminate our service with NET99/AGIS.
The statement of malicious intent is untrue and libellous and must be withdrawn immediately by Mr. Katz.
I will clarify by stating that my judgment of malicious intent was based upon two facts: 1) As I said above, AGIS specifically referred my to our own employees as the reason for the more specific routes orginating at AGIS, and 2) Shortly after we were able to restore routing, AGIS pulled the reverse DNS from all of the class C's in our CIDR block, without warning. Was this another 'mistake' or 'oversight'?
*Another LIE*
I contacted MCI again and they contacted AGIS/NET99 with the ultimate threat being for MCI to stop carrying AGIS' traffic if they would not cease broadcasting the bogus routes. Shortly after being contacted by MCI, AGIS/NET99 stopped announcing the bogus routes to MCI, but continued advertising these routes to their other peers at the NAPs.
Nope. Like I said, MCI didn't threaten us at all, and worked with us to see that the routing was correct. Net99 isn't announcing those routes anywhere. Both AGIS and Net99 announce all routes at all exchange points to all providers in compliance with our peering contracts. If MCI had called back and said, "we still have a problem," we'd work with them until it was taken care of.
Are you disputing that you continued to advertise routes for the Class C's in our CIDR block to your other peers at the NAPs? What in what I said are you claiming is a lie? I don't get it. Your statements seem entirely consistent with mine.
Fortunately, MCI was able to reconfigure our access lists at their router to allow us to broadcast competing more specific routes for each individual class C in the CIDR block. Thus (more or less) restoring IGC's net connectivity.
More specifics aren't competing. They take precedence. Sometimes it's better to be silent and thought dumb...
More specific than our normal aggregate routes, competing with your new routes.
Ah, more libel:
NET99's deliberate, malicious actions are contrary to the very spirit of the Internet. They resulted in DALnet (the third largest IRC network) being split to ribbons, depriving 2,500 people of the ability to converse with each other and cutting off DALnet's security and access control system which is hosted here.
Another retraction required ASAP, Joel.
Restore our reverse DNS for those class C's and give us a reasonable six months to renumber and I'll retract my statements.
AND NOW, here's what Joel left out:
IGC called our NOC up, screaming, the night this occurred, and "put us on notice" that "if we didn't disconnect our circuit to them immediately [we all know that phone companies do nothing immediately] they were going to flood ping [another Miami customer] until we did." We had no reason to disbelieve them so we did turn down the port, and had the routing unwound within a couple of hours, which isn't bad given the nonexistent notice and other things going on.
In other words, Joel comes to the court of public opinion with unclean hands. Joel threatened an illegal act against another customer. Joel's righteousness rings hollow. Joel must recant immediately.
Umm, I didn't threaten anyone or anybody. I'll spare the rigamarole of 'is libelous, must be retracted'.
And this is weird. The sig and the FROM don't match. Is this all a joke?
David Schwartz Director of Network Services Internet Gateway Connections (954)-430-3030
We only wish. If you think this is a joke, call MCI or myself.
*The lie:*
excess of an hour. This outage was deliberately engineered by AGIS/NET99.
*The fact:*
IGC's routes were pulled from the Net99 net within 90 minutes of their being turned off *at IGC's request*.
We asked for the circuit to be shutdown, not our network to be maliciously mangled by a disgruntled AGIS representative, in this case, Mr. Peter Kline, himself.
First, a little history:
*The lie:*
A few days ago, someone at Netrunner got upset with someone at NET99/AGIS and shut off _our_ service. For some reason (we strongly suspect a configuration error at NET99) our traffic failed to reroute over our MCI T1 for several hours. (They have a habit of broadcasting static routes to route around outages, thus completely screwing their multi-homed customers.)
*The truth:*
The referenced disconnection did occur, and was restored. This kind of problem is a very real danger of the way Net99 colocated in certain cities. That's why AGIS only colocates in POP's and private spaces. And in the event that a customer is dually homed and not running BGP, it's incumbent upon the customer to make sure that *both providers* are aware of the fact and properly setting local prefs. IGC did not do that and bears full responsibility for any failure to route properly during an outage.
Peter: Why did it occur? Where did we lie? OK, we'll make this easy for the folks at home. IGC's side of the Netrunner incident: Mark Cole from Netrunner intentionally shutdown OUR Net99 T1 because he couldn't reach Alan, the NOC Manager at AGIS/Net99 for hours. He got upset at your company, and took it out on us. (ok Peter, your turn, fill in the blank now) NET99's side of the story:
Aha: a true statement:
We asked AGIS for a one month refund to compensate us for our losses due to the sabotage. Instead, they issued us a _one_day_ credit.
Yes, of course it was true, we don't lie out of our ass like AGIS.
Ah, another true statement:
We got upset, both with the ability of our T to pass traffic, with NET99's continual routing problems, with NET99's backbone being disassembled from under us, and with NET99/AGIS's continual delays in migrating us to the AGIS network. We asked them to discontinue our service.
True, but we certainly wouldn't have left them without a circuit. We've tried mightily to install a new circuit to their premises, and ran into a b8zs facilities problem. We could have installed it AMI as a temporary measure with no problem.
NOT TRUE AT ALL. There are PLENTY of B8ZS facilities where we are. That's funny. In fact, if anyone would like to verify, feel free to call our BellSouth rep, Shamus Murphy at 305-351-3949. Also, we don't have any requirement that says we MUST have B8ZS, AMI would have been fine had it been offered or there was any mention of a problem. Peter, you're not a very good liar.
*Another LIE*
They did. They also deliberately and maliciously began broadcasting more specific routes for the individual class C networks in our CIDR block. (We confirmed this with MCI.)
*The truth:*
Over the months various static routes have been installed by various people on Net99 to route around some problems (a practice which predates our purchase). You run into some curious situations like a static on a Chicago router pointing to Phoenix. In this case, there were statics of differing prefix lengths on different routers. The aggregate address statement got pulled, and we didn't catch the more specifics on another router. As soon as we found out what happened, we pulled them. In fact, we pulled them while we were on the phone with MCI so that they could confirm that they were gone.
Oh, so AGIS is NOT responsible for the actions of Net99? Didn't you purchase Net99 as a whole, including liabilities?
*Another LIE*
I contacted MCI again and they contacted AGIS/NET99 with the ultimate threat being for MCI to stop carrying AGIS' traffic if they would not cease broadcasting the bogus routes. Shortly after being contacted by MCI, AGIS/NET99 stopped announcing the bogus routes to MCI, but continued advertising these routes to their other peers at the NAPs.
Nope. Like I said, MCI didn't threaten us at all, and worked with us to see that the routing was correct. Net99 isn't announcing those routes anywhere. Both AGIS and Net99 announce all routes at all exchange points to all providers in compliance with our peering contracts. If MCI had called back and said, "we still have a problem," we'd work with them until it was taken care of.
MCI contacted AGIS many times, finally, taking action into their own hands. This will all come out in court as we have MCI engineers that would be glad to testify.
participants (3)
-
David ``Joel Katz'' Schwartz -
Peter Kline, Sr. Network Engineer -
Randy Epstein