I think it's important, as a service provider, to promptly inform your customers and affected networks of issues like this. And this isn't just an Exodus issue. There are a number of providers that simply ignore requests for information or are very slow about propagating exploit details quickly enough to matter. While they're not a provider, you can send a detailed exploit to CERT and then wait months before they get around to letting other folks know about the problem. And that's from an entity that supposedly exists to propagate useful information to prevent exploits....In the meantime, affected systems fall like flies. It's fortunate that venues like NANOG and BUQTRAQ are around to disseminate this type of information in a timeframe more useful to us all. And back to the subject matter....I have no doubt that Exodus was working on the problem. It just would have been nice to be informed by *anyone* official there in a timely manner of the problem. That's both from a customer's standpoint, and the Internet at large. Chris Chris Mauritz Director, Systems Administration Rare Medium, Inc. chrism@raremedium.com -----Original Message----- From: Steven J. Sobol [mailto:sjsobol@nacs.net] Sent: Wednesday, November 18, 1998 3:07 PM To: Steve Noble Cc: Jay R. Ashworth; nanog@merit.edu Subject: Re: Exodus / Clue problems On Wed, Nov 18, 1998 at 10:23:46AM -0800, Steve Noble wrote:
that work was/is not being done on the issue. Once Exodus spent the time assembling and presenting the information to the customer, their job was done. It is now up to the customer to speak (or not speak) about the issue.
I'm not saying Exodus wasn't working on it. I would have just like to hear some confirmation. A one-line message would have been cool. And yes, I think a discussion of how ISP's deal with problems like this is a good idea. -- Steve Sobol [sjsobol@nacs.net] Part-time Support Droid [support@nacs.net] NACS Spaminator [abuse@nacs.net] Spotted on a bumper sticker: "Possum. The other white meat."
You know.. The legality issues here are amazing, just think to yourself if say a machine at your company was compromized, and your ISP told all the rest of its customers and the world of the event (and possibly why it happened). Just how would you react? On Wed, Nov 18, 1998 at 03:26:18PM -0500, Chris Mauritz wrote:
I think it's important, as a service provider, to promptly inform your customers and affected networks of issues like this. And this isn't just an Exodus issue. There are a number of providers that simply ignore requests for information or are very slow about propagating exploit details quickly enough to matter. While they're not a provider, you can send a detailed exploit to CERT and then wait months before they get around to letting other folks know about the problem. And that's from an entity that supposedly exists to propagate useful information to prevent exploits....In the meantime, affected systems fall like flies. It's fortunate that venues like NANOG and BUQTRAQ are around to disseminate this type of information in a timeframe more useful to us all.
And back to the subject matter....I have no doubt that Exodus was working on the problem. It just would have been nice to be informed by *anyone* official there in a timely manner of the problem. That's both from a customer's standpoint, and the Internet at large.
I'm glad to see that everyone is agreeing here, that there was no doubt Exodus was working hard to end the issue. Why don't we just go onto another thread :) -- ------------------------------------------------------------------------------- : Steven Noble / Network Janitor / Be free my soul and leave this world alone : : My views = My views != The views of any of my past or present employers : -------------------------------------------------------------------------------
On Wed, Nov 18, 1998 at 01:02:45PM -0800, Steve Noble wrote:
You know.. The legality issues here are amazing, just think to yourself if say a machine at your company was compromized, and your ISP told all the rest of its customers and the world of the event (and possibly why it happened). Just how would you react?
Slightly different issue. The rest of the world already saw problems. I was simply commenting on the fact that Exodus didn't respond and say they were at least working on it. -- Steve Sobol [sjsobol@nacs.net] Part-time Support Droid [support@nacs.net] NACS Spaminator [abuse@nacs.net] Spotted on a bumper sticker: "Possum. The other white meat."
I don't disagree, but on the other hand, there are hundreds if not thousands of operational issue mailing lists, I don't see why it would be expected that Exodus would post on each of them that the issue was brought up on about what was going on. Anyone who called in got a reply that the issue was being worked on, and someone on the list actually passed that on. Simply put we can either work on the issue and resolve it, or spend our time answering questions and wading through non-operational garbage to try and find out who is complaining. The real issue here is that the problem WAS resolved, and it was done in a very timly manner, much faster then I have seen most companies get them dealt with. I think we should focus on operational issues and the current round of attacks rather then grinding this one into the ground. It's over, we can stop posting about it. On Wed, Nov 18, 1998 at 06:20:13PM -0500, Steven J. Sobol wrote:
On Wed, Nov 18, 1998 at 01:02:45PM -0800, Steve Noble wrote:
You know.. The legality issues here are amazing, just think to yourself if say a machine at your company was compromized, and your ISP told all the rest of its customers and the world of the event (and possibly why it happened). Just how would you react?
Slightly different issue. The rest of the world already saw problems. I was simply commenting on the fact that Exodus didn't respond and say they were at least working on it.
-- Steve Sobol [sjsobol@nacs.net] Part-time Support Droid [support@nacs.net] NACS Spaminator [abuse@nacs.net]
Spotted on a bumper sticker: "Possum. The other white meat."
-- ------------------------------------------------------------------------------- : Steven Noble / Network Janitor / Be free my soul and leave this world alone : : My views = My views != The views of any of my past or present employers : -------------------------------------------------------------------------------
On Wed, Nov 18, 1998 at 03:29:18PM -0800, Steve Noble wrote:
I don't disagree, but on the other hand, there are hundreds if not thousands of operational issue mailing lists, I don't see why it would be expected that Exodus would post on each of them that the issue was brought up on about what was going on.
I don't know, because several Exodus employees happen to post here, maybe? On a regular basis, at that. And when everyone suddenly went silent, it didn't look too impressive for your company. Again, at the risk of repeating myself a third time: A one-line message would probably have been enough... -- Steve Sobol [sjsobol@nacs.net] Part-time Support Droid [support@nacs.net] NACS Spaminator [abuse@nacs.net] Spotted on a bumper sticker: "Possum. The other white meat."
If this issue directly affected you, you should have contacted us and you would have been given the information (as much as we could give). If you were not directly affected, or you did not contact us, you should not expect timely information. A post was made to nanog by the correct people once there was a solution and everything was over, and yet this thread STILL goes on. It makes me wonder if people want answers or something to complain about. Of course, I don't see anyone else who is posting here as an owner of one of the other blocks, so I guess Exodus is ahead in that line. I do like how everyone jumps the issue: the problem WAS taken care of and in a timly manner, much better then I have personally seen when dealing with attacks of this sort with other ISP's. All I see is a bunch of people complaining that Exodus didn't do this, or Exodus didn't do that, all of that is secondary to the primary issue, the problem was resolved. Lets get back to real operational issues. On Wed, Nov 18, 1998 at 06:34:26PM -0500, Steven J. Sobol wrote:
I don't know, because several Exodus employees happen to post here, maybe?
Exodus employee's normally post durring such things as fiber cuts and real operational issues to spread as much useful information as possible whenver possible.
And when everyone suddenly went silent, it didn't look too impressive for your company.
No one 'went' silent, most of the people who post were either asleep, not around or working on the issue.
Again, at the risk of repeating myself a third time: A one-line message would probably have been enough...
and again at the risk of repeating myself, the problem was dealt with in a timly manner, I don't see why everyone is complaining. -- ------------------------------------------------------------------------------- : Steven Noble / Network Janitor / Be free my soul and leave this world alone : : My views = My views != The views of any of my past or present employers : -------------------------------------------------------------------------------
On Wed, 18 Nov 1998, Steve Noble wrote:
If this issue directly affected you, you should have contacted us and you would have been given the information (as much as we could give). If you
For the sake of clarification, could you please define "as much as we could give"?
were not directly affected, or you did not contact us, you should not expect timely information. A post was made to nanog by the correct people once there was a solution and everything was over, and yet this thread STILL goes on. It makes me wonder if people want answers or something to complain about.
It's not over till it's over. And, AFAIK, it was not over when Exodus claimed it was. In fact, do we know as a fact that it's over now? I've been routing 209.67.50.0/24 to where it belongs (Null0), so if any access attempts were made, I wouldn't have noticed... sorry to sound in the dark here.
Of course, I don't see anyone else who is posting here as an owner of one of the other blocks, so I guess Exodus is ahead in that line.
Possibly. Then again, from what I've seen, the majority of the portscanning/flooding originated from 209.67.50.0/24, not some other provider's blocks. SO...
Exodus employee's normally post durring such things as fiber cuts and real operational issues to spread as much useful information as possible whenver possible.
I'm confused. How is a widespread network security issue not of operational concern? Thanks, -asr (speaking on behalf of myself only)
On Thu, Nov 19, 1998 at 01:16:22AM -0500, Adam Rothschild wrote:
On Wed, 18 Nov 1998, Steve Noble wrote:
If this issue directly affected you, you should have contacted us and you would have been given the information (as much as we could give). If you
For the sake of clarification, could you please define "as much as we could give"?
Exactly what I said, as much as they could give. If you turn the situation around and you were the one with the security issue, exactly how much information would you want your ISP to give out? Probably very little, other then that the situation has been handled. I am sure that you would also not want your ISP medling in your situation unless you requsted it. You have to remember, Exodus is only the ISP, while they are happy to contact and assist any customer with a security problem, it is the customers responsibility to deal with it. If you have any other issues with the customer feel free to contact them directly or Exodus if they are uncooperative.
It's not over till it's over. And, AFAIK, it was not over when Exodus claimed it was. In fact, do we know as a fact that it's over now? I've been routing 209.67.50.0/24 to where it belongs (Null0), so if any access attempts were made, I wouldn't have noticed... sorry to sound in the dark here.
Of course, all I've seen have been very small issues which could be attributed to dns lookups and other such things, nothing malicious since that day.
Possibly. Then again, from what I've seen, the majority of the portscanning/flooding originated from 209.67.50.0/24, not some other provider's blocks. SO...
Not so true, you posted some yourself : Date: Mon, 16 Nov 1998 17:30:39 -0500 (EST) From: Adam Rothschild <asr@millburn.net> Subject: Exodus: this is bad Hrrrm, I'm seeing 38.29.63.195 trying to telnet to every IP addr in one of my Exodus /24's... (around 4.30p EST) --- Of course I see no reason why you put Exodus: this is bad as the topic of the post but well, I don't understand half of what you say anyways :) Did you have problems contacting PSI about this and getting it resolved? We're they helpful? I am sure people from PSI read this list, I haven't seen any responses from them. Also This one : Date: Mon, 16 Nov 1998 18:05:25 -0500 (EST) From: Adam Rothschild <asr@millburn.net> Subject: RE: Exodus: this is bad True... and in rapid succession, too. Anyone notice anything fishy from this fucker as well? [root@oven log]# cat secure | more Nov 15 23:41:36 oven in.telnetd[20426]: connect from 207.104.58.91 Nov 15 23:41:36 oven in.telnetd[20427]: connect from 207.104.58.91 --- Now other then your seemingly angry demeanor, this set of IP's seem to be causing you problems too.. How did the ISP holding these ip's react? is the system shut down? I didn't see any posts from them on NANOG.. And of course, without your name attached : Date: Mon, 16 Nov 1998 17:16:36 -0500 From: Richard Irving <rirving@onecall.net> Subject: Another origin IP 209.119.115.65 telnetd a mile a minute....... --- It seems pretty clear to me that more then just Exodus was involved to a bigger degree then you were saying... I'd quote more, but I don't want to have a 100 page post.
I'm confused. How is a widespread network security issue not of operational concern?
Of course a widespread issue is, but harping on the people who resolved the issue is not. I understand now why most large ISP's don't even discuss problems publically just from the amount of trouble it causes. Just ignore and it all goes away. -- ------------------------------------------------------------------------------- : Steven Noble / Network Janitor / Be free my soul and leave this world alone : : My views = My views != The views of any of my past or present employers : -------------------------------------------------------------------------------
You know.. The legality issues here are amazing, just think to yourself if say a machine at your company was compromized, and your ISP told all the rest of its customers and the world of the event (and possibly why it happened).
and what if it turned out to be incorrectly diagnosed? lawyer fodder^2. randy
On Wed, Nov 18, 1998 at 07:08:39PM -0800, Randy Bush wrote:
and what if it turned out to be incorrectly diagnosed? lawyer fodder^2.
That is exactly the issue, I'm sure no one wants their security issues aired out infront of the world, especially by their provider.. -- ------------------------------------------------------------------------------- : Steven Noble / Network Janitor / Be free my soul and leave this world alone : : My views = My views != The views of any of my past or present employers : -------------------------------------------------------------------------------
participants (5)
-
Adam Rothschild -
Chris Mauritz -
Randy Bush -
Steve Noble -
Steven J. Sobol