December 2004 - Test - lists.nanog.org

Re: [OT] Re: Banned on NANOG
by Paul Vixie 02 Dec '04

02 Dec '04

> From: Daniel Golding <dgolding(a)burtongroup.com> > ... > Its entirely possible for nanog-l to be self policing, or, failing > that, for users to simply use procmail on those who wander off-topic > (for some definition of off-topic). Putting an [OT] subject banner on > such posts is also nice. i don't want widescale procmail to be the only way nanog@ is readable by a big subset of the netops community, simply because i know a lot of the folks here (lazy overworked disorganized bums, mostly) and if it takes way more effort to be subscribed than not, many will just unsubscribe. > There's such a thing as throwing the baby out with the bathwater. When > highly clued, genuinely contributing folks are treated poorly for the > occasional in-joke or comment, the S:N ratio will suffer in the longer > term. nope nope nope no-no-nope. that's a subjective standard. there's no way to moderate based on "does more good than harm" without strong and formal and objective definitions of what "good" is and what "harm" is, plus an appeals process. trust me: we don't want "strong" "formal" "process". (my own system, which has produced only two warnings in about 10 years, is to make the good:harm ratio high enough in any given message that the "in" jokes are merely a tolerable percentage of the mass of THAT message; what i see some other bums doing, though, is pure-"in"-joke messages.) > I'm certainly hoping that the network operations community will feel > no need to "talk with their feet" after we all sit down with the Merit > staff and let our feelings be known, but that is certainly a possibility. like the libertarians say, "use your dollar votes!" i'm comfortable with a system whereby susan occasionally turns around in the front seat of ye olde station wagon and says "you'd better stop that right now, because if i have to stop this car and come back there, you'll be sorry" and the rest of the time we just keep the fighting down to (bloodless) dull roar. but if you have a better system in mind you should propose it; and if you can't get traction for it inside nanog, there's always room for another ops list. (in Usenet days we used to say "could you move this thread to $other_group, where it will be on-topic, and where i'm not a subscriber?" and it WORKED a lot of the time, just to wake folks up and show that topic-consensus was a property both nec'y and desireable in ALL forums, digital or otherwise.)

1 0

RE: How many backbones here are filtering the makelovenotspam scr eensaver site?
by Chad Skidmore 02 Dec '04

02 Dec '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > -----Original Message----- > From: Steven Champeon [mailto:schampeo@hesketh.com] > Posted At: Thursday, December 02, 2004 1:09 PM > Posted To: NANOG > Conversation: How many backbones here are filtering the > makelovenotspam scr eensaver site? > Subject: Re: How many backbones here are filtering the > makelovenotspam scr eensaver site? > > My point was to Martin's question about what would happen if > - god forbid - there were large botnets under the control of > spammers; a careful reading will suggest that my major point > was, duh, that there already are large botnets under the > control of spammers. I realize that is the point you were trying to make. I also realize that Martin is pretty well aware of botnets and the threat they create. I suspect that most other readers on NANOG are also well aware. What doesn't seem to be as common knowledge as I would expect is that botnets are a commodity. As such they are traded, sold, purchased and even stolen. That last point is particularly important in this case. Lycos has created a large botnet (at least by most people's definition) that is hidden in the guise of a screen saver claiming to only go after the "bad guys". This botnet uses a command and control server that is now well publicized, and uses a communication channel that is not encrypted or obfuscated in any way. That makes it a botnet just asking to be stolen. Fortunately the C&C server is blackholed by what seem to be a large number of providers and the botnet is now fairly useless. > Good point. Simply put, I can (and do) read my own mail server > logs. And I can see that many ISPs - regardless of what they may be > doing in onesy-twosy increments - simply aren't doing enough > to prevent new botnet infections from wasting my server's > cycles in futile attempts to deliver spam, outscatter, virus > warnings, etc. etc. ad infinitum. It is certainly more than "onesy-twosy increments" but I agree that the problem is large enough that it certainly feels like a weak attempt from the average user/operator's point of view. > This costs me time and money, and many of the same ISPs > mentioned above are simply cost-shifting their own > responsibility onto me and everyone else, and I'm tired of it. I encourage everyone to vote with their wallet when it comes to this type of thing. Buy your transit from organizations with dedicated security teams that actively engage in SPAM/Bot/Worm/Viri fighting efforts. Those things cost money and take time and are usually unacknowledged efforts. Larger providers seem to make easier targets when it comes to placing blame and saying that they aren't doing enough to combat miscreant activity. I don't believe that is the case overall. They just have a much larger customer base, higher volumes of traffic to inspect, and more politics to work within. > Not to say there aren't responsible ISPs, and I hope that > anyone who /is/ a part of the solution, rather than the > fertile substrate for the problem, is capable of recognizing > that and not taking offense when I point out there are others > who could do more. I believe that EVERYONE could do more on this front. It is a moving battle that requires constant improvement just to stay afloat, let alone get ahead. For those genuinely interested in improving what they are doing on this front I strongly encourage you to attend the NSP-Sec BOFs at NANOG. You might be surprised what you learn and who you meet that can be helpful. > As for go180.net, you don't show up much on my radar, but on > Nov 9th we were hit by a spammer from > SpokaneHotZone-63.go180.net [66.225.5.63]. > I trust this is not a legitimate mail server and I can block > it and any other host that looks like it within the same > domain, right? Thanks. > Otherwise, you may want to do something to distinguish it > from the other generic hosts in the same range. Glad you don't see much from us, must mean that the effort put forth by some of our team is not going to waste. You are correct, that is not a legitimate mail server but is an IP from a City Wide wireless network. That network has since been secured to restrict TCP 25 outbound (along with other typical miscreant traffic) so you shouldn't see anything again from that network on port 25. If we rise up on your radar in the future feel free to make use of the typical NOC and Abuse e-mail addresses, they do get answered and acted upon here. Regards, Chad - ---------------------------- Chad E Skidmore One Eighty Networks, Inc. http://www.go180.net 509-688-8180 -----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBQa+VUk2RUJ5udBnvEQJXPQCeMhYgS4vHzmjP2fpgVeEFySQWw4QAn1f/ g70E3QaL3VOcZvILXD80AqjF =he0W -----END PGP SIGNATURE-----

3 2

RE: What good is a noc team? How do you mitigate this? [was: How many backbones ...]
by Chad Skidmore 02 Dec '04

02 Dec '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > -----Original Message----- > From: Gadi Evron [mailto:ge@linuxbox.org] > Sent: Thursday, December 02, 2004 3:21 PM > To: Chad Skidmore > Cc: Aaron Glenn; nanog(a)merit.edu > Subject: What good is a noc team? How do you mitigate this? > [was: How many backbones ...] > > > Okay, making this an operational issue. Say you are attacked. > Say it isn't even a botnet. Say a new worm is out and you are > getting traffic from 19 different class A's. > > Who do you call? What do you block? > > How can a noc team here help? > > "Please block any outgoing connections from your network to > ours on port 25? Please?" I tried this once.. it doesn't > help. I ended up blackholing an entire country just to > mitigate it a bit, for a few hours. > > Any practical suggestions? > > Gadi. > Well, the easy answer is that it depends. Lets use SQL Slammer as one example that might be comparable to the scenario you mention. During Slammer some networks did stay up. We'd have to ask each one of them what they did to know why they stayed up but I think I can guess at some. Shortly after Slammer there was a NANOG presentation on Slammer and some discussion at the NSP-Sec BOF at that NANOG regarding why some people survived and others didn't. What came out of that was enlightening, if not obvious in hind sight. 1. Those providers that made use of contacts at other providers and worked together, shared information, etc. were less affected than those that did not. 2. Those providers that had various mechanisms in place for just such an issue did better than those that did not. This included, but was not limited to, darknet monitoring & quick reaction to darknet data anomalies, automated and semi-automated sifting of Netflow data, pre-staged classification ACLs on at least key backbone/peering/transit routers, and BGP (or other) triggered blackhole mechanisms. 3. Teams with dedicated incident response teams did better than those that didn't. 4. Those with grossly oversubscribed networks did worse than those with sufficient bandwidth to handle the ebb and flow of traffic that rides the Internet today. Good traffic engineering practices don't mean that you have to purchase lots of excess bandwidth to make this happen. Not being oversubscribed is also not just an issue of circuit utilization. For example, make sure you have enough CPU on your routers, line cards, whatever so that you can turn various features on to help track and mitigate an attack without making your routers fall over. So, armed with that data you can assume the following. With good darknet monitoring practices you would likely see a rapid up tick in scanning, backscatter, etc. and could start investigating the cause prior to the issue becoming service affecting. Maybe it is so crazy and randomized that you don't see it on your darknet monitoring but you see it on your PPS data collection. More often than not I know we see indications of miscreant activity on PPS monitoring first. The classification ACLs are a good way to turn the router into a poor mans sniffer (assuming it isn't so heavily loaded already that it falls over) so you can see what types of traffic you are dealing with. Using MCI/UUs method you could track any spoofed traffic back to where it enters your network pretty easily. I know that Chris and company do it with amazing speed across 701. If it works for them then it likely works for the rest of you. Netflow data would likely lead you to sources of the most pain so you could go after those first. Fighting an attack isn't always about making the attack go away. Often times the key to not getting killed is to find the "big guns" and get them silenced first. Sure, you're still getting shot, but it isn't going to kill you and you can take some additional time to find the smaller guns. If you are seeing the bulk of the attack come from a few sources let their security teams deal with it and take the pain away from you. Armed with the data you glean from this approach you will usually be able to get a positive response from your upstream or peers. If not make a quick note to yourself that you need to replace them once your attack is over and done with. If all else fails blackhole the host under attack at your borders, or even better on your upstream's network via BGP triggered blackhole (if they don't support it make a note to replace them with someone who does when the attack is over). You might sacrifice that host but you'll save the rest of your network and likely buy yourself some more time to track back to the source and kill it. I'm certainly not suggesting I have all the answers or that I have it all figured out. I also realize that the world is not a rosy place where inter-provider communication is perfect and I always get the answers I need when I call them. I'm just tired of seeing people play the victim, complaining how the "Big Providers" won't protect them, etc. without looking in the mirror and deciding that today is the day I take my network back and take care of myself. Gadi and Aaron, this isn't directed specifically at you so please don't take that as a personal flame to you. I've personally had great luck in getting quick reaction from a number of providers when an attack is ongoing. That certainly isn't always the case but more often than not it is. Some that have done a great job in the past that come to mind for me are 701, 2914, 3549, 1239, 3356. That isn't all of them but they are the ones that come to mind the quickest. Chad - ---------------------------- Chad E Skidmore One Eighty Networks, Inc. http://www.go180.net 509-688-8180 -----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBQa+tZ02RUJ5udBnvEQJk8QCeLel/Nj4FFIIXInGSEF/i5RovlW0An1E8 UECC1HswfhvbtyIJi0QL8vS0 =FHWN -----END PGP SIGNATURE-----

1 0

RE: How many backbones here are filtering the makelovenotspam scr eensaver site?
by Chad Skidmore 02 Dec '04

02 Dec '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > -----Original Message----- > From: Aaron Glenn [mailto:aaron.glenn@gmail.com] > Sent: Thursday, December 02, 2004 2:52 PM > To: Chad Skidmore > Cc: nanog(a)merit.edu > Subject: Re: How many backbones here are filtering the > makelovenotspam scr eensaver site? > > On Thu, 2 Dec 2004 12:55:02 -0800, Chad Skidmore > <cskidmore(a)go180.net> wrote: > > To your other point, how do you know that other botnets are > not being > > identified and taken down every day by network operators? I > know for a > > fact that they are, they just are not nearly as public as > this one so > > those activities go largely unacknowledged. > > I find that very hard to believe. After getting nailed > (900Mbps/4000 unique hosts from the 1 second network capture > we could get) by a > relatively(?) small botnet, and doing all the hard work for > them, not one of the 20 networks we contacted (9 being very > very large) gave a flying peice of excriment as to what was going > on. > > It wasn't the first and probably won't be the last. Is that > too small a fish to fry? Do ops only care when its 2Gbps of > sustained traffic chocking their border routers, because I'm > half way there... > > </rant> > > Regards, > Aaron > Sorry your experience has been different, this is definitely one of those YMMV kinds of deals. That is a significant attack by most anyone's standards. Getting to the right security team usually ends up being the challenge. Once there however we have found many providers do a great job of dealing with attacks quickly. Use of BGP triggered blackholes can be a great help and going to the NOC/Abuse team with lots of good information from the start helps you get to the people that can pull the attack of quickly. You have to remember that, like all of us, larger service providers have their share of low clue factor customers. The quicker you can help them realize that you have a fairly high clue factor the quicker you'll get to folks on their side with a high clue factor. During times of outages, attacks, etc. it is easy to get agitated quickly and that often times doesn't help you get through the first couple of barrier noc techs. Anyway, just my $.02 worth and as we can see YMMV. Chad - ---------------------------- Chad E Skidmore One Eighty Networks, Inc. http://www.go180.net 509-688-8180 -----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBQa+fTk2RUJ5udBnvEQIMeACeOEuV1XA64yujh+hKEypGPedyL4kAoN0I tjq/VueRQrb0gjJ2aHxHy4KY =yFzW -----END PGP SIGNATURE-----

2 1

RE: How many backbones here are filtering the makelovenotspam scr eensaver site?
by Hannigan, Martin 02 Dec '04

02 Dec '04

> -----Original Message----- > From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] > Sent: Thursday, December 02, 2004 4:14 PM > To: nanog list > Subject: Re: How many backbones here are filtering the makelovenotspam > scr eensaver site? > > > > on Thu, Dec 02, 2004 at 08:58:03PM +0000, Christopher L. Morrow wrote: > > > > On Thu, 2 Dec 2004, Steven Champeon wrote: > > > > > > > > on Thu, Dec 02, 2004 at 02:56:29PM -0500, Hannigan, Martin wrote: > > > > Possibly. What will happen if the Lycos botnet gets hijacked? > > > > > > > > The conversations between the clients and the servers > don't appear > > > > to be keyed. If a million clients got owned, it would be the > > > > equivalent of an electronic Bubonic Plague with no antidote. > > > > > > You mean, like the existing botnets we already know exist but are > > > already under the control of spammers? > > > > > > What's the difference? Why is everyone so upset about > Lycos and nobody > > > seems to be doing much of anything about the /existing > botnets/, which > > > conservative estimates[1] already put at anywhere from > 1-3K per botnet > > > to upwards of 1-5M hosts total[2]? > > > > perhaps the difference is 'reponsible people' don't go out > and recruit > > botnets... Lycos, as a corporate entity with it's business > model dependent > > upon the health and wellbeing of the Internet would try to be > > 'responsible', or so I would have thought. > > I agree. I also think it's up to the companies providing the Internet > connectivity to the non-Lycos-"owned" botnets to prevent such activity > from affecting others. > > > arguing that there are murderers and rapists out there and > that 'nothing > > is being done' is hardly reason to become one yourself. > > I couldn't agree more that vigilantism isn't the answer. My earlier > remarks were directed to the shock and awe evident in the possibility > that - via Lycos - there might be, heaven forbid, /large numbers of > computers under the control of spammers, that could be used > in spamming > and abuse/. Can you direct me toward a singluar entity of 1MM bots controlled by a single master? > > All I was pointing out was that, surprise, surprise, there > already are. > So why anyone thinks Lycos' botnet being hacked is /any > different/ from > /the current situation/ is utterly beyond my ken. Why would > any spammer > bother to hack Lycos' botnet? They /already have their own/. I think you might be behind on what's going on in botland lately.

5 6

RE: How many backbones here are filtering the makelovenotspam scr eensaver site?
by Hannigan, Martin 02 Dec '04

02 Dec '04

> -----Original Message----- > From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] > Sent: Thursday, December 02, 2004 5:21 PM > To: nanog(a)merit.edu > Subject: RE: How many backbones here are filtering the makelovenotspam > scr eensaver site? > > > [SNIP] > > > As for go180.net, you don't show up much on my radar, but on > > Nov 9th we were hit by a spammer from > > SpokaneHotZone-63.go180.net [66.225.5.63]. > > I trust this is not a legitimate mail server and I can block > > it and any other host that looks like it within the same > > domain, right? Thanks. > > Otherwise, you may want to do something to distinguish it > > from the other generic hosts in the same range. > > Glad you don't see much from us, must mean that the effort put forth > by some of our team is not going to waste. You are correct, that is > not a legitimate mail server but is an IP from a City Wide wireless > network. That network has since been secured to restrict TCP 25 > outbound (along with other typical miscreant traffic) so you > shouldn't see anything again from that network on port 25. If we rise > up on your radar in the future feel free to make use of the typical > NOC and Abuse e-mail addresses, they do get answered and acted upon > here. > > Glad to hear that. Overall, I'm offering some operational content on the publicity intensive Lycos botnet and provide some level of operational analysis free of judgement of Lycos. I'd be happy to argue about breadth, depth, and width of botnets and their commodity status in email. :) -M<

1 0

RE: How many backbones here are filtering the makelovenotspam scr eensaver site?
by Hannigan, Martin 02 Dec '04

02 Dec '04

> -----Original Message----- > From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] > Sent: Thursday, December 02, 2004 4:28 PM > To: nanog(a)merit.edu > Subject: Re: How many backbones here are filtering the makelovenotspam > scr eensaver site? > > > > on Thu, Dec 02, 2004 at 04:15:34PM -0500, Hannigan, Martin wrote: > quoting me: > > > My point was to Martin's question about what would happen if - god > > > forbid - there were large botnets under the control of spammers; a > > > careful reading will suggest that my major point was, > duh, that there > > > already are large botnets under the control of spammers. > > > > Um, not 1 million bots - in concert. > > And you know this how, exactly? I'm sure not convinced. http://w3.cambridge-news.co.uk/business/story.asp?StoryID=65877 "Lycos Europe's 20 million users will all be invited to download the software, but it is available to anyone with an internet connection running either Windows or Mac OSX or Mac OS9 operating systems." http://edition.cnn.com/2004/TECH/internet/12/02/anti.spamvigi.ap/ "Around 65,000 people already signed up for the offensive, called "Make Love not Spam" before Tuesday's official launch on a website by the same name, the company said. It is urging its 22 million users to download the screen-saver, but says anyone with a computer is welcome to it."

2 1

RE: How many backbones here are filtering the makelovenotspam scr eensaver site?
by Hannigan, Martin 02 Dec '04

02 Dec '04

> -----Original Message----- > From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] > Sent: Thursday, December 02, 2004 4:09 PM > To: nanog(a)merit.edu > Subject: Re: How many backbones here are filtering the makelovenotspam > scr eensaver site? > > > > on Thu, Dec 02, 2004 at 12:55:02PM -0800, Chad Skidmore wrote: > quoting me: > > >What's the difference? Why is everyone so upset about Lycos and > > >nobody seems to be doing much of anything about the /existing > > >botnets/, which conservative estimates[1] already put at anywhere > > >from 1-3K per botnet to upwards of 1-5M hosts total[2]? > > > > Well, the primary difference is that Lycos is trying to market what > > they are doing as a "good" thing in a fairly public manner. If their > > vigilante efforts become accepted as "OK" then it further opens the > > door for others to take the next step towards making dDOS attacks ok > > as long as you feel your motivations are pure. As network operators > > we all need to make sure that we enforce our AUPs and make it known > > that breaking those AUPs is not ok just because you feel > your motives > > are pure. Most AUPs have some language that basically states that > > dDOS and simlar activities are "bad" and we will take action if you > > engage in said "bad" activities. > > My point was to Martin's question about what would happen if - god > forbid - there were large botnets under the control of spammers; a > careful reading will suggest that my major point was, duh, that there > already are large botnets under the control of spammers. Um, not 1 million bots - in concert. -M<

2 1

RE: How many backbones here are filtering the makelovenotspam scr eensaver site?
by Chad Skidmore 02 Dec '04

02 Dec '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -----Original Message----- From: Steven Champeon [mailto:schampeo@hesketh.com] Posted At: Thursday, December 02, 2004 12:22 PM Posted To: NANOG Conversation: How many backbones here are filtering the makelovenotspam scr eensaver site? Subject: Re: How many backbones here are filtering the makelovenotspam scr eensaver site? >>on Thu, Dec 02, 2004 at 02:56:29PM -0500, Hannigan, Martin wrote: >> Possibly. What will happen if the Lycos botnet gets hijacked? >> >> The conversations between the clients and the servers don't appear >> to be keyed. If a million clients got owned, it would be the >> equivalent of an electronic Bubonic Plague with no antidote. > >You mean, like the existing botnets we already know exist but are >already under the control of spammers? > >What's the difference? Why is everyone so upset about Lycos and >nobody seems to be doing much of anything about the /existing >botnets/, which conservative estimates[1] already put at anywhere >from 1-3K per botnet to upwards of 1-5M hosts total[2]? Well, the primary difference is that Lycos is trying to market what they are doing as a "good" thing in a fairly public manner. If their vigilante efforts become accepted as "OK" then it further opens the door for others to take the next step towards making dDOS attacks ok as long as you feel your motivations are pure. As network operators we all need to make sure that we enforce our AUPs and make it known that breaking those AUPs is not ok just because you feel your motives are pure. Most AUPs have some language that basically states that dDOS and simlar activities are "bad" and we will take action if you engage in said "bad" activities. To your other point, how do you know that other botnets are not being identified and taken down every day by network operators? I know for a fact that they are, they just are not nearly as public as this one so those activities go largely unacknowledged. Regards, Chad - ---------------------------- Chad E Skidmore One Eighty Networks, Inc. http://www.go180.net 509-688-8180 -----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBQa+BJU2RUJ5udBnvEQLzYACg7cc+5QPHyY0km3kK2H4kQ7ovcdMAoOG3 /C/QeyrlyeOxbVu2uQkJPi3a =ybZk -----END PGP SIGNATURE-----

2 1

"Make love, not spam"....
by Fergie (Paul Ferguson) 02 Dec '04

02 Dec '04

Techdirt has an article this morning that discusses how Lycos Europe is encouraging their users to run a screensaver that constantly "pings servers suspected to be used by spammers" and also suggests that "In other words, it's a distributed denial of service attack against spammers by Lycos." The Techdirt article referenced is on Heise Online: http://www.heise.de/english/newsticker/news/53697 I'd be curious to hear what NANOG readers thoughts are on this. Techdirt is located at http://www.techdirt.com/ - ferg -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(a)netzero.net or fergdawg(a)sbcglobal.net

8 7