RE: NOC Automation / Best Practices
-----Original Message----- Date: Wed, 08 Sep 2010 08:54:20 -0700 From: Charles N Wyble <charles@knownelement.com> Subject: NOC Automation / Best Practices To: nanog@nanog.org
NOGGERS,
(...) The way I see it, an ounce of prevention is worth a pound of cure. Along those lines, I'm putting in some mitigation techniques are as follows (hopefully this will reduce the number of incidents and therefore calls to the abuse desk). I would appreciate any feedback folks can give me.
A) Force any outbound mail through my SMTP server with AV/spam filtering. B) Force HTTP traffic through a SQUID proxy with SNORT/ClamAV running (several other WISPs are doing this with fairly substantial bandwidth savings. However I realize that many sites aren't cache friendly. Anyone know of a good way to check for that? Look at HTTP headers?). Do the bandwidth savings/security checking outweigh the increased support calls due to "broken" web sites? C) Force DNS to go through my server. I hope to reduce DNS hijacking attacks this way.
Thanks!
For either A, B or C you won't get my business, let alone a combination of all 3. *wah!* There is too much FORCE here. :-) #m
On Sep 8, 2010, at 12:59 PM, Martin Hotze wrote:
-----Original Message----- Date: Wed, 08 Sep 2010 08:54:20 -0700 From: Charles N Wyble <charles@knownelement.com> Subject: NOC Automation / Best Practices To: nanog@nanog.org
NOGGERS,
(...) The way I see it, an ounce of prevention is worth a pound of cure. Along those lines, I'm putting in some mitigation techniques are as follows (hopefully this will reduce the number of incidents and therefore calls to the abuse desk). I would appreciate any feedback folks can give me.
A) Force any outbound mail through my SMTP server with AV/spam filtering. B) Force HTTP traffic through a SQUID proxy with SNORT/ClamAV running (several other WISPs are doing this with fairly substantial bandwidth savings. However I realize that many sites aren't cache friendly. Anyone know of a good way to check for that? Look at HTTP headers?). Do the bandwidth savings/security checking outweigh the increased support calls due to "broken" web sites? C) Force DNS to go through my server. I hope to reduce DNS hijacking attacks this way.
Thanks!
For either A, B or C you won't get my business, let alone a combination of all 3. *wah!* There is too much FORCE here. :-)
So A) is fairly common in "hotel" networks. Make sure you only are looking at tcp/25 and not tcp/587. B) is fairly common in "hotel" networks. There are a lot of things you need to do to make things work "correctly". I've found some websites will actually block you if you are behind a cache and it adds the Via: headers per standard. I've had to turn a lot of these options off in my home setup (ie: break standards on purpose). You may also want to reach out to the CDNs themselves, eg:akamai, llnw, etc.. as they may have a way to just drop the cache in your network and send your customers there automagically. C) is also common in a number of networks. You may want to 'whitelist' some other common open resolvers that are intended to be open. (eg: OpenDNS). You may be able to approach dns operators to have them put an instance in your network. Make sure you don't construct the network such that you're forced to do this for all subscribers. Many WISPs have a 'flat' or simple routed network. This is because the hardware doesn't always support nice routing protocols eg: OSPF/ISIS so you're stuck with RIP/RIPv2 (ick!). Here's some settings that I use, to optimize for software updates and other items. If you have a lot of Windows machines, you may need to read this page: http://wiki.squid-cache.org/SquidFaq/WindowsUpdate -- snip -- # hide our existance forwarded_for off via off # workaround facebook bug ignore_expect_100 on # Comcast is sometimes busted ignore_unknown_nameservers off # allow up to 8G to be cached maximum_object_size 8192 MB # allow squid daemon to get 1024 MB ahead of client read_ahead_gap 1024 MB
We run a *free* WISP and block 25 but I'm not sure why you would want to force all traffic through it. That's a touchy argument but it would really bother me as a paying subscriber. We use customized squid to haproxy (custom) to route traffic. Our main business is ddos protection and we use datacenters in multiple places. However, when we wanted to offer free wireless out of our office in Oxford, MS we found getting 10Mbps+ of bandwidth was nearly impossible. So we setup a few caching load balancers in the Oxford office which connect out through two load-balanced proxy systems sitting in the datacenter on 1Gbps connections. Works well - some sites cannot be cached but we are able to enforce gzip compression on everything, cache dns, images, etc. We get upwards of 100 users concurrently. Works beautifully - we can see our office pushing 40Mbps+ and the downstream at 5Mbps or less (total available we have is 2x10Mbps links). Requires some serious tuning but if you got time, this is the way to go. We do block bittorrent traffic which we find more of a threat than properly monitored smtp traffic. -----Original Message----- From: Martin Hotze <M.Hotze@hotze.com> Date: Wed, 8 Sep 2010 16:59:14 To: nanog@nanog.org<nanog@nanog.org> Subject: RE: NOC Automation / Best Practices
-----Original Message----- Date: Wed, 08 Sep 2010 08:54:20 -0700 From: Charles N Wyble <charles@knownelement.com> Subject: NOC Automation / Best Practices To: nanog@nanog.org
NOGGERS,
(...) The way I see it, an ounce of prevention is worth a pound of cure. Along those lines, I'm putting in some mitigation techniques are as follows (hopefully this will reduce the number of incidents and therefore calls to the abuse desk). I would appreciate any feedback folks can give me.
A) Force any outbound mail through my SMTP server with AV/spam filtering. B) Force HTTP traffic through a SQUID proxy with SNORT/ClamAV running (several other WISPs are doing this with fairly substantial bandwidth savings. However I realize that many sites aren't cache friendly. Anyone know of a good way to check for that? Look at HTTP headers?). Do the bandwidth savings/security checking outweigh the increased support calls due to "broken" web sites? C) Force DNS to go through my server. I hope to reduce DNS hijacking attacks this way.
Thanks!
For either A, B or C you won't get my business, let alone a combination of all 3. *wah!* There is too much FORCE here. :-) #m
For either A, B or C you won't get my business, let alone a combination of all 3. *wah!* There is too much FORCE here. :-)
Agreed. Just provide tubes and shut down infected customers until they clean up. Keep it simple. For content delivery, there are several non-evil ways of doing localization that don't involve caching the internet to hell and back - they'll work a LOT better and they place support in the content distributors court . At the very least, peering with them at the local IX should help a lot. If you're going to try to control mail, do it explicitly. Block 25 and provide a ratelimited, scoped relay server that people have to manually use if they NEED to use 25 - again, they should be using MSA instead, but...
On Sep 8, 2010, at 9:59 AM, Martin Hotze wrote:
-----Original Message----- Date: Wed, 08 Sep 2010 08:54:20 -0700 From: Charles N Wyble <charles@knownelement.com> Subject: NOC Automation / Best Practices To: nanog@nanog.org
NOGGERS,
(...) The way I see it, an ounce of prevention is worth a pound of cure. Along those lines, I'm putting in some mitigation techniques are as follows (hopefully this will reduce the number of incidents and therefore calls to the abuse desk). I would appreciate any feedback folks can give me.
A) Force any outbound mail through my SMTP server with AV/spam filtering. B) Force HTTP traffic through a SQUID proxy with SNORT/ClamAV running (several other WISPs are doing this with fairly substantial bandwidth savings. However I realize that many sites aren't cache friendly. Anyone know of a good way to check for that? Look at HTTP headers?). Do the bandwidth savings/security checking outweigh the increased support calls due to "broken" web sites? C) Force DNS to go through my server. I hope to reduce DNS hijacking attacks this way.
Thanks!
For either A, B or C you won't get my business, let alone a combination of all 3. *wah!* There is too much FORCE here. :-)
#m
+1 Owen
participants (5)
-
Jared Mauch -
khatfield＠socllc.net -
Martin Hotze -
Nathan Eisenberg -
Owen DeLong