March 2004 - Test - lists.nanog.org

Re: Enterprise Multihoming
by John Neiberger 13 Mar '04

13 Mar '04

>>> "Stephen J. Wilcox" <steve(a)telecomplete.co.uk> 3/12/04 9:06:38 AM >>> >I dont agree that connecting to two+ upstreams makes you better. In my >experience end networks have a couple of orders of magnitude more downtime than >a PoP in any reasonably large ISP. Ie the percentage theoretical improvement is >small. > >In addition you seriously increase the complexity of your system, chances are >you're using the cheapest kit you could find (or at least cheaper and smaller >than what I would use).. its not great at BGP and may fall over when you get a >minor DoS attack, you probably generate flaps quite a bit from adhoc changes and >if you're announcing a /24 then thats going to get you dampened quickly.. so you >actually create a new weakest link. Also most of the corporates I've dealt with >take defaults rather than full tables.. so if the provider does have an issue >you still forward the traffic, theres no failover of outbound routing. > >Even if you spend (waste) the money on some decent gear, you're on your own and >when a problem occurs the ISPs are going to be less helpful to you (not by >choice, I mean they dont have control of your network any more.. there knowledge >of whats causing problems is limited to the bit that they provide to you), so >chances are your problems may be more serious and take longer to diagnose and >fix. The above arguments are rather similar to the ones I heard on the other discussion list I mentioned, and they were somewhat compelling. > >IMHO avoid multihoming. You will know when you are big enough and you *need* to >do it, if you're not sure or you only want to do it cause you heard everyone >else is and its real cool then I suggest you dont. In our case, we already are multihoming and I'm considering moving away from that to a simpler solution. It's been my assertion that we didn't need to multihome in the beginning. The decision was made at a level higher than me. However, now that we have it I'm trying to determine the pros and cons related to moving to a single provider. Thanks, John --

3 2

RE: who offers cheap (personal) 1U colo?
by Michel Py 13 Mar '04

13 Mar '04

>> Michel Py wrote: >> I pay $36/mo for my aDSL. $50 _more_ sounds a lot. > Paul Vixie wrote: > rest assured, some of the mail i've received in > response to this has even lower price points. I'm not the one to convince; you're preaching the choir. What do you say to the non-geek female Supreme Commander that oversees the home budget? I'm walking a fine line already with 9 computers, a 12U router that makes more noise than a 747, and a fair amount of other indispensable network geek items. > most domestic residences don't have UPS I do, and the geek clientele that reads this that would colo a 1U likely does as well. As a matter of fact, I have 5 UPSes at home: one for the server, one for the router (needs one on its own), one for my PC, one for my wife's PC and even one for the DirecTiVo DVR (an old 200va that can't power any recent computer). > or high speed IP with static global addresses. I also have this and it would also be higher on the list of many network geeks than the 1U colo me thinks. > jlewis(a)lewis.org wrote: > (though I imagine they'd also do some personal web hosting > and maybe other things as long as the machine was there), I don't see what. For the kind of personal web hosting I care of, my 384 upstream is more than I need. Besides, half of the stuff on my home page would have to come from the home server in the first place. > You'd have to be a pretty hard core netgeek and have the > disposible income ($600/year + the server...I can think > of lots of better ways to spend that) So can my wife. Michel.

1 0

RE: who offers cheap (personal) 1U colo?
by Michel Py 13 Mar '04

13 Mar '04

> Paul Vixie wrote: > every time i tell somebody that they shouldn't bother > trying to send e-mail from their dsl or cablemodem ip > address due to the unlikelihood of a well staffed and > well trained and empowered abuse desk defending the > reputation of that address space, i also say "buy a > 1U and put it someplace with a real abuse desk, and > use your dsl or cablemodem to tunnel to that place." <me puts the devil's advocate suit on> $50 is a lot of money; I currently send email from my aDSL address because a) my ISP's smarthost sucks b) historically their SMTP hosts have been blacklisted more than mine c) even if they did not suck (which has improved a lot recently, actually) they still won't accept large attachments or mailing-list traffic. I pay $36/mo for my aDSL. $50 _more_ sounds a lot. </me puts the devil's advocate suit on> Besides, although this list is definitely the right place to find people that would operate a personal SMTP relay in a colo just by the virtue that it's the geeky thing to do, what does it change in the big scheme of things? All these small business customers (20 persons) that I have that use a sub-$100 "business" DSL and M$ Small Business Server + Exchange are not going to go for it, because the cost then will suddenly become $50 plus the 1U server plus my time plus maintaining it. Michel.

3 2

possible new DoS?
by Christopher J. Wolff 13 Mar '04

13 Mar '04

Over the past week the following error started to appear in the router logs; Mar 9 19:44:16 fe-0-1-100.blah.net 16: Mar 10 02:44:15.477: %CRYPTO-4-IKMP_NO_SA: IKE message from 206.207.248.58 has no SA and is not an initialization offer. According to Cisco, 1. %CRYPTO-4-IKMP_NO_SA: IKE message from [IP_address] has no SA and is not an initialization offer IKE maintains the current state for a communication in the form of security associations. No security association exists for the specified packet, and it is not an initial offer from the peer to establish one. This situation could indicate a denial-of-service attack. Any suggestions are appreciated. The router that generated those log files dropped part of an IGP routing table. Since I've never seen this log entry before, I'm curious whether it's a 'new' DoS. Thank you. Regards, Christopher J. Wolff, VP CIO Broadband Laboratories, Inc. http://www.bblabs.com

1 0

Telia v ATT?
by Timothy Brown 13 Mar '04

13 Mar '04

Can anyone shed any light on the difficulties last night between Telia and {AT&T, ATDN, others}? For a while it seemed like it was AT&T specific. The problem started about 9:30pm Eastern and didn't get resolved until about 12:30am; during this time routes were being seen but traffic wasn't passing over interfaces between Teliasonera and AT&T at a minimum, and probably several other providers as well. Feel free to respond privately. Tim

1 0

RE: Load Balancing Multiple DS3s (outgoing) on a 7500
by Michel Py 12 Mar '04

12 Mar '04

> Patrick W.Gilmore wrote: > It's just a little difficult for BGP to "load balance" > outbound bits when the bulk of the Internet these days > is 2 AS hops away from each of four upstreams. Not > impossible, but it doesn't happen by default either. Indeed. If the following conditions are met: a) all four upstreams are transit providers (and therefore even a default to any would be fair game) b) the goal is to distribute more evenly (in terms of bandwidth) the egress traffic between the four upstreams (in other terms, the egress traffic tends to peg one of the upstreams (which are being paid to carry the traffic) for no clear reason) IMHO there is nothing wrong with some WAEG route-map to police egress traffic to a different upstream that the one the BGP process would have chose at the 5th tie-breaker, all 4 including the AS-PATH being equal. My $0.02 plus tax. Michel.

1 0

Port 80 Navigation Problem (was:FW: hey had eric sent you)
by Riley, Marty 12 Mar '04

12 Mar '04

My apologies to the list - I haven't slept in a day or two and completely forgot to make a semi-intelligent subject line... mjr -----Original Message----- From: Riley, Marty Sent: Friday, March 12, 2004 22:18 To: nanog(a)merit.edu Subject: FW: hey had eric sent you I'm running short on theories and options, so I thought I would see if any other ISPs have seen this problem on your network(s). If so, what was the cure? mjr -----Original Message----- The Unknown problem. Symptoms: At random times dialup, dedicated, & internal network users are unable to pass TCP traffic to off network sites. ICMP and UDP appears to be uneffected by the outage which lasts anywhere from 2 to 5 minutes. The problem appears to be wide spread with similar reports from WVNET and other ISPS. nTelos is experiencing a similar problem but we have yet to confirm it is the same. Problem has changed in it's action but remained similar enough to consider it the same problem. Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, & 2003 Server. Uneffected Platforms: Unix, MacOS (?) History: During the week of 2/9/04 the call center started recieving reports of users being unable to connect to sites off the CityNet network. Sites hosted on the internal network are uneffected by the outage. Initally it was thought to be a Internet Explorer problem possably caused by the KB832894 / IE SP1 or other updates but after further investigation it was found that Mozilla users were encountering the same problem. After several days of testing it was determined that during the outage any TCP session started on any port would fail. TCP sessions started before the outage continue to work and show no ill effects from the outage. After logging connection attempts at various intervals on many machines there appears to be no sort of pattern in the outages. Most machines encounter the problem, some more than others and a few do not encounter it at all. The duration and frequency of the outage is very fluid. During an outage, we can verify that the packet does seem to leave and reenter the network: Mar 5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) -> 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17084: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3376), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17085: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) -> 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17086: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 1 packet Mar 5 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 7 packets Mar 5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3376), 17 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17092: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) -> 216.41.224.3(80), 7 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17093: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) -> 216.41.224.3(80), 18 packets Mar 5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) -> 216.41.224.3(80), 1 packet Mar 5 00:58:30 pittpa-clarwv-ds3 16063: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3183), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16067: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) -> 216.41.224.3(80), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16068: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3228), 1 packet Mar 5 01:03:39 pittpa-clarwv-ds3 16072: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) -> 216.41.224.3(80), 1 packet Mar 5 01:03:47 pittpa-clarwv-ds3 16073: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) -> 216.41.224.3(80), 74 packets Mar 5 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3183), 72 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16078: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3218) -> 216.41.224.3(80), 4 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16079: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) -> 216.41.224.3(80), 3 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16080: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3221) -> 216.41.224.3(80), 19 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16081: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 5 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16082: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3229) -> 216.41.224.3(80), 6 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16083: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3236) -> 216.41.224.3(80), 9 packets Mar 5 01:08:47 pittpa-clarwv-ds3 16084: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3233) -> 216.41.224.3(80), 12 packets Mar 5 01:08:47 pittpa-clarwv-ds3 16085: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) -> 216.41.224.3(80), 21 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16087: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3239), 19 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16088: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3228), 4 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16089: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 2 packets Mar 5 01:09:12 pittpa-clarwv-ds3 16091: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3218), 3 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16092: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3221), 17 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16093: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3229), 5 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16094: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3236), 7 packets Mar 5 01:09:13 pittpa-clarwv-ds3 16096: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3233), 9 packets Network analysis showed significant amounts of spoofed multicast traffic and odd arp traffic. 10:17:16.416222 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 278 10:17:16.421886 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 334 10:17:16.423873 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 262 10:17:16.426948 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 254 10:17:16.432095 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 298 10:17:16.435921 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 274 10:17:16.439959 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 328 10:17:16.445317 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 326 10:17:16.449688 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 330 10:17:16.463537 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 322 Steps were taken to elminiate the spoofed traffic on the routers and access servers in the form of ACLs and filter lists. Neither have eliminiated the problem... but to what extent they might have helped has yet to be determined. The problem is still occuring, for some users the duration of the outage seems to have shortened other users notice no difference. It is not yet known if the filtering on the routers and access servers or the conversion to the 10.x.x.x network has made any difference. We should have a better idea in the upcoming days. What Doesn't help: Removing windows updates. Turning off XP firewall. Searching for malware. (SpyBot-SD, Adaware) Virus scanning (Various softwares) Specifying dns servers. Reinstalling windows.

1 0

RE: Enterprise Multihoming
by Burton, Chris 12 Mar '04

12 Mar '04

Address portability all depends on if you IP blocks are assigned by ARIN/RIPE/APNIC/ISP portable or if you are using the ISP's address space. It has been my experience that multi-homing to diverse ISP's with multiple circuits per ISP (i.e. Primary/Secondary with ISP-A and Primary/Secondary with ISP-B) is the best option if you can afford the cost and your bandwidth requires it. Like it was stated before, if you can afford the possible downtime associated with multi-homing to a single ISP then yes there are definitely cost savings to be had and reduced administrative overhead; but, if you cannot afford the possibility of downtime then separate ISP's is the only way to go. Chris Burton Network Engineer Walt Disney Internet Group: Network Services The information contained in this e-mail message is confidential, intended only for the use of the individual or entity named above. If the reader of this e-mail is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please contact Walt Disney Internet Group at 206-664-4000. -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Scott McGrath Sent: Friday, March 12, 2004 5:50 PM To: nanog(a)merit.edu Subject: Re: Enterprise Multihoming As Marshall noted multi-homing gives you the ability to switch providers easily. This ability also gives you leverage with your network providers since vendor lock-in does not exist. This is a strong business case for multihoming and is one the financial types understand and appreciate. In a prior incarnation I worked for a distributor who had a online ordering system. Our telcom coordinator got a "great" deal on bundled internet service and telephony from a unnamed vendor. Due to the peering arrangements the carrier had major customers were unable to place orders in a timely fashion. I set up a new AS and set up multihoming with another carrier and made our customers happy again. Subsequently said carrier had an outage which took down our link to them for 7 weeks. Since this was an internal problem at our provider multiple links to this carrier would not have benefited us in the least. A multihoming strategy also allows you to select providers who provide connectivty to your business partners and customers which is another win for obvious reasons. Scott C. McGrath On Thu, 11 Mar 2004, Marshall Eubanks wrote: > > There is another thing - if you are multi-homed, and want to switch > providers, it is pretty seamless and painless - no renumbering, no > loss of connection, etc., as you always have a redundant path. > > > On Thursday, March 11, 2004, at 12:34 PM, Pekka Savola wrote: > > > > > <On Thu, 11 Mar 2004, Gregory Taylor wrote: > >> Mutli-homing a non-ISP network or system on multiple carriers is a > >> good > >> way to maintain independent links to the internet by means of > >> different > >> peering, uplinks, over-all routing and reliability. My network on > >> NAIS > >> is currently multi-homed through AT&T. I use a single provider as > >> both > >> of my redundant links via 100% Fiber network. Even though this is > >> cheaper for me, all it takes is for AT&T to have some major outage > >> and I > >> will be screwed. If I have a backup fiber line from say, Global > >> Crossing, then it doesn't matter if AT&T takes a nose dive, I still > >> have > >> my redundancy there. > > > > Well, I think this, in many cases, boils down to being able to pick > > the right provider. > > > > I mean, some providers go belly-up from time to time. Others are > > designed/run better. > > > > For a major provider, complete outage of all of its customers is such > > a big thing they'll want to avoid it always. If it happens, for a > > brief moment, once in five years (for example), for most companies > > that's an acceptable level of risk. > > > > -- > > Pekka Savola "You each name yourselves king, yet the > > Netcore Oy kingdom bleeds." > > Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings > > > > > Regards > Marshall Eubanks > > T.M. Eubanks > e-mail : marshall.eubanks(a)telesuite.com > http://www.telesuite.com >

1 0

Enterprise Multihoming
by John Neiberger 12 Mar '04

12 Mar '04

On another list we've been having multihoming discussions again and I wanted to get some fresh opinions from you. For the past few years it has been fairly common for non-ISPs to multihome to different providers for additional redundancy in case a single provider has problems. I know this is frowned upon now, especially since it helped increase the number of autonomous systems and routing table prefixes beyond what was really necessary. It seems to me that a large number of companies that did this could just have well ordered multiple, geographically separate links to the same provider. What is the prevailing wisdom now? At what point do you feel that it is justified for a non-ISP to multihome to multiple providers? I ask because we have three links: two from Sprint and one from Global Crossing. I'm considering dropping the GC circuit and adding another geographically-diverse connection to Sprint, and then removing BGP from our routers. I see a few upsides to this, but are there any real downsides? Flame on. :-) Thanks, John --

16 18

wholesalebandwidth.com major sponsor of spammers refuses to accept email at abuse
by Henry Linneweh 12 Mar '04

12 Mar '04

I have received almost 200 different spam messages from domains hosted by this provider from russain domains attempting to sell pharmacueticals and other unsolicited services that I do not want tekmailer.com and moosq.com are 2 of the primary abusers from this hosting company -Henry Message from yahoo.com. Unable to deliver message to the following address(es). <abuse(a)wholesalebandwidth.com>: 69.6.21.60 does not like recipient. Remote host said: 550 5.7.1 <abuse(a)wholesalebandwidth.com>... Relaying denied Giving up on 69.6.21.60.

12 13