- NANOG - lists.nanog.org

[Q] BGP filtering policies
by Henry Yen 09 Apr '02

09 Apr '02

We were recently assigned a /22 from UUNet in conjunction with some transit we're buying from them. The space is inside their superblock, 65.242.0.0/14. We are concerned that our route announcement of this block would be filtered out by some other providers, as it's not class C/swamp space (or even class B space for that matter). Verio's current policy, for one, indicates that this would be so. This is of particular concern to us as our little network encompasses several physical partially-meshed locations, with a mix of varying bandwidths both upstream as well as intra-location. Traffic Engineering is what we think is a reasonable (business) approach to address our flexibility needs, and so we're trying to move to address space(s) that would be least likely to be BGP filtered. We've asked for a different block from UUNet but the request didn't meet with success; UUNet suggested that any problems encountered as a result of this allocation could probably solved by e-mailing any NSP whose traffic interchange with us might be negatively affected (unlikely, to be sure, but still...), and would then change their filter (I'm unconvinced of this scenario). I briefly browsed the NANOG archives, and didn't see this issue discussed recently. Have the BGP filtering policies for "most" ISP/NSP's been relaxed to the level of "accept /24's from class A (ARIN-allocated) space"? Am I mis-reading Verio's posted policy? Is there anyone from UUNet who might choose to comment? Is there something else I'm misunderstanding? -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York

2 1

RE: packet reordering at exchange points
by Kavi, Prabhu 09 Apr '02

09 Apr '02

An interesting historical observation, Many years ago when I used to create discrete event simulation network models for a living, I had one project which was to model (what was then) a widely implemented PC TCP stack. I remember that one wart of this implementation was that when packet reordering occurred it collapsed the window size to 1! Anyone know if strange warts like this still exist in desktop systems? Prabhu ---------------------------------------------------------------------- Prabhu Kavi Phone: 1-978-264-4900 x125 Director, Adv. Prod. Planning Fax: 1-978-264-0671 Tenor Networks Email: prabhu_kavi(a)tenornetworks.com 100 Nagog Park WWW: www.tenornetworks.com Acton, MA 01720 > -----Original Message----- > From: Iljitsch van Beijnum [mailto:iljitsch@muada.com] > Sent: Tuesday, April 09, 2002 12:36 PM > To: Stephen Sprunk > Cc: nanog(a)merit.edu > Subject: Re: packet reordering at exchange points > > > > On Mon, 8 Apr 2002, Stephen Sprunk wrote: > > > Thus spake "Iljitsch van Beijnum" <iljitsch(a)muada.com> > > > But how is packet reordering on two parallell gigabit interfaces > > > ever going to translate into reordered packets for individual > > > streams? > > > Think of a large FTP between two well-connected machines. > Such flows tend > > to generate periodic clumps of packets; split one of these > clumps across two > > pipes and the clump will arrive out of order at the other end. The > > resulting mess will create a clump of retransmissions, then > another bigger > > clump of new data, ... > > I don't think it will be this bad, even if hosts are > connected at GigE and > the trunk is 2 x GigE. In this case, a (delayed) ACK will usually > acknowledge 2 segments so it will trigger transmission of two new > segments. These will arrive back to back at the router/switch > doing the > load balancing. Since there is obviously need for more than 1 > Gbit worth > of bandwidth, it is likely the average queue size is at least > close to 1 > (= ~65% line use) or even higher. If this is the case, there > is a _chance_ > the second packet gains a full packet time over the first and arrives > first at the destination. However, this is NOT especially > likely if both > packets are the same size: the _average_ queue sizes will be > the same so > in half the cases the first packet gains an even bigger > advance over the > second, and only in a fraction of half the cases the second > packet gains > enough over the first to pass it. And then, the destination host still > only sees a single packet coming in out of order, which isn't > enough to > trigger fast retransmit. > > You need to load balance over more than two connections to trigger > unnecessary fast retransmit (over two lines, packet #3 isn't > going to pass > by packet #1), AND you need to send more than two packets > back to back. > Also, you need to be at the same speed as the load balanced lines, > otherwise your packet train gets split up by traffic from > other interfaces > or idle time on the line. > > And _then_, if all of this happens, all the retransmitted > data is left of > window. I'm not even sure if those packets generate an ACK, > and if they > do, if the sender takes any action on this ACK. If this > triggers another > round of fast retransmit, the FR implementation should be considered > broken, IMO. > > > > Packets for streams that are subject to header compression or > > > for voice over IP or even Mbone are nearly always transmitted > > > at relatively large intervals, so they can't travel down parallell > > > paths simultaneously. > > > RTP reordering isn't a problem in my experience, probably > since RTP has an > > inherent resequencing mechanism. > > My point is real time protocols will not see reordering > unless they are > using up nearly the full line speed or there is congestion, > because these > protocols don't send out packets back to back like TCP > sometimes does. How > big are VoIP packets? Even with an 80 byte payload you get > 100 packets per > second = 10 ms between packets, which is more than 80 packet times for > GigE = congestion. And if there is congestion, all > performance bets are > off. > > It seems to me spending (CPU) time and money to do more complex load > balancing than per packet round robing in order to avoid > reordering only > helps some people with GigE connected hosts some of the time. > Using this > time or money to overcome congestion is probably a better investment. > > PS. For everyone looking at their netstat -p tcp output: > packet loss also > counts towards the out of order packets, it is hard to > get the real > out of order figures. > > PS2. Isn't it annoying to have to think about layer 4 to > build layer 2 stuff? > >

1 0

RE: Qwest Transit
by LeBlanc, Jason 09 Apr '02

09 Apr '02

Being a coworker of Dre's, I can attest that his crack pipe is one of the finest. =P We love dre. ;) -----Original Message----- From: Gironda, Andre [mailto:agironda@ebay.com] Sent: Monday, April 08, 2002 9:35 PM To: 'Richard A Steenbergen '; Gironda, Andre Cc: ''Daniel Golding' '; ''Alex Rubenstein' '; ''Andy Dills' '; 'nanog(a)merit.edu ' Subject: RE: Qwest Transit I meant any sales guy selling transit would "like to" ask for strict traffic ratios, while in reality, they don't actually do this. Your email is right on otherwise. I do believe that many transit offerings in the past and currently require some kind of strict traffic ratio to *some* companies. If you still don't think so, drop me an email offline and we can chat more about it. -dre > Thats a mighty fine crack pipe you're smoking from. > > The majority of 95th percentile providers that I am aware of will > charge you only for whatever is higher, inbound OR outbound (the notable > exception to this being Exodus, who added in+out and THEN took 95th > percentile, to extract every last penny from your pocket). > > Infact depending on the provider you choose, you might even be able > to strike some better deals based on your ratios. For example, rumor > has it that Google struck a great deal with AboveNet because all their > inbound traffic (from spidering) helped balance out AboveNet's > peering links (I don't know if that story is accurate or not, but > it has a ring of truth to it). > > To my knowledge Cogent is the only provider who asks for traffic > ratios on their transit connections. The reason? Probably because > Cogent is already taking a massive massive loss on anything they > must transit. Their only chance to make money at the end of the day > is to get as much peering as quickly as possible (hence their buying > spree of "hosed" companys who just happened to have lots of legacy > peering), and since they are answerable to their peers for their > ratios they must pass on those requirements to their customers. > > It's interesting to note how much inbound traffic is "in demand" > by hosting providers. With the breakup of @Home into many regional > cable companies, most of whom havn't the slightest bit of clue how > to build a network let alone a backbone, the traffic profiles change > greatly. My prediction is that a lot of traffic which used to be > peered into @Home at "major exchange points" will turn into transit > connections from other providers. Unfortunately for the cable > companies, the people who they could get the best deals from (the > "mostly hosters") tend to be highly based around the "major exchange > points" cities (to most efficiently pump traffic into the rest of > the internet), not the "rest of the world". > > -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras > PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 > 36 FE B6)

1 0

High speed packet capture
by Mike Duckett 09 Apr '02

09 Apr '02

Looking for clues where I can analyze OC3 and DS3 links with ATM encap carrying broadband/DSL traffic. Stack is IP/PPP/LT2P/UDP/IP/AAL5. Want to capture L2TP control messages *only* on a DS3 and OC3 link. Any suggestions?

1 0

Qwest Support
by Andy Dills 09 Apr '02

09 Apr '02

Wow, Qwest support is indeed terrible. Turned up the DS3 today...the connectivity seems fine. I decided to check a couple of routeservers (nitrous); all had my much-prepended UUnet announcement, but NONE had my Qwest announcement. Not a huge deal, but curious to me. Is Qwest just not at the public peering points? When I checked route-views.oregan-ix.net, I felt better, but yet annoyed. Even with the prepends, most networks were announcing UUnet's path. So I decided to call them and ask...man what a mistake. The guy is like, "Ok, hold on, let me get somebody from our IP noc." 10 minutes goes by, and he comes back with "Couldn't get anybody in the IP noc, let me try to get somebody in your install group" (being that I turned up the DS3 today). Comes back another 10 minutes later with "Well, I left a message for them, but there isn't much I can do. Nobody seems to be answering their phone. If somebody doesn't call you back within 30 minutes, here's a number to call..." So what if my routes were actually hosed? I'd just be screwed because they can't get anybody at the IP noc? I wait. Nobody calls back within 30 minutes. I call the number he gave me. Busy. You gotta be kidding me. So I call the main number again, talk to somebody different. She has me hold, and then brings some guy on the line "who can help me". I start to talk about route servers, and he's immediately like "Woah, this is a BGP problem...I can't help you. Let me try to get somebody from the IP noc." So, I wait on hold for about 15 minutes, only to be given dial tone. Please tell me it isn't always THIS bad? Andy xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Andy Dills 301-682-9972 Xecunet, LLC www.xecu.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Dialup * Webhosting * E-Commerce * High-Speed Access

7 12

RE: Qwest Transit
by Gironda, Andre 09 Apr '02

09 Apr '02

I meant any sales guy selling transit would "like to" ask for strict traffic ratios, while in reality, they don't actually do this. Your email is right on otherwise. I do believe that many transit offerings in the past and currently require some kind of strict traffic ratio to *some* companies. If you still don't think so, drop me an email offline and we can chat more about it. -dre > Thats a mighty fine crack pipe you're smoking from. > > The majority of 95th percentile providers that I am aware of will > charge you only for whatever is higher, inbound OR outbound (the notable > exception to this being Exodus, who added in+out and THEN took 95th > percentile, to extract every last penny from your pocket). > > Infact depending on the provider you choose, you might even be able > to strike some better deals based on your ratios. For example, rumor > has it that Google struck a great deal with AboveNet because all their > inbound traffic (from spidering) helped balance out AboveNet's > peering links (I don't know if that story is accurate or not, but > it has a ring of truth to it). > > To my knowledge Cogent is the only provider who asks for traffic > ratios on their transit connections. The reason? Probably because > Cogent is already taking a massive massive loss on anything they > must transit. Their only chance to make money at the end of the day > is to get as much peering as quickly as possible (hence their buying > spree of "hosed" companys who just happened to have lots of legacy > peering), and since they are answerable to their peers for their > ratios they must pass on those requirements to their customers. > > It's interesting to note how much inbound traffic is "in demand" > by hosting providers. With the breakup of @Home into many regional > cable companies, most of whom havn't the slightest bit of clue how > to build a network let alone a backbone, the traffic profiles change > greatly. My prediction is that a lot of traffic which used to be > peered into @Home at "major exchange points" will turn into transit > connections from other providers. Unfortunately for the cable > companies, the people who they could get the best deals from (the > "mostly hosters") tend to be highly based around the "major exchange > points" cities (to most efficiently pump traffic into the rest of > the internet), not the "rest of the world". > > -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras > PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 > 36 FE B6)

2 1

RE: Qwest Transit
by Gironda, Andre 09 Apr '02

09 Apr '02

All ISP's selling transit ask for strict traffic ratios. How often do you think they get what they ask for? I would guess not very often. People like flat rate 95th% with no minimal commitment (both the seller and buyer) because that's easy to keep track of. Simplicity is king, again. Cogent's deals were to make things easy, right? I don't know what they charge, but anyone can see that an offer like 100Mbps for $10,000 a month makes sense in terms of simplicity (not saying it makes sense in terms of a transit provider making any money, tho) ;> -dre > -----Original Message----- > From: Daniel Golding [mailto:dgolding@sockeye.com] > Sent: Monday, April 08, 2002 6:52 PM > To: 'Alex Rubenstein'; 'Gironda, Andre' > Cc: 'Andy Dills'; nanog(a)merit.edu > Subject: RE: Qwest Transit > > > Hmm. Cogent does require some semi-strict traffic ratios to get the > really good deals. If it's not violating an NDA, is Qwest asking for > similar ones, these days? > > - Dan

3 2

RE: Qwest Transit
by Gironda, Andre 09 Apr '02

09 Apr '02

We probably get Qwest transit cheaper than you, and yes, it is the cheapest in the industry. Not too bad considering Qwest is definitely in the top ten largest Tier 1 providers, and probably even the 3rd or 4th largest transit-AS networks according to a recent CAIDA study: http://www.caida.org/analysis/topology/rank_as/ We connect to them at one of their Cybercenters, and I can only speak for the transit relationship we have. Connected to them 10/23, BGP sessions have been up for longer than 4 weeks. I remember calling them once when a BGP session went down over two weeks ago, and they said they didn't have 24/7 network support at first, then forwarded me on to somebody on the east coast who said they "normally are 9-5 EST". That was the only thing that worried me. We connect to them via 2xGbE to dual MSFC2 Catalyst 65xx/SUP2's. One of the MSFC's had reset and it did take them a long time to determine this. It's good that they did determine the RFO and, in fact, were very quick to respond (I wish they were proactive about it, however). http://www.qwest.com/about/qwest/QwestCyberCenters/index.html However, they are only our 20+ largest AS for destinations in terms of traffic. They are in our top 30, which means they are still probably worth the money that we are paying them. And it's over 107,000 routes, so that's a pretty big table for transit-AS destinations. I think their peering is sufficient, but it's hard to tell with our other directly connected transits and peers. We use them for a large portion of our outbound traffic right now, and I haven't seen any performance impact. Take at look at their network with some 'show ip bgp' action: http://www.routeviews.org/ http://stat.qwest.net/looking_glass.html It sounds like you have confidence in the circuit, so, in your shoes, I would move a lot of traffic to Qwest (especially if cost savings is what you are looking for). As a long term strategy, you might want to look at the cost of Qwest transit vs. the cost of moving a lot of your traffic to peering. Having an SLA might be nice for management, but having those peering relationships can really help general network operation and performance, as well as the bottom line in terms of dollars. Here is some data on the Qwest SLA and network: http://www.networkcomputing.com/1114/1114f15.html I think Qwest does try to do some good 'best-exit', but they clearly don't have the experience that other providers do. You can local_pref and path- prepend all you want, but make sure you have a good eye on all your traffic and routing table changes. If anybody has more information on how Qwest does 'best-exit' and handles MED's or anything else like their acceptance of RFC1998++ ISP-style BGP communities or route filters or any other operational experience, please post. -dre > -----Original Message----- > From: Andy Dills [mailto:andy@xecu.net] > Sent: Tuesday, April 02, 2002 1:46 PM > To: nanog(a)merit.edu > Subject: Qwest Transit > > Ok, we have an opportunity to get Qwest bandwidth amazingly cheap. I > can't disclose the details (NDA), but bottom line is it looks like the > cheapest in the industry and because there is no loop > involved, it could > be one hell of a deal on a ds3. > > I'm not a fan of Qwest, traditionally. I've had some (somewhat famous) > run-ins with their sales people, and I personally know plenty > of companies > who over several years have had tons of complaints with > various aspects of > their organization. > > However, I'm not interested in all of that at the moment. I need to > evaulate, without all of my longheld bias, whether or not Qwest can > provide quality DS3 transit. The traditional knock on Qwest is that > they're an over-built under-peered network with poor support. > But a lot of > time has passed, so surely they would have addressed the > peering issues by > now? I can deal with poor support...the circuit is a simple > cross connect, > and our bgp filter changes are fairly uncommon. > > Can anybody speak positively or negatively about the quality of Qwest > transit? We ordered a metered ds3 for a backup circuit, but with > the pricing I'm seeing, I'm really tempted to localpref and > prepend our > announcements such that most of the traffic flows through > Qwest. I plan on > trying it out regardless, but I'd like to hear from the list first. > > Thanks, > Andy

4 3

Re: packet reordering at exchange points
by Mark Allman 09 Apr '02

09 Apr '02

Paul- > more to the point, does anybody still care about packet reordering > at exchange points? we (paix) go through significant effort to > prevent it, and interswitch trunking with round robin would be a > lot easier. are we chasing an urban legend here, or would > reordering still cause pain? Yep. Reordering causes pain to TCP performance. The basic idea is that if packets get jumbled up they trigger duplicate ACKs from the receiver. If things are badly enough reordered we end up with >= 3 duplicate ACKs arriving at the sender. According to the fast retransmit algorithm 3 duplicate ACKs are taken by the TCP as an indication of packet loss -- and, hence, congestion. So, we end up needlessly cutting our congestion window in half. Ethan Blanton and I just published a paper on what might be done to make TCP more robust to paths that reorder segments (which would, in turn, make such paths be less problematic). The paper is: Ethan Blanton, Mark Allman. On Making TCP More Robust to Packet Reordering}. ACM Computer Communication Review, 32(1), January 2002. http://roland.grc.nasa.gov/~mallman/papers/tcp-reorder-ccr.ps You might not necessarily be interested in the entire paper. But, the first part shows the performance problems caused by packet reordering. allman -- Mark Allman -- NASA GRC/BBN -- http://roland.grc.nasa.gov/~mallman/

1 0

Ph.D. student looking for data on network failure causes
by David L. Oppenheimer 08 Apr '02

08 Apr '02

Hello network operators! I'm a Ph.D. student at UC Berkeley working for Dave Patterson on the ROC project, which is investgiating techniques for improving the availability and manageability of large-scale Internet services and systems. I'm currently conducting a study of the root causes (hardware, softare, human, etc.) and durations of failures in such systems. To do this, I have been examining the operations trouble ticket databases from several large-scale Internet services (of the Hotmail, eBay, Yahoo!, etc. type). In doing this research, it has become apparent that for many services (especially geographically distributed ones, e.g. those that use multiple colocation facilities), a major cause of problems is failures of various types in the Internet. Thus I've become interested in finding out the types and root causes of problems in wide-area networks, e.g. within the kinds of large-scale ASes that are administered by the folks on this list. I'm not sure how your services track failures and problems; the problem tracking databases at the services I've examined have been a great source of data about problem scale, symptoms, root causes, durations, steps (and missteps!) taken in diagnosing and fixing problems, etc. I'm writing to the list because I'm very interested in working with network operators to study the causes of failures in large networks. I realize this type of data is very sensitive to your organizations. I would be happy to talk offline with anyone who is interested in the possibility of sharing data, about how I've overcome the multitude of objections that have been raised by folks I have solicited for data (protecting their customers' privacy, securing datasets when they are not examined on the premises of the services, anonymizing and aggregating data in reporting, etc. etc.). I'm interested in the relative causes of failures, *not* overall availability numbers. As a result of the precautions we've taken, several household-name Internet services have allowed me to examine and report on the problems their servcies have experienced. If you're interested in discussing the possibility of sharing access to this kind of data about your service, please contact me. I'm willing to examine data on the premises of your service, to anonymize it fully, to submit any results I want to publish to your organization prior to publication, to sign any necessary NDAs, etc. In return, I'm happy to share with you any insights I have about the problems your service experiences, and you'll contribute to the world's knowledge of why bad things happen to good networks. :-) If you're not the right person in your organization to contact with this request, but you think your organization might be interested in participating in this study, perhaps you could forward this email to the appropriate person or let me know who the right person to contact in your organization would be. Many thanks! David

1 0