September 1997 - NANOG - lists.nanog.org

Re: Traffic Engineering [was Chanukah [was Re: Hezbollah]]
by pkavi＠pcmail.casc.com 21 Sep '97

21 Sep '97

Kent, As a former network design guy who's done traffic engineering and design (and redesign) on many networks (Internet and otherwise), I disagree that traffic engineering doesn't work for the Internet. I've seen many people go with the "throw bandwidth at the problem" as a cure all. While it tends to work, it tends to be the most expensive method of solving the problem. Doing traffic engineering right is hard. The telcos have it down pat for their voice networks, and telco-based ISPs often have applied this design expertise to their ISP network. Having a person do traffic engineering can save the ISP big bucks. The traffic engineering techniques I'm talking can't handle wildly dynamic situations. For example, a news event like Princess Di's death greatly increases traffic to/from England which plays temporary havoc with forecasted traffic projections. However, outside of these anomalies, traffic projections work pretty well. I've outlined a basic technique below which works for many types of networks, and has some ISP specific steps. The key to this analysis is that it takes into account the underlying traffic flows and then determines the appropriate physical backbone topology, or the changes to be made to an existing topology. This is directly in contrast to the "throw bandwidth at the problem" case that patches a backbone topology which might be sub-optimal in the first place. Here's the overall outline: 1. Divide your network into a small number of geographic areas (between five and ten). Each geographic area you choose probably has a large city that serves as a major traffic source for that area. These cities are usually the natural cities for backbone connectivity. Create an NxN matrix, where N is the total number of areas in your network. Each cell in the matrix will represent the total traffic demand between each source/destination geographic area. There are several factors which effect this matrix, each of which will be discussed below. 1. The locality of traffic. 2. The typical utilization of customers. 3. The entry/exit points of traffic from your network. 2. Identify which % of traffic, if any, has regional locality. For pure Internet traffic, the probability that the source and destinatino of traffic are within the same metropolitan area tends to be low (10% or lower for metros within the US). However, there are exceptions. Telecommuting applications tend to have very high locality. People close to work dial into work through an ISP, so both the source and destination of traffic tends to stay local (70% or higher). Places like the Bay Area in California also tend to have higher traffic locality. This is because the Bay Area has lots of Internet users (which tend to be traffic sinks), and lots of web sites (which tend to be traffic sources). ISPs outside of the US tend to have a higher percentage of traffic staying within the country, especially non-english speaking countries. 3. Measure/estimate the typical utilization of customers. Utilization needs to be measured/estimated in both send and receive directions. Dial-up users typically receive almost seven times as much as they send. Corporate customers not doing telecommuting applications tend to receive about four times as much as they send (less because corporations have web sites that others access). Web server farms have the opposite characteristics of dial-up users. Percentage utilization tends to increase with bandwidth. In the U.S., a T1 customer connection typically has a peak recieve utilization of 20% or less. However, a DS3 customer can easily have a receive utilization of 50% or more. Simple explanation is that someone paying big bucks for a DS3 wants to make sure it is justified. So, take the total number of users in each area, the connection speeds and customer types, multiply by the appropriate factors, and you get the total demand you are trying to serve out of each area. Take this traffic demand, and multiply it by the non-local traffic. This represents the total traffic that you need to get either in/out of the network, or in/out of this particular part of your network. 4. Determine the entry/exit points for traffic with your network, and its effect upon the traffic matrix. How do you setup your routing policies? Many ISPs use nearest exit. If the nearest exit is in the same geographic area, the traffic sent by your customers does not affect any other part of the overall traffic matrix. If the nearest exit is not within the same geographic area, determine the area where this traffic will be sent. Enter this value in the appropriate source/destination box of the traffic matrix. It gets harder when peering with many other ISPs, some of whom you connect to in the same area, and others in remote areas. In this case, determine which percentage of the traffic goes into each particular region, and The main traffic sources into your network (excluding your customers) are your peering points (both public and private). The amount of traffic from each peering point is measurable. You can generally estimate that this traffic is to be distributed proportional to the overall traffic demand in each geographic area. This is a significant amount of matrix math, but the overall concept is simple. Determine the overall flow between one part of your network to another. 5. With me so far? Good, now it's time to design your backbone to handle your demands. You can use dedicated lines or layer two services such as Frame Relay or ATM. The simplicity of using Frame Relay or ATM is that the circuits you need between each geographic area has been defined by your traffic matrix. This is part of the appeal of using public L2 services for a backbone. Designing your own backbone is a bit harder. The actual topology tends to be straightforward--you need to connect up the major cities in each of the geographic areas. For five areas, a simple ring suffices. For up to 10 areas, this tends to be rings bisected once or twice. The real work in designing your own backbone is in satisfying the traffic demands going across your network. Remember that geographic areas in the center of your network have to carry the traffic demands going across your network. This imposes a heavier burden in the center of the network than the traffic matrix would indicate. You also have to worry about resiliency, having sufficient bandwidth when the backhoes go fiber hunting, etc. 6. Design the network within each geographic areas. The steps for designing the network within each geographic area tend to be similar to that of designing the overall network. Breaking the overall design process into a regional network and backbone network makes the problem more tractable. 7. Measure data from a real network. This is really important. You've made lots of assumptions. Regularly check the overall traffic to see if it matches the assumptions. Refine the traffic matrix to see if it still represents reality. Create trendlines which show the overall traffic changes to/from each area, and project these trendlines into the future. You will tend to have pretty good certainty about 4 months into the future, with the value of the information decaying after that. Use this data to determine where to add additional peering points. Estimate what impact this would have on the traffic matrix. 8. Factor the measured and projected data into the next network backbone design. This next backbone design gives you the optimum backbone given the underlying flows in your network. See what changes you need to make to your backbone to get to this new optimum backbone, and order the circuits. Phew! Like I said earlier, it is hard to do right, and I've left out quite a few details in the above outline. But having been there, done that, (quite a few times) I can say it really works. And it saves ISPs money! Question for NANOG members. How important is traffic engineering given that it is fairly hard to do properly and you folks have enough other things to think about? Prabhu Kavi IP Business Marketing Manager Ascend Communications prabhu.kavi(a)ascend.com ______________________________ Reply Separator _________________________________ Subject: Chanukah [was Re: Hezbollah] Author: "Kent W. England" <kwe(a)geo.net> at smtplink Date: 9/16/97 2:09 PM At 05:03 PM 14-09-97 -0400, Dorian R. Kim wrote: >... One of the >things that needs to be engineered into building and maintaining >national/international backbones is traffic accounting to an arbitrary >granularity that paves the way for better traffic engineering and >bandwidth projections. There are already ample tools to to per-prefix >matrix of traffic right now. Tying this in with good sales projections >will alleviate much of the last minute fire fighting. > >This will most likely never be 100% accurate and precise, but there is >no reason why we can't get a better handle on bandwidth forecasts. (say to >95% percentile) Dorian; I don't want to throw cold water on the value of planning and foresight, but in terms of predicting traffic patterns it has never worked on the Internet. It sounds good and that was the argument that all the mainframe networkers made to us early Internet networkers -- Why can't you tell me upfront what your bandwidth requirements are going to be? Don't you know exactly how many terminals you have and where they are and what application keystrokes are going to be pressed at any given time? How else can you guarantee response time in your network? This Internet stuff is stupid. It'll never work. Somehow with the way that HTTP/HTML caught fire and Internet-CB (aka VocalTec and CUSeeMe) took off, I would be loath to think I could project my backbone needs with any reliability based on *historical* projections. > >Furthermore, with the deployment of WDM and Internet core devices moving >closer to the transmission gear, if you have access to fiber, getting more >bandwidth may become as straightforward as using an additional wavelength >on the ADM that your router's plugged into. > >-dorian > > > This I like a lot better as a design technique. Throwing more bandwidth at the problem almost always works (unless the transport protocol is broken). Like Peter Kline said, Turn up the speed dial upon onset of congestion. Simple. Effective. Then again, creating a data architecture for the web (a problem that has been recognized, but not addressed in the last five years) would eliminate much of the backbone bandwidth demand. What would happen if -- presto -- a data architecture for the web showed up one day? A lot of backbone bandwidth would become surplus and a lot more edge bandwidth would be needed ASAP. What does that do to historical projections? --Kent

19 29

BGP4 on a /20
by Phil Howard 20 Sep '97

20 Sep '97

I'm trying to understand what all the implications of running BGP4 on a network with a prefix longer than 19 bits. Here are some of the points I am thinking about. <flameshields> If I go ahead and announce a /20 via two backbones, one of which is the provider of the address space, then there will be redundant routes for this space as the backbone provider will be announcing the /19 (or shorter) block themselves. If I do this, it adds to the routing table glut, among other things. The advantage gained is questionable. If my link to the provider that the space comes from goes down, they are still announcing and I'll only be able to reach where my path via the alternate provider is shorter than the path to the down provider itself. OTOH If the provider were to be convinced to stop announcing for my /20, then I'm going to get filtered at Sprint and AGIS and whoever else is doing this and there won't be any /19 announcement that I can use a default path on. But the real catch here is that for the provider to stop announcing my /20 they have to split their /19 into two /20's. And if that was really a /18 that means they will be announcing a /19 and a /20 where before only a /18. This gets worse the larger their block was. Even worse than that, by doing this, they now have a /20 (the other half of the /19 my /20 is in) with other customers who will now also be filtered out at Sprint and AGIS and whoever else. While it can be OK to me if I want to give up that reachability, this is also imposing this on the other customer(s) in the other /20. So that provider is not even likely to do that. So, should I add to the glut of routes or should I add to the glut of routes? This needs to be simpler. </flameshields> -- Phil Howard +-------------------------------------------------------------+ KA9WGN | House committee changes freedom bill to privacy invasion !! | phil at | more info: http://www.news.com/News/Item/0,4,14180,00.html | milepost.com +-------------------------------------------------------------+

3 4

Re: BGP4 on a /20
by Security Administrator 20 Sep '97

20 Sep '97

If all the customers on a multi-billion dollar network could not reach my network I would think about paying a different upstream provider to peer with me. I announce a /19 and a /22 to NetRail and UUnet, and they both do an excellent job of getting Sprint's customers routed to my network. Marcus R. Williams, Jr. security(a)tomco.net ISP Programmer / Engineer On Fri, 19 Sep 1997, Brian Horvitz wrote: > Date: Fri, 19 Sep 1997 10:54:05 -0400 > From: Brian Horvitz <horvitz(a)nsa.shore.net> > To: Security Administrator <security(a)tomco.net> > Subject: Re: BGP4 on a /20 > > Because if all the customers on a multi-billion dollar network can't get to > him it's not real good. > > Brian > > At 10:32 AM 9/19/97 -0400, you wrote: > >Why should you concern yourself with the problems of a multi-billion > >dollar company like Sprint? > > > >Marcus R. Williams, Jr. > >security(a)tomco.net > >ISP Programmer / Engineer > > > >On Thu, 18 Sep 1997, Phil Howard wrote: > > > >> Date: Thu, 18 Sep 1997 18:22:19 -0500 > >> From: Phil Howard <phil(a)charon.milepost.com> > >> To: nanog(a)merit.edu > >> Subject: BGP4 on a /20 > >> > >> I'm trying to understand what all the implications of running BGP4 on > >> a network with a prefix longer than 19 bits. Here are some of the points > >> I am thinking about. > >> > >> <flameshields> > >> > >> If I go ahead and announce a /20 via two backbones, one of which is > >> the provider of the address space, then there will be redundant routes > >> for this space as the backbone provider will be announcing the /19 > >> (or shorter) block themselves. > >> > >> If I do this, it adds to the routing table glut, among other things. > >> The advantage gained is questionable. If my link to the provider that > >> the space comes from goes down, they are still announcing and I'll only > >> be able to reach where my path via the alternate provider is shorter > >> than the path to the down provider itself. > >> > >> OTOH > >> > >> If the provider were to be convinced to stop announcing for my /20, > >> then I'm going to get filtered at Sprint and AGIS and whoever else > >> is doing this and there won't be any /19 announcement that I can use > >> a default path on. > >> > >> But the real catch here is that for the provider to stop announcing > >> my /20 they have to split their /19 into two /20's. And if that was > >> really a /18 that means they will be announcing a /19 and a /20 where > >> before only a /18. This gets worse the larger their block was. > >> > >> Even worse than that, by doing this, they now have a /20 (the other > >> half of the /19 my /20 is in) with other customers who will now also > >> be filtered out at Sprint and AGIS and whoever else. While it can be > >> OK to me if I want to give up that reachability, this is also imposing > >> this on the other customer(s) in the other /20. So that provider is > >> not even likely to do that. > >> > >> So, should I add to the glut of routes or should I add to the glut of > >> routes? > >> > >> This needs to be simpler. > >> > >> </flameshields> > >> > >> -- > >> Phil Howard > +-------------------------------------------------------------+ > >> KA9WGN | House committee changes freedom bill to privacy invasion > !! | > >> phil at | more info: > http://www.news.com/News/Item/0,4,14180,00.html | > >> milepost.com > +-------------------------------------------------------------+ > >> > > > > >

2 1

The Cidr Report
by Tony Bates 19 Sep '97

19 Sep '97

1 0

Routing without source information and Traffic self-similarity
by Nathan Boyd 19 Sep '97

19 Sep '97

Hello all, I would appreciate any help with the following. First, what problems arise in a network that routes without information about the sender? In other words, imagine that the IP header did not contain the source address - what problems would this raise? Are optimal routing, security, billing, traceability, network management, etc., suddenly intractable? Second, to what degree do you believe Internet traffic exhibits self-similarity, and how does this change with varying levels of traffic aggregation? I have read papers from Bellcore and others suggesting that no matter the level of aggregation (be it a single Internet user or a major NAP), Internet traffic exhibits high degrees of self-similarity - but more recent research does not seem to agree with this. I am sorry if this is an inappropriate forum for the topics I have raised, however after reading through this list for some time it is apparent that we have assembled here a good deal of real-world experience, something I believe is requisite for a meaningful discussion of real-world network issues. Very interested to hear what you have to say, Nate Boyd MIT Laboratory for Computer Science

3 2

Test....
by Huohom 19 Sep '97

19 Sep '97

Test....

1 0

Re: LSR and packet filters
by Vadim Antonov 19 Sep '97

19 Sep '97

Sean M. Doran <smd(a)clock.org> wrote: >How does a security policy of "turn off LSR handling" translate >into anything other than an admission of completely >missing the point that one should never ever ever ever >trust any data based soley on where the network leads one >to believe it has come from? Hey, LSR is useful for all kinds of very interesting denial of service attacks. A clever combination of LSR and forged source addresses may make the attack virtually untraceable. >Turning off a useful but under-implemented network service... Useful for what? traceroute -g is the _only_ useful application for LSR. Disabling LSR and adding an application level service for tracing back would be just as useful. >Quoting Radia Perlman: > "The goal is to design a network that will guarantee that > a packet transmitted between two nonfaulty end systems A > and B will have a high probability of being delivered, > provided that at least one path consists of nonfaulty > components connects the two end systems. [...] The > network layer makes no attempt to keep conversations > private. If privacy is necessary, encryption must be > done at a higher layer. Also, the network layer need not > certify data that it delivers. For instance, it is > possible for some malicious node C to generate data, get > it delivered to B, and claim that the data was from A. > It is up to the higher layer in B to differentiate > between corrupted or counterfeit data and real data, > using known cryptographic techniques". Encryption is an overkill for 99% of all applications. Disabling LSR and doing SA filtering can take care of _most_ security problems. And it is computationally cheap. >The proposal to turn off LSRR is a tremendously incomplete >proposal to have the network certify the validity of the >origin, at the cost of some additional utility and >robustness from the point of view of end systems. I would wholeheartedly agree with that proposal, if some useful substitute for traceroute -g is provided. This will not make the network absolutely secure (there ain't no such thing as absolute security), but it definitely will make it _more_ secure. >In any event I don't think that disabling LSRR in any way >makes networking equipment more robust, and that is what >you appear to be arguing. To make equipment more robust you may want to use gold-plated connectors, have redundant power, etc. I love playing dumb :) >> I will note that its none of someone else's business what >> one's internal topology looks like. >Sure it is. Or rather, it is useful to be able to infer a >number of path characteristics between two communicating >endpoints for such things as flow control and route >selection. That should be a matter of preference. Knowing network topology of a large corporation can give tremendous amounts of information about the corporation's structure and give clues to what projects the company's working on. Connection analysis is an old and venerable intelligence tool. >Networks and computers exist to offer services to users, >and obscuring information that makes service use or >selection difficult is a poor policy. Hey. What information to disclose should be a voluntary choice. >How does LSR reduce network availability except in the >case of implementations that slow-switch option-ridden IP >datagrams and have no equivalent of SPD, or intermediate >systems (gee, it's OSIspeek day, isn't it?) that don't >use reasonable means to authenticate and keep private >things like routing information exchanges or management sessions{ How'd you like to get a stream of nasty bogons aimed at your router(s) and arriving from virtually all directions? There's a number of ways to kill ciscos with pretty low-rate streams. --vadim

3 3

Re: Traffic Engineering (fwd)
by shields＠crosslink.net 19 Sep '97

19 Sep '97

In article <ytafhal5xz.fsf(a)cesium.clock.org>, "Sean M. Doran" <smd(a)clock.org> wrote: > Why? ping and traceroute are poor predictors of locality > and available bandwidth between the originator and target > of those tools. You can get an interesting set of data by sending a few rounds of pings of different sizes. Make a scatterplot of ping packet size vs. delay and find the line that fits the data; its slope represents the inverse of available bandwidth and the y-intercept represents the raw latency. Standard deviation gives you an idea of congestion. Is it crude? Yes, but I don't know of anything better. Is it ugly? Yes, but no uglier than traceroute. This isn't my idea; I saw it in "bing". Once you have a metric coded up, you can have each potential data source measure the quality of their connectivity to the endpoint. After they have agreed amongst themselves (probably via an intermediary server) which server is best they can cause the session to be moved to the most probably optimal server. -- Shields, CrossLink.

1 0

Re: Traffic Engineering (fwd)
by Sean M. Doran 19 Sep '97

19 Sep '97

Avi Freedman <freedman(a)netaxs.com> writes: > The cyclic stuff in News recently has hammered home that the search > for elegance pays off. I "initially" solved the problem of having > multiple machines with the same IP by postulating IP-stack hackery > involving forwarding any packets that come in for sockets that > don't exist to a central database-forwarder machine that keeps track > of all open tcp sessions on all replicated machines, but it turns > out that there are some much easier non-stack-hackery-mandatory > solutions. Your approach also is not a neat solution for two further reasons. Firstly, it requires maintaining connection state for all connections on a central repository, and secondly, it requires that your network be stable at the point when a connection needs to migrate. You should be able to deal with the case that during the lifetime of a TCP session something causes traffic that was hitting site A suddenly arrive at similarly-numbered site B at a time when connectivity within your network is unstable. (This likely can be done with tunnels, following the NAT approach, if synchronization is between things located at the edges of your network, rather than in a star pattern from something inside your network). You should also avoid scaling limits inherent in dumping lots of traffic at some central host rather than doing content-driven fail-over checkpointing for bulk data transfers from "static" content. Sean. P.S.: do you go about guaranteeing that you have unique connection "names" across all your simliarly-numbered hosts?

3 2

Re: Traffic Engineering (fwd)
by Sean M. Doran 19 Sep '97

19 Sep '97

Avi Freedman <freedman(a)netaxs.com> writes: > We at Net Access have figured out a way (we believe) to get > around the stability-of-routing issue for already-established > TCP sessions in the above approach (multiple machines with the > same IP externally, plus an internally different IP, each > running gated to announce their /32(s) to your IGP) - > hint: a question I asked on NANOG a few days back - > And Alec Peterson (now of Erols) has figured out an even > arguably slicker way to do it. > > I'll see if Merit wants to have Alec and I do a presentation > on the methods @ NANOG. If you have neatly solved the problem of undetectable route flutter, whereby something distant from your similarly-numbered machines causes datagrams destined for those machines to flip among several of them at intervals (particularly relatively short intervals) then I think that would be worthy of a talk almost anywhere. Moreover, if you can generalize the "slick" solution into moving traffic by choice from one of your similarly- numbered machines to another (thinking of reboots & backups, server-driven load balancing and the like) independent of service (although you could require it to be a NAT-friendly service, I guess) then that might make a trip to Phoenix worthwhile. Sean.

2 1