- NANOG - lists.nanog.org

RE: [nsp] Cisco DS3 Questions..
by Daniel Golding 22 Feb '02

22 Feb '02

That sub-interface numbering is useless for a point-to-point circuits. - Daniel Golding > -----Original Message----- > From: Eric So [mailto:eric.so@graduate.hku.hk] > Sent: Thursday, February 21, 2002 5:16 PM > To: Gyorfy, Shawn > Cc: 'nanog(a)merit.edu'; 'cisco-nsp(a)puck.nether.net' > Subject: Re: [nsp] Cisco DS3 Questions.. > > > The proprietry HDLC will give you headache if you want > to change the platform in the future. > > FR encap gives you sub-interface numbering which acts > as an identifier to network devices through a proper > indexing scheme. > > -e > > On Thu, 21 Feb 2002, Gyorfy, Shawn wrote: > > > Since the topic exploded, what are your opinions on > encapsulation of leased > > line DS3s. We currently use Frame Relay for out Point to Point DS3 > > connections. Personally, I don't know why we use FR as our > encapsulation, > > and so the question to all. If you are running Cisco to Cisco, > would it be > > wise to run HDLC or PPP? Our DS3s' here are hardly maxed out, > 15% or so, so > > I'm not complaining about the few extra bits I can squeeze out them but > > maybe that 15% can shrink to 10% with less overhead. Opinions > or examples > > of life appreciated. > > > > Thanks > > > > shawn > > > > -----Original Message----- > > From: Iljitsch van Beijnum [mailto:iljitsch@muada.com] > > Sent: Wednesday, February 20, 2002 4:28 PM > > To: Jon Mansey > > Cc: nanog(a)merit.edu > > Subject: Re: Cisco PPP DS-3 limitations - 42.9Mbpbs? > > > > > > On Wed, 20 Feb 2002, Jon Mansey wrote: > > > > > OMG! Arent we missing the point here? What about never > running links above > > > 60% or so to allow for bursts against the 5 min average, and <shudder> > > > upgrading or adding capacity when we get too little headroom. > > > > > And here we are, nickel and diming over a few MBps near to > 45M on a DS3... > > > > And why not? Obviously there is a reason why they're not upgrading, > > because there is plenty of traffic to fill up a second or faster circuit > > if packets are being dropped because of congestion. (Which has not been > > confirmed so far.) > > > > There shouldn't be any problems pushing a DS3 well beyond 99% > utilization, > > by the way. With an average packet size of 500 bytes and 98 > packets in the > > output queue on average, 99% only introduces a 9 ms delay. The extra RTT > > will also slow TCP down, but not in such a brutal way as significant > > numbers of lost packets will. Just use a queue size of 500 or so, and > > enable (W)RED to throttle back TCP when there are large bursts. > > >

1 0

Strange behavior
by Christopher J. Wolff 21 Feb '02

21 Feb '02

Greetings, For the last two days I've seen massive delays around the following hops: Does anyone have any thoughts on what is going on? 9 15 ms 31 ms 32 ms 0.so-0-3-0.XL1.LAX9.ALTER.NET [152.63.115.2] 10 32 ms 31 ms 31 ms 0.so-1-0-0.TL1.LAX9.ALTER.NET [152.63.115.142] 11 47 ms 31 ms 47 ms 0.so-4-0-0.TL1.SAC1.ALTER.NET [152.63.0.98] 12 1109 ms 1360 ms 1547 ms 0.so-7-0-0.XL1.SJC2.ALTER.NET [152.63.55.105] 13 2110 ms 2093 ms 2016 ms POS1-0.XR1.SJC2.ALTER.NET [152.63.56.138] 14 1906 ms 1766 ms 2046 ms 193.ATM7-0.GW5.SJC2.ALTER.NET [146.188.144.25] Regards, Christopher J. Wolff, VP CIO Broadband Laboratories http://www.bblabs.com

3 2

RE: red [was Re: Cisco PPP DS-3 limitations - 42.9Mbpbs?]
by Brennan_Murphy＠NAI.com 21 Feb '02

21 Feb '02

Remember too that if you have multiple types of flows on your link (FTP, TELNET, HTTP) the key isn't so much to utilize your DS3 99% as it is to ensure that the end user's expectations aren't upset. So, you might use WRED to drop bulky FTP packets instead of Voice, Video, Telnet, some HTTP, etc...in those cases where you reach the threshold that you specify. Does an FTP user really notice if their download starts at 160kbs and gyrates around between 30-150kbs..especially if they are very large downloads? But a web page taking 12 seconds to load...not good. -----Original Message----- From: smd(a)clock.org [mailto:smd@clock.org] Sent: Wednesday, February 20, 2002 1:02 PM To: nanog(a)merit.edu Subject: red [was Re: Cisco PPP DS-3 limitations - 42.9Mbpbs?] randy bush writes: | you can turn on [x]red. it will make a better choice of which packets | to drop. but packets will still be dropped [0]. fsvo "better". red drops packets randomly, a little earlier than packets will be tail-dropped by being unable to find buffer space in a queue. in most cases, tail-drop is worst for TCP connections in slow-start, and leads to such fun things as web-page-slowdown-syndrome, or ssh-stall-syndrome, with the attendant hitting of the "reload" button or frustrated banging on the keyboard trying to get one's keys to echo back, or trying to unstick the page of text being sent to /dev/tty on the remote end. "r" for random: this spreads out loss randomly, and in most cases, this means that several different TCPs each observe a single lost segment, which is quickly detected and causes an attendant slow-down, but not a stall. "e" for early: we have to keep the queue from filling, because that leads to tail-drop, so we have to drop packets when we could really send them. there is no way around this -- in virtually any implementation, the mechanism to turn on red can be translated into more fifo-queue space. the trade-off (early random dropping) is usually worth it. there are some corner cases where there is insufficient ability to spread drops across enough flows, where resulting behaviour can be alot like tail-drop (a huge difference between fast input speed and slow output speed, with only a few bulk-transfer flows), but these cases are rare, particularly where average_segment_size / bottleneck_kbps <= 0.4, with the 0.4 being a historical minimum target for queueing in the presence of flows which experience long RTTs (like ones across oceans). in general, you don't have to worry about that, though. | you can increase buffers blah blah blah. but twenty tomatoes will still | not fit in a fifteen tomato can. yup, the trick is to deploy a red control law that drops just before tail-dropping would start, so we don't lose packets unnecessarily, yet early enough before tail-dropping would start that the senders have time to back off in response to the detected lost segment, which is driven by the RTT. drop too early, and you get an underused line. drop too late and you get tail-drop, which also gives you an underused line. drop just right, and you can keep your line very very very close to 100% for sustained periods of time (hours!), assuming a large enough set of TCP flows that random dropping spreads the early dropping across different flows. | [0] - corollary: qos mechanisims decide which packets to drop. but isps | are paid not to drop any packets. it is impossible to get 0 drop and enjoy statistical multiplexing gain. statmux gain keeps the network cheap enough to use. red minimizes the liklihood that anyone will complain about performance during those inevitable periods when one's statistical bet goes wrong. Sean.

2 1

OT: Looking for advicein designing an private ATM network over a public carrier
by Rich Sena 21 Feb '02

21 Feb '02

I'm working a project in designing a private ATM network over a public provider. I have had a few options advanced to me and would like to discuss a few with some folk who have done a similar implementation. What I would like is to have a common virtual path with multiple nodes and set up my own virtual circuits between end nodes. Please reply off-list to me directly. Reply-To has been set... -- Rich Sena - ras(a)thick.net ThickNET Consulting "On the way to understanding; you understand, and forget."

1 1

best way to get to UUNet 'New York City' colo w/o car?
by bert hubert 21 Feb '02

21 Feb '02

Hi, We have half a rack in what UUNet like to call their 'New York City' colo but which is so far outside NY City that nobody there knows where it is. Now, because our people going there can't/don't want to rent a car, we find ourselves at a loss to figure out how to get there easily. The address is 401 Fieldcrest Drive, Elmsford, NY 10523-1026. We would like to know how to get their directly from JFK and from Manhattan, if possible. Last time we took a cab and that was very tricky - the cab driver proved to be very flexible and understood the output of Yahoo's driving instructions luckily, but you can't count on them being that good. It was also exceedingly expensive. Anything is fine with us, car service, public transport, whatever. Thanks for your time! Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services http://www.tk the dot in .tk Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - Linux Advanced Routing & Traffic Control: http://ds9a.nl/lartc

2 2

Cisco PPP DS-3 limitations - 42.9Mbpbs?
by Ehud Gavron 20 Feb '02

20 Feb '02

At a recent consulting opportunity, a client was only getting 40.9Mbps on their DS-3 (clearchannel DS3 over OC12 to the IP provider). (PA-2T3 to PA-2T3, unloaded CPU, drops occur when peak octets out per sec/8=40.9M) The client asked us why 40.9 was less than 44.210. The IP provider indicated that "Cisco told us the maximum theoretical rate of an interface doing DS-3 with PPP encapsulation is 42.9." I can't find documentation to the 42.9. I can't find documentation that indicates that the PPP overhead is somehow magically excluded from the IfOctetsIn and Out... I can't find documentation as to why 40.9 is the observed rate. Frankly, all my DS-3s (HDLC encap) go up to 44.210, so I was hoping I could share in the collective wisdom of NANOG readers who have DS-3s to IP providers, and can provide real observed feedback. Please reply privately. Please let me know privately if you want a summary. Thanks! Ehud Gavron gavron(a)wetwork.net

11 17

Re: Cisco PPP DS-3 limitations - 42.9Mbpbs?
by Dave Israel 20 Feb '02

20 Feb '02

The place where QOS always made sense to me is not where they want to put it (and it's very apropos to the current conversation.) Specifically, at the customer edge, as opposed to the provider core. Some folks are always going to have higher bandwidth needs than money. We can complain all we want about how, in a perfect world, everybody should buy more bandwidth before queue delay and congestion are a problem, but in the end, Generic Office X's communication budget is probably for a single T1, which can either run VOIP or be channelized into nxDS0 for phone and 24-nxDS0 for data. In either case, it doesn't take much legitimate traffic before you start seeing problems, especially if you're doing VOIP, video- conferencing, web serving, and Bob is sitting in his office listening streaming video and browsing alt.sex.toasters. And sure, it would be nice if people were responsible and didn't use bandwidth frivolously, but we're in the real world, and when people won't behave, the system should have the option of trying to do the right thing. So, at the edge, you prioritize voice over streaming media over web over news over mail, and everything behaves acceptably. *That* is where QOS really shines. For some reason, though, nobody wants to sell it there. Must be more profitable to sell 'em a bigger line. -Dave On 2/20/2002 at 14:11:01 -0800, Tony Rall said: > > On Wednesday, 2002/02/20 at 09:08 PST, Randy Bush <randy(a)psg.com> wrote: > > [0] - corollary: qos mechanisims decide which packets to drop. but isps > > are paid not to drop any packets. > > Exactly. I've been saying this to vendors for the last few years, as they > try to push their QOS mechanisms on me. > > I'm not in the business of trying to engineer some optimum packet discard > strategy. I would rather spend my time and money trying to minimize the > drop percentage. (I haven't been tested yet with the task of trying to > minimize or standardize latency for traffic like VOIP - might change my > tune if I was dealing with that.) > > Tony Rall

2 1

Re: Cisco PPP DS-3 limitations - 42.9Mbpbs?
by Tony Rall 20 Feb '02

20 Feb '02

On Wednesday, 2002/02/20 at 09:08 PST, Randy Bush <randy(a)psg.com> wrote: > [0] - corollary: qos mechanisims decide which packets to drop. but isps > are paid not to drop any packets. Exactly. I've been saying this to vendors for the last few years, as they try to push their QOS mechanisms on me. I'm not in the business of trying to engineer some optimum packet discard strategy. I would rather spend my time and money trying to minimize the drop percentage. (I haven't been tested yet with the task of trying to minimize or standardize latency for traffic like VOIP - might change my tune if I was dealing with that.) Tony Rall

2 1

RE: Cisco PPP DS-3 limitations - 42.9Mbpbs?
by Goodwin, Dustin T [IT] 20 Feb '02

20 Feb '02

I think that page 10 says it all in regards to how RED can improve circuit utilization. ftp://ftp.ee.lbl.gov/talks/vj-nanog-red.pdf - Dustin - -----Original Message----- From: Steve Naslund [mailto:snaslund@interaccess.com] Sent: Wednesday, February 20, 2002 4:24 PM To: nanog(a)merit.edu Subject: RE: Cisco PPP DS-3 limitations - 42.9Mbpbs? > On Wed, 20 Feb 2002, Jon Mansey wrote: > > > OMG! Arent we missing the point here? What about never running > links above > > 60% or so to allow for bursts against the 5 min average, and <shudder> > > upgrading or adding capacity when we get too little headroom. > > > And here we are, nickel and diming over a few MBps near to 45M > on a DS3... > > And why not? Obviously there is a reason why they're not upgrading, > because there is plenty of traffic to fill up a second or faster circuit > if packets are being dropped because of congestion. (Which has not been > confirmed so far.) > > There shouldn't be any problems pushing a DS3 well beyond 99% utilization, > by the way. With an average packet size of 500 bytes and 98 packets in the > output queue on average, 99% only introduces a 9 ms delay. The extra RTT > will also slow TCP down, but not in such a brutal way as significant > numbers of lost packets will. Just use a queue size of 500 or so, and > enable (W)RED to throttle back TCP when there are large bursts. > One problem you have here is how you are getting the utilization statistics. Since you are looking at 5 minute averages, chances are real good that your instantaneous figure is probably at the full capacity of the DS-3. As you approach the maximum capacity and start dropping packets, your throughput on the line will bounce around near the 45 megs figure but the "goodput" that the customer sees will drop dramatically. You will be sending retransmissions of the dropped packet, which then causes less bandwidth to be available for current traffic, which causes more drops, which causes more retransmissions and backoffs. My experience in the real world is that once you get over 40 mbps on a DS-3 you need to look at upgrading. Another thing I would question is where the Cisco is counting traffic. Since it is most likely looking at the real user traffic there is more likely some overhead to manage the DS-3 itself and some L2 protocol stuff also. This is definitely a factor when running ATM because the router only counts the input/output traffic after the ATM overhead has been stripped away unless you are looking at the controller and not the interface. Remember the interface is just that, an interface to the line itself. You may be able to get more accurate data by looking at the controller stats because they operate at a lower level than the interface. Steven Naslund Network Engineering Manager Hosting.com - Chicago

1 0

Re: Looking for ISP in Beirut
by Steven G. Huter 20 Feb '02

20 Feb '02

> Hello all, I'm looking for an ISP in Beirut that has 200ms or less to > New York City (a tall order I know). Hello Rishi, There is a listing of the ISPs in Lebanon, maintained at http://www.mpt.gov.lb/isplist.htm You might try checking with the engineers, cc:d above, for Fiberlink Networks and TerraNet - I believe those two have fiber connections to the East Coast. Steve Huter

1 0