Sean M. Doran wrote:

Cool, I love being talked down to by old guys. It's refreshing and doesn't happen nearly frequently enough.

I am not that old. And I did not intentionally talk down, I was tired and grumpy, I am sorry if it came out that way. But, I would love to respond to the technical points in your extremely "learn-ed" email. ( That is a genuine compliment, not sarcasm. ) Now remember, I am argueing for ATM, not about the natural color of my hair. Or, the speed of light when chasing a south bound sparrow. ;)

PDH is dead.

Uhhh....

End-to-end POTS is already dying. Worldcom is making a big deal over relatively simple technology which shuffles fax traffic over the Internet.

Yeah, and they are pushing it around over ATM.

However, there are neat plans for SS7 and neater plans for doing clever things with interpreting DTMF.

Hey, did you ever notice that ATM is called broadband ISDN, and regular ISDN is called narrow band.. And that SS7 is designed to eventually intermesh these two technologies? This is still not an argument: CON ATM. This is not intended to be flippant.... ?sp?

In this model POTS and historical telco voice and data schemes become services rather than infrastructure.

Yes, but the end user, in most cases, will still get POTS, as he knew it. And don't argue for ISDN... It is far easier to push voice across a clear channel T1 (Over ATM) using ISDN, than it is to map the RBS onto the stuff. But I agree completely.

The reason that you see "strange" or at least "unsmooth" load balancing along parallel paths is that except with fib and slow switching cisco had always forwarded packets towards the same destination out the same interface, and load balancing was performed by assigning (upon a cache fault) particular destinations to particular interfaces.

(Leon was invented to blow away cached entries so that over time prefixes would slosh about from one interface to another as they were re-demand-filled into the cache.

So ,you are saying, it is only Cisco who demonstrates this variability ? Can anyone back that ?

Points if you know who Leon and Viktor are. Hint: they're both as dead as PDH.)

This reminds me of Englands Queen. While "PDH may be dead", it is, and will be, emulated for a long time to come. So, "The Queen is dead, long live the Queen" ;)

With CEF these days you can load-balance on a per-packet basis. This has the side effect that you cannot guarantee that packets will remain in sequence if the one way delay across the load-balanced paths is off by more than about half a packet transmission time. However, you also get much more even link utilization and no ugly cache/uncache/recache at frequent intervals (which really sucks because unfortunately you have to push a packet through the slow path at every recache).

I can't ever remember a sequencing issue over parallel ATM paths... Perhaps I am just not experienced enough ;>

So anyway, as I was saying, I'm ignorant about such matters.

Hardly.

...

Audio sounds great with lots of variability

So if you aren't holding a full-duplex human-to-human conversation you introduce delay on the receiver side proportional to something like the 95th percentile and throw away outliers. If you're holding a full-duplex long-distance human-to-human conversation you can use POTS (which is dying but which will live on in emulation) and pay lots of money or you can use one of a number of rather clever VON member packages and pay alot less money but put up with little nagging problems. For local or toll-free stuff, to expect better price-performance from an end-user perspective now would require taking enormous doses of reality-altering drugs.

All is quiet in the BoardRoom, the hostile merger is about to occur. All the Executives stand pensive, awaiting the last second instructions from the CEO, in germany. " And now, whatever you do, don't forget the *snap*, *crackle*, *pop*, or it will cost us millions, Oops! gotta run, ta. (click)" I am certainly not saying voice over IP is not viable... for the right situations. Matter of Fact, voice over IP, over ATM, is pretty solid ;)

Soft real time things can be implemented across a wide variety of unpredictable media depending on the window available to service the real time events and the slope of the utility decay function.

For instance, interactive voice and video have a number of milliseconds leeway before a human audience will notice lag. Inducing a delay to avoid missing the end of the optimal window for receiving in-sequence frames or blobs of compressed voice data is wise engineering, particularly if the induced delay is adjusted to avoid it itself leading to loss of data utility.

So , what do you think CDVT is all about ;) (I know , you already knew this, I just couldn't resist )I am still looking for the CON ATM in this point....

...

However, I have NEVER failed to get the bandwith "promised" in our nets.

Sure, the problem is with mixing TCP and other window-based congestion control schemes which rely on implicit feedback with a rate-based congestion control scheme, particularly when it relies on explicit feedback. The problem is exacerbated when the former overlaps the latter, such that only a part of the path between transmitter and receiver is congestion controlled by the same rate-based explicit feedback mechanism.

What happens is that in the presence of transient congestion unless timing is very tightly synchronized (Van Jacobson has some really entertaining rants about this) the "outer loop" will react by either hovering around the equivalent of the CIR or by filling the pipe until the rate based mechanism induces queue drops. In easily observable pathological cases there is a stair step or vacillation effect resembling an old TCP sawtooth pattern rather than the much nicer patterns you get from a modern TCP with FT/FR/1321 stamps/SACK.

In other words your goodput suffers dramatically.

Good point. However, FGCRA helps, graceful drops and all that... And yes, it will be a while before the entire mechanism is completely tied together. Oh, never mind, you make my point in the following. And remember, I specifically noted that legacy equipment does not count.

...

But, doesn't that same thing happen when you over-run the receiving router ?????

Yes, and with OFRV's older equipment the lack of decent buffering (where decent output buffering is, per port, roughly the bandwidth x delay product across the network) was obvious as bandwidth * delay products increased.

With this now fixed in modern equipment and WRED available, the implicit feedback is not so much dropped packets as delayed ACKs, which leads to a much nicer subtractive slow-down by the transmitter, rather than a multiplicative backing off.

BYTW: I was arguing for ATM, not just ABR. But, the key here is to provide a reliable method whereby ATM can convey information in the same pro-active strategy. My original point is that in the ATM exchange environment, it is far less likely that an irresponsible neighbor will interrupt your service. This has scaling capacity .... And also a side effect I really like, only the ones who are abusing their lines, drop packets. (Usually...)

Finally another ABR demon is in the decay of the rate at which a VS is allowed to send traffic, which in the face of bursty traffic (as one tends to see with most TCP-based protocols) throttles goodput rather dramatically. Having to wait an RTT before an RM cell returns tends to produce unfortunate effects, and the patch around this is to try to adjust the scr contract to some decent but low value and assure that there is enough buffering to allow a VS's burst to wait to be serviced and hope that this doesn't worsen the bursty pattern by bunching up alot of data until an RM returns allowing the queue to drain suddenly.

Or, just don't overaggregate to the Nth degree. ;\ Patient: Doctor, Doctor, it hurts when I do this..... Doctor: So, don't do that. That will be 50 dollars. ;) However, you have a very good point regarding the effect of delay awaiting the RM to indicate congestion. The solution does not exist, yet.... ATM is not without design considerations. *Nothing* is a substitute for good planning. It is easier to avoid bottlenecks when the top circuit is 622mbs, and you can run parallel paths.

...

Ahhh.. We await the completion, and proper interaction of RM, ILMI, and OAM. These will, (and in some cases already DO), provide that information back to the router/tag switch. Now do they use it well ????? That is a different story....

The problem is that you need the source to slow transmission down, and the only mechanism to do that is to delay ACKs or induce packet drops. Even translating FECN/BECN into source quench or a drop close to the source is unhelpful since the data in flight will already lead to feedback which will slow down the source.

Agreed, it is not currently as pro-active as an ideal situation would be. Weaknesses still exist providing layer 2 information to the layer 3 devices. The temporary fix is to provide large ingress and egress buffers, throughout the path, and provide as much end-to-end feedback as possible, preferably with the ability for all devices in a path to react to that feedback. OAM, RM, etc. Then to do as you say, set a reasonable SCR, and hope you bought enough buffers, and the CDVT is adequate to prevent loss.

The congestion control schemes are essentially fundamentally incompatible.

Lets just say "not optimal".

...

...

Therefore, unless ABR is deliberately inducing queueing delays, there is no way your delay can be decreased when you send lots of traffic unless the ATM people have found a way to accelerate photons given enough pressure in the queues.

More available bandwidth = quicker transmission.

I still say lets switch to tachyons ;) (Just for those who don't know already, I am not serious)

...

6 core2-fddi3-0.san-francisco.yourtransit.net (-.174.56.2) 567 ms 154 ms 292 ms

...

...

...

...

>>>>>>>>>> Tell me this is a speed of light issue. >>>>>>>>>> From the FDDI to the HSSI on the same router.

This has nothing to do with the router's switching or route lookup mechanism. Router requirements allow routers to be selective in generating ICMP messages, and cisco's implementation on non-CEF routers will hand the task of generating ICMP time exceededs, port unreachables and echo replies to the main processor, which gets to the task as a low priority when it's good and ready. If the processor is doing anything else at the time you get rather long delays in replies, and if it's busy enough to start doing SPD you get nothing.

I have noted the conversations regarding this. However, I should note that according to the data we collected, there is a direct correlation between the timings increasing, and the variability/latency/loss of packets travelling the path. My argument is based on our charting of timings of this nature, and the results from BoardWatch reviews. Our timing charts matched almost perfectly to boardwatches timing of major carriers. The lower the timings, and closer to minimal variability from these timings, the better the final score from boardwatch. Perhaps this relationship is merely demonstrating that, indeed, ICMP timing is "retarded" during periods of high router utilization, due to prioritizing of processes. However, the fact that this prioritization is being "kicked in" proves to be a good indicator that a router is experiencing a serious load. And routers experiencing heavy loads, are *usually* the cuplrit in packet delay/variance/loss. Not conclusive enough for the courts, but quite informational , none-the-less.

...

Flow switching does a route determination once per flow, after that the packets are switched down a predetermined path "The Flow". Hence the term "flow switching". This reduces the variability of the entire flow.

Um, no it doesn't. As with all demand-cached forwarding schemes you have to process a packet heavily when you have a cache miss. Darren Kerr did some really neat things to make it less disgusting than previous demand-cached switching schemes emanating out of OFRV, particularly with respect to gleaning lots of useful information out of the side-effects of a hand-tuned fast path that was designed to account for all the header processing one could expect.

I have NO excuse for simplifying, other than a lack of need to get more detailed. I am now a little more versed in my intended audience.... (Postmortem) However, the concept still holds true. The flow switching is "in essence" demonstrating the advantage of minimizing interim processing. We have collected similar data, as well. (Hand tuned) BTW: Everyone seem to have missed that flow switching , in its current incarnation, does not scale well. Sort of like ethernet, there is a critical mass point. I was surprised no one attacked with that.. (not that it was important to the issue, or anything)

Flow switching does magic matching of related packets to cache entries which describe the disposition of the packet that in previous caching schemes could only be determined by processing individual packets to see if they matched various access lists and the like. It's principal neat feature is that less per-packet processing means more pps throughput.

MPLS is conceptually related, btw.

Flow switching does not improve queueing delays or speed up photons and electrons, however, nor does it worsen them, therefore the effect of flow switching on variability of normal traffic is nil.

Hold it.... doesn't a better PPS throughput imply reduced que latency ? Not the physical process of entering the que, or the physical process of exiting the que, but the median time the data is "que homed". (again, full latency of ingress to egress) Kind of "by definition" ? The more packets that you can clear, the less congested the pipe. The less congested a pipe, lower the probability that a packet will need to experience delay. Or, have I over simplified ? Or, is your operational word "normal traffic"? That would make sense.

Flow switching has mechanisms cleverer than Leon the Cleaner to delete entries from the cache and consequently there are much reduced odds of a cache fault during a long-lived flow that is constantly sending at least occasional traffic. You may see this as reducing variability. I see it as fixing a openly-admitted design flaw.

Ok. But, it still works. Reduce the time spent processing the packet/or cell, and things get smoother... Why should it be otherwise?

...

PS MAC Layer Switching, and ATM switching are apples and oranges. Although, one could be used to do the other. (Told you Dorian)

Huh?

Sorry, A response to another users mail. They referenced a holy war in the archives. I derived from that thread that it was an argument about traditional LAN switches, and their MAC based flow descisions -versus- Layer 3 Routing descisions. I guess technically speaking, it was related. But,it was not the immediate issue. I guess in the picture of things I created flame bait. I claim naivette' (sp ?) to the NANOG archives . However, I found the discussion refreshing. It was a pleasure ;) Richard