On Fri, 19 Jun 2020 at 11:35, Mark Tinka <mark.tinka@seacom.mu> wrote:

...

So instead of making it easy for software to generate MPLS packets. We are making it easy for hardware teo generate complex IP packets. Bizarre, but somewhat rational if you start from compute looking out to networks, instead of starting from networks.

Which I totally appreciate and, fundamentally, have nothing against.

My concern is when we, service providers, start to get affected because equipment manufacturers need to follow the data centre money hard, often at our expense. This is not only in the IP world, but also in the Transport world, where service providers are having to buy DWDM gear formatted for DCI. Yes, it does work, but it's not without its eccentricities.

Cycling, over the past decade, between TRILL, OTV, SPB, FabricPath, VXLAN, NV-GRE, ACI... and perhaps even EVPN, there is probably a lesson to be learned.

Maybe this is fundamental and unavoidable, maybe some systematic bias in human thinking drives us towards simple software and complex hardware. Is there an alternative future, where we went with Itanium? Where we have simple hardware and an increasingly complex compiler and increasingly complex runtime making sure the program runs fast on that simple hardware? Instead we have two guys in tel aviv waking up in night terror every night over confusion why does x86 run any code at all, how come it works. And I'm at home 'hehe new intc make program go fast:)))' Now that we have comparatively simple compilers and often no runtime at all, the hardware has to optimise the shitty program for us, but as we don't get to see how the sausage is made, we think it's probably something that is well done, robust and correct. If we'd do this in software, we'd all have to suffer how fragile the compiler and runtime are and how unapproachable they are. -- ++ytti

Maybe this is fundamental and unavoidable, maybe some systematic bias in human thinking drives us towards simple software and complex hardware.

how has software progressed in the last 50-70 years as compared to hardware? my watch has how many orders of magnitude of compute vs the computer which took a large room when i was but a youth? and we have barely avoided writing stacks in cobol; but we're close, assembler++. randy

On 19/Jun/20 16:45, Masataka Ohta wrote:

The problem of MPLS, or label switching in general, is that, though it was advertised to be topology driven to scale better than flow driven, it is actually flow driven with poor scalability.

Thus, it is impossible to deploy any technology scalably over MPLS.

MPLS was considered to scale, because it supports nested labels corresponding to hierarchical, thus, scalable, routing table.

However, to assign nested labels at the source, the source must know hierarchical routing table at the destination, even though the source only knows hierarchical routing table at the source itself.

So, the routing table must be flat, which dose not scale, or the source must detect flows to somehow request hierarchical destination routing table on demand, which means MPLS is flow driven.

People, including some data center people, avoiding MPLS, know network scalability better than those deploying MPLS.

It is true that some performance improvement is possible with label switching by flow driven ways, if flows are manually detected. But, it means extra label-switching-capable equipment and administrative effort to detect flows, neither of which do not scale and cost a lot.

It cost a lot less to have more plain IP routers than insisting on having a little fewer MPLS routers.

I wouldn't agree. MPLS is a purely forwarding paradigm, as is hop-by-hop IP. Even with hop-by-hop IP, you need the edge to be routing-aware. I wasn't at the table when the MPLS spec. was being dreamed up, but I'd find it very hard to accept that someone drafting the idea advertised it as being a replacement or alternative for end-to-end IP routing and forwarding. Whether you run MPLS or not, you will always have routing table scaling concerns. So I'm not quite sure how that is MPLS's problem. If you can tell me how NOT running MPLS affords you a "hierarchical, scalable" routing table, I'm all ears. Whether you forward in IP or in MPLS, scaling routing is an ever clear & present concern. Where MPLS can directly mitigate that particular concern is in the core, where you can remove BGP. But you still need routing in the edge, whether you forward in IP or MPLS. Mark.

MPLS was since day one proposed as enabler for services originally L3VPNs and RSVP-TE.

MPLS day one was mike o'dell wanting to move his city/city traffic matrix from ATM to tag switching and open cascade's hold on tags. randy

< ranting of a curmudgeonly old privileged white male >

...

...

MPLS was since day one proposed as enabler for services originally L3VPNs and RSVP-TE. MPLS day one was mike o'dell wanting to move his city/city traffic matrix from ATM to tag switching and open cascade's hold on tags. And IIRC, Tag switching day one was Cisco overreacting to Ipsilon.

i had not thought of it as overreacting; more embrace and devour. mo and yakov, aided and abetted by sob and other ietf illuminati, helped cisco take the ball away from Ipsilon, Force10, ... but that is water over the damn, and my head is hurting a bit from thinking on too many levels at once. there is saku's point of distributing labels in IGP TLVs/LSAs. i suspect he is correct, but good luck getting that anywhere in the internet vendor task force. and that tells us a lot about whether we can actually effect useful simplification and change. is a significant part of the perception that there is a forwarding problem the result of the vendors, 25 years later, still not designing for v4/v6 parity? there is the argument that switching MPLS is faster than IP; when the pressure points i see are more at routing (BGP/LDP/RSVP/whatever), recovery, and convergence. did we really learn so little from IP routing that we need to recreate analogous complexity and fragility in the MPLS control plane? ( sound of steam eminating from saku's ears :) and then there is buffering; which seems more serious than simple forwarding rate. get it there faster so it can wait in a queue? my principal impression of the Stanford/Google workshops was the parable of the blind men and the elephant. though maybe Matt had the main point: given scaling 4x, Moore's law can not save us and it will all become paced protocols. will we now have a decade+ of BBR evolution and tuning? if so, how do we engineer our networks for that? and up 10,000m, we watch vendor software engineers hand crafting in an assembler language with if/then/case/for, and running a chain of checking software to look for horrors in their assembler programs. it's the bleeping 21st century. why are the protocol specs not formal and verified, and the code formally generated and verified? and don't give me too slow given that the hardware folk seem to be able to do 10x in the time it takes to run valgrind a few dozen times. we're extracting ore with hammers and chisels, and then hammering it into shiny objects rather than safe and securable network design and construction tools. apologies. i hope you did not read this far. randy

The problem of MPLS, however, is that, it must also be flow driven, because detailed route information at the destination is necessary to prepare nested labels at the source, which costs a lot and should be attempted only for detected flows.

MPLS is not flow driven. I sent some mail about it but perhaps it bounced. MPLS LDP or L3VPNs was NEVER flow driven. Since day one till today it was and still is purely destination based. Transport is using LSP to egress PE (dst IP). L3VPNs are using either per dst prefix, or per CE or per VRF labels. No implementation does anything upon "flow detection" - to prepare any nested labels. Even in FIBs all information is preprogrammed in hierarchical fashion well before any flow packet arrives. Thx, R.

...

there is the argument that switching MPLS is faster than IP; when the pressure points i see are more at routing (BGP/LDP/RSVP/whatever), recovery, and convergence.

Routing table at IPv4 backbone today needs at most 16M entries to be looked up by simple SRAM, which is as fast as MPLS look up, which is one of a reason why we should obsolete IPv6.

Though resource reserved flows need their own routing table entries, they should be charged proportional to duration of the reservation, which can scale to afford the cost to have the entries.

Masataka Ohta

Robert Raszuk wrote:

MPLS LDP or L3VPNs was NEVER flow driven.

Since day one till today it was and still is purely destination based.

If information to create labels at or near sources to all the possible destinations is distributed in advance, may be. But it is effectively flat routing, or, in extreme cases, flat host routing. Or, if information to create labels to all the active destinations is supplied on demand, it is flow driven. On day one, Yakov said MPLS had scaled because of nested labels corresponding to routing hierarchy. Masataka Ohta

Mark Tinka wrote:

...

If information to create labels at or near sources to all the possible destinations is distributed in advance, may be.

But this is what happens today.

That is a tragedy.

Whether you do it manually or use a label distribution protocol, FEC's are pre-computed ahead of time.

What am I missing?

If all the link-wise (or, worse, host-wise) information of possible destinations is distributed in advance to all the possible sources, it is not hierarchical but flat (host) routing, which scales poorly. Right?

...

But it is effectively flat routing, or, in extreme cases, flat host routing.

I still don't get it.

Why, do you think, flat routing does not but hierarchical routing does scale? It is because detailed information to reach destinations below certain level is advertised not globally but only for small part of the network around the destinations. That is, with hierarchical routing, detailed information around destinations is actively hidden from sources. So, with hierarchical routing, routing protocols can carry only rough information around destinations, from which, source side can not construct detailed (often purposelessly nested) labels required for MPLS.

So why create labels on-demand if a box to handle the traffic is already in place and actively working, day-in, day-out?

According to your theory to ignore routing traffic, we can be happy with global *host* routing table with 4G entries for IPv4 and a lot lot lot more than that for IPv6. CIDR should be unnecessary complication to the Internet With nested labels, you don't need so much labels at certain nesting level, which was the point of Yakov, which does not mean you don't need so much information to create entire nested labels at or near the sources. The problem is that we can't afford traffic (and associated processing by all the related routers or things like those) and storage (at or near source) for routing (or MPLS, SR* or whatever) with such detailed routing at the destinations. Masataka Ohta

Robert Raszuk wrote:

Neither link wise nor host wise information is required to accomplish say L3VPN services. Imagine you have three sites which would like to interconnect each with 1000s of users.

For a single customer of an ISP with 1000s of end users. OK. But, it should be noted that a single class B routing table entry often serves for an organization with 10000s of users, which is at least our case here at titech.ac.jp. It should also be noted that, my concern is scalability in ISP side.

Moreover if you have 1000 PEs and those three sites are attached only to 6 of them - only those 6 PEs will need to learn those routes (Hint: RTC - RFC4684)

If you have 1000 PEs, you should be serving for somewhere around 1000 customers. And, if I understand BGP-MP correctly, all the routing information of all the customers is flooded by BGP-MP in the ISP. Then, it should be a lot better to let customer edges encapsulate L2 or L3 over IP, with which, routing information within customers is exchanged by customer provided VPN without requiring extra overhead of maintaining customer local routing information by the ISP. If a customer want customer-specific SLA, it can be described as SLA between customer edge routers, for which, intra-ISP MPLS may or may not be used. For the ISP, it can be as profitable as PE-based VRF solutions, because customers so relying on ISPs will let the ISP provide and maintain customer edges. The only difference should be on profitability for router makers, which want to make routing system as complex as possible or even a lot more than that to make backbone routers a lot profitable product.

...

With nested labels, you don't need so much labels at certain nesting level, which was the point of Yakov, which does not mean you don't need so much information to create entire nested labels at or near the sources.

Label stack is here from day one.

Label stack was there, because of, now recognized to be wrong, statement of Yakov on day one and I can see no reason still to keep it. Masataka Ohta

Mark Tinka wrote:

...

But, it should be noted that a single class B...

CIDR - let's not teach the kids old news :-).

Saying /16 is ambiguous depends on IP version.

...

And, if I understand BGP-MP correctly, all the routing information of all the customers is flooded by BGP-MP in the ISP.

Yes, best practice is in iBGP.

Some operators may still be using an IGP for this. It would work, but scales poorly.

The amount of flooded traffic is not so different.

...

Then, it should be a lot better to let customer edges encapsulate L2 or L3 over IP, with which, routing information within customers is exchanged by customer provided VPN without requiring extra overhead of maintaining customer local routing information by the ISP.

You mean like IP-in-IP or GRE? That already happens today, without any intervention from the ISP.

I know, though I didn't know ISP's are not offering SLA for it.

There are few ISP's who would be able to terminate an IP or GRE tunnel on-net, end-to-end.

And even then, they might be reluctant to offer any SLA's because those tunnels are built on the CPE, typically outside of their control.

The condition to offer SLA beyond a network of an ISP should not "trusted NNI" but policing by the ISP with ISP's own equipment, which prevent too much traffic enter the network.

If ISP's didn't make money from MPLS/VPN's, router vendors would not be as keen on adding the capability in their boxes.

It is like telco was making money by expensive telephone exchangers only to be replaced by ISPs, I'm afraid.

Label stacking is fundamental to the "MP" part of MPLS. Whether your payload is IP, ATM, Ethernet, Frame Relay, PPP, HDLC, e.t.c., the ability to stack labels is what makes an MPLS network payload agnostic. There is value in that.

What? You are saying "payload" not something carrying "payload" is MP. Then, plain Ethernet is MP with EtherType, isn't it? Masataka Ohta

Owen DeLong wrote:

...

Saying /16 is ambiguous depends on IP version.

Not really… A /16 in IPv6 is a lot more addresses, but its still using the first 16 bits to specify the prefix, same as IPv4.

As I wrote: : But, it should be noted that a single class B routing table entry : often serves for an organization with 10000s of users, which is : at least our case here at titech.ac.jp. the number of remaining bits save the first 16 matters, which depends on IP version. Masataka Ohta

Masataka Ohta Sent: Monday, June 22, 2020 1:49 PM

Robert Raszuk wrote:

...

Moreover if you have 1000 PEs and those three sites are attached only to 6 of them - only those 6 PEs will need to learn those routes (Hint: RTC - RFC4684)

If you have 1000 PEs, you should be serving for somewhere around 1000 customers.

And, if I understand BGP-MP correctly, all the routing information of all the customers is flooded by BGP-MP in the ISP.

Not quite, The routing information is flooded by default, but the receivers will cherry pick what they need and drop the rest. And even if the default flooding of all and dropping most is a concern -it can be addressed where only the relevant subset of all the routing info is sent to each receiver. The key takeaway however is that no single entity in SP network, be it PE, or RR, or ASBR...., ever needs everything, you can always slice and dice indefinitely. So to sum it up you simply can not run into any scaling ceiling with MP-BGP architecture. adam

adamv0025@netconsultings.com wrote:

The key takeaway however is that no single entity in SP network, be it PE, or RR, or ASBR...., ever needs everything, you can always slice and dice indefinitely. So to sum it up you simply can not run into any scaling ceiling with MP-BGP architecture.

Flooding nature of BGP requires all the related entities treat everything, regardless of whether they need it entirely or not. Masataka Ohta

On 21/Jun/20 14:36, Masataka Ohta wrote:

 

That is a tragedy.

Well...

If all the link-wise (or, worse, host-wise) information of possible destinations is distributed in advance to all the possible sources, it is not hierarchical but flat (host) routing, which scales poorly.

Right?

Host NLRI is summarized in iBGP within the domain, and eBGP outside the domain. It's no longer novel to distribute end-user NLRI in the IGP. If folk are still doing that, I can't feel sympathy for the pain they may experience.

Why, do you think, flat routing does not but hierarchical routing does scale?

It is because detailed information to reach destinations below certain level is advertised not globally but only for small part of the network around the destinations.

That is, with hierarchical routing, detailed information around destinations is actively hidden from sources.

So, with hierarchical routing, routing protocols can carry only rough information around destinations, from which, source side can not construct detailed (often purposelessly nested) labels required for MPLS.

But hosts often point default to a clever router. That clever router could also either point default to the provider, or carry a full BGP table from the provider. Neither the host nor their first-hop gateway need to be MPLS-aware. There are use-cases where a customer CPE can be MPLS-aware, but I'd say that in nearly 99.999% of all cases, CPE are never MPLS-aware.

According to your theory to ignore routing traffic, we can be happy with global *host* routing table with 4G entries for IPv4 and a lot lot lot more than that for IPv6. CIDR should be unnecessary complication to the Internet

Not sure what Internet you're running, but I, generally, accept aggregate IPv4 and IPv6 BGP routes from other AS's. I don't need to know every /32 or /128 host that sits behind them.

With nested labels, you don't need so much labels at certain nesting level, which was the point of Yakov, which does not mean you don't need so much information to create entire nested labels at or near the sources.

I don't know what Yakov advertised back in the day, but looking at what I and a ton of others are running in practice, in the real world, today, I don't see what you're talking about. Again, if you can identify an actual scenario today, in a live, large scale (or even small scale) network, I'd like to know. I'm talking about what's in practice, not theory.

The problem is that we can't afford traffic (and associated processing by all the related routers or things like those) and storage (at or near source) for routing (or MPLS, SR* or whatever) with such detailed routing at the destinations.

Again, I disagree as I mentioned earlier, because you won't be able to buy a router today that does only IP any cheaper than it does both IP and MPLS. MPLS has become mainstream, that its economies of scale have made the consideration between it and IP a non-starter. Heck, you can even do it in Linux... Mark.

On 22/Jun/20 15:08, Masataka Ohta wrote:

  The requirement from the E2E principle is that routers should be dumb and hosts should be clever or the entire system do not. scale reliably.

And yet in the PTT world, it was the other way around. Clever switching and dumb telephone boxes. How things have since evened out. I can understand the concern about making the network smart. But even a smart network is not as smart as a host. My laptop can do a lot of things more cleverly than any of the routers in my network. It just can't do them at scale, consistently, for a bunch of users. So the responsibility gets to be shared, with the number of users being served diminishing as you enter and exit the edge of the network. It's probably not yet an ideal networking paradigm, but it's the one we have now that is a reasonably fair compromise.

In this case, such clever router can ever exist only near the destination unless very detailed routing information is flooded all over the network to all the possible sources.

I will admit that bloating router code over recent years to become terribly smart (CGN, Acceleration, DoS mitigation, VPN's, SD-WAN, IDS, Video Monitoring, e.t.c.) can become a run away problem. I've often joked that with all the things being thrown into BGP, we may just see it carrying DNS too, hehe. Personally, the level of intelligence we have in routers now beyond being just Layer 1, 2, 3 - and maybe 4 - crunching machines is just as far as I'm willing to go. If, like me, you keep pushing back on vendors trying to make your routers also clean your dishes, they'll take the hint and stop bloating the code. Are MPLS/VPN's overly clever? I think so. But considering the pay-off and how much worse it could get, I'm willing to accept that.

A router can't be clever on something, unless it is provided with very detailed information on all the possible destinations, which needs a lot of routing traffic making entire system not to scale.

Well, if you can propose a better way to locate hosts on a global network not owned by anyone, in a connectionless manner, I'm sure we'd all be interested. Mark.

On Mon, Jun 22, 2020 at 7:18 PM Randy Bush <randy@psg.com> wrote:

how did that work out for the ptts? :)

Though its release slipped by three years, by 1995 ATM had started to replace IP as the protocol of choice. By 1999, IP was used only by a small number of academic networks. Nah, I don't think there is anywhere in the multiverse where fat pipes and dumb switches doesn't win. -- Fletcher Kittredge GWI 207-602-1134 www.gwi.net

Masataka Ohta Sent: Sunday, June 21, 2020 1:37 PM

...

Whether you do it manually or use a label distribution protocol, FEC's are pre-computed ahead of time.

What am I missing?

If all the link-wise (or, worse, host-wise) information of possible destinations is distributed in advance to all the possible sources, it is not hierarchical but flat (host) routing, which scales poorly.

Right?

On the Internet yes in controlled environments no, as in these environments the set of possible destinations is well scoped. Take an MPLS enabled DC for instance, every VM does need to talk to only a small subset of all the VMs hosted in a DC. Hence each VM gets flow transport labels programmed via centralized end-to-end flow controllers on a need to know bases (not everything to everyone). (E.g. dear vm1 this is how you get your EF/BE flows via load-balancer and FW to backend VMs in your local pod, this is how you get via local pod fw to internet gw, etc..., done) Now that you have these neat "pipes" all over the place connecting VMs it's easy for the switching fabric controller to shuffle elephant and mice flows around in order to avoid any link saturation. And now imagine a bit further doing the same as above but with CPEs on a Service Provider network... yep, no PEs acting as chokepoints for MPLS label switch paths to flow assignment, needing massive FIBs and even bigger, just dumb MPLS switch fabric, all the "hard-work" is offloaded to centralized controllers (and CPEs for label stack imposition) -but only on a need to know bases (not everything to everyone). Now in both cases you're free to choose to what extent should the MPLS switch fabric be involved with the end-to-end flows by imposing hierarchies to the MPLS stack. In light of the above, does it suck to have just 20bits of MPLS label space? Absolutely. Adam

But MPLS can be made flow driven (it can be made whatever the policy dictates), for instance DSCP driven… adam From: NANOG <nanog-bounces+adamv0025=netconsultings.com@nanog.org> On Behalf Of Robert Raszuk Sent: Saturday, June 20, 2020 4:13 PM To: Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> Cc: North American Network Operators' Group <nanog@nanog.org> Subject: Re: why am i in this handbasket? (was Devil's Advocate - Segment Routing, Why?) The problem of MPLS, however, is that, it must also be flow driven, because detailed route information at the destination is necessary to prepare nested labels at the source, which costs a lot and should be attempted only for detected flows. MPLS is not flow driven. I sent some mail about it but perhaps it bounced. MPLS LDP or L3VPNs was NEVER flow driven. Since day one till today it was and still is purely destination based. Transport is using LSP to egress PE (dst IP). L3VPNs are using either per dst prefix, or per CE or per VRF labels. No implementation does anything upon "flow detection" - to prepare any nested labels. Even in FIBs all information is preprogrammed in hierarchical fashion well before any flow packet arrives. Thx, R.

there is the argument that switching MPLS is faster than IP; when the pressure points i see are more at routing (BGP/LDP/RSVP/whatever), recovery, and convergence.

Routing table at IPv4 backbone today needs at most 16M entries to be looked up by simple SRAM, which is as fast as MPLS look up, which is one of a reason why we should obsolete IPv6. Though resource reserved flows need their own routing table entries, they should be charged proportional to duration of the reservation, which can scale to afford the cost to have the entries. Masataka Ohta

From: Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> Sent: Monday, June 22, 2020 2:17 PM

adamv0025@netconsultings.com wrote:

...

But MPLS can be made flow driven (it can be made whatever the policy dictates), for instance DSCP driven.

The point of Yakov on day one was that, flow driven approach of Ipsilon does not scale and is unacceptable.

Though I agree with Yakov here, we must also eliminate all the flow driven approaches by MPLS or whatever.

First I'd need a definition of what flow means in this discussion are we considering 5-tuple or 4-tuple or just SRC-IP & DST-IP, is DSCP marking part of it? Second, although I agree that ~1M unique identifiers is not ideal, can you provide examples of MPLS applications where 1M is limiting? What particular aspect? Is it 1M interfaces per MPLS switching fabric box? Or 1M unique flows (or better flow groups) originated by a given VM/Container/CPE? Or 1M destination entities (IPs or apps on those IPs) that any particular VM/Container/CPE needs to talk to? Or 1M customer VPNs or 1M PE-CPE links, if PE acts as a bottleneck? adam

Masataka Ohta wrote:

...

The point of Yakov on day one was that, flow driven approach of Ipsilon does not scale and is unacceptable.

Though I agree with Yakov here, we must also eliminate all the flow driven approaches by MPLS or whatever.

I still don't see them in practice, even though they may have been proposed.

I don't know, either, as it's Adam who said:

But MPLS can be made flow driven (it can be made whatever the policy dictates), for instance DSCP driven…

Masataka Ohta

there is saku's point of distributing labels in IGP TLVs/LSAs. i suspect he is correct, but good luck getting that anywhere in the internet vendor task force.

Perhaps I will surprise a few but this is not only already in RFC formats - it is also shipping already across vendors for some time now. SR-MPLS (as part of its spec) does exactly that. You do not need to use any SR if you do not want, you still can encapsulate your packets with transport label corresponding to your exit at any ingress and forget about LDP for good. But with that let's not forget that aggregation here is still not spec-ed out well and to the best of my knowledge it is also not shipping yet. I recently proposed an idea how to aggregate SRGBs .. one vendor is analyzing it. Best, R. On Sat, Jun 20, 2020 at 1:33 AM Randy Bush <randy@psg.com> wrote:

< ranting of a curmudgeonly old privileged white male >

...

...

...

MPLS was since day one proposed as enabler for services originally L3VPNs and RSVP-TE. MPLS day one was mike o'dell wanting to move his city/city traffic matrix from ATM to tag switching and open cascade's hold on tags. And IIRC, Tag switching day one was Cisco overreacting to Ipsilon.

i had not thought of it as overreacting; more embrace and devour. mo and yakov, aided and abetted by sob and other ietf illuminati, helped cisco take the ball away from Ipsilon, Force10, ...

but that is water over the damn, and my head is hurting a bit from thinking on too many levels at once.

there is saku's point of distributing labels in IGP TLVs/LSAs. i suspect he is correct, but good luck getting that anywhere in the internet vendor task force. and that tells us a lot about whether we can actually effect useful simplification and change.

is a significant part of the perception that there is a forwarding problem the result of the vendors, 25 years later, still not designing for v4/v6 parity?

there is the argument that switching MPLS is faster than IP; when the pressure points i see are more at routing (BGP/LDP/RSVP/whatever), recovery, and convergence.

did we really learn so little from IP routing that we need to recreate analogous complexity and fragility in the MPLS control plane? ( sound of steam eminating from saku's ears :)

and then there is buffering; which seems more serious than simple forwarding rate. get it there faster so it can wait in a queue? my principal impression of the Stanford/Google workshops was the parable of the blind men and the elephant. though maybe Matt had the main point: given scaling 4x, Moore's law can not save us and it will all become paced protocols. will we now have a decade+ of BBR evolution and tuning? if so, how do we engineer our networks for that?

and up 10,000m, we watch vendor software engineers hand crafting in an assembler language with if/then/case/for, and running a chain of checking software to look for horrors in their assembler programs. it's the bleeping 21st century. why are the protocol specs not formal and verified, and the code formally generated and verified? and don't give me too slow given that the hardware folk seem to be able to do 10x in the time it takes to run valgrind a few dozen times.

we're extracting ore with hammers and chisels, and then hammering it into shiny objects rather than safe and securable network design and construction tools.

apologies. i hope you did not read this far.

randy

On 20/Jun/20 01:32, Randy Bush wrote:

there is saku's point of distributing labels in IGP TLVs/LSAs. i suspect he is correct, but good luck getting that anywhere in the internet vendor task force. and that tells us a lot about whether we can actually effect useful simplification and change.

This is shipping today with SR-MPLS. Besides still being brand new and not yet fully field tested by the community, my other concern is unless you are running a Juniper and have the energy to pull a "Vijay Gill" and move your entire backbone to IS-IS, you'll get either no SR-ISISv6 support, no SR-OSPFv3 support, or both, with all the vendors. Which brings me back to the same piss-poor attention LDPv6 is getting, which is, really, poor attention to IPv6. Kind of hard for operators to take IPv6 seriously at this level if the vendors, themselves, aren't.

is a significant part of the perception that there is a forwarding problem the result of the vendors, 25 years later, still not designing for v4/v6 parity?

I think the forwarding is fine, if you're carrying the payload in MPLS. The problem is the control plane. It's not insurmountable; the vendors just want to do less work. The issue is IPv4 is gone, and trying to keep it around will only lead to the creation of more hacks, which will further complicate the control and data plane.

there is the argument that switching MPLS is faster than IP; when the pressure points i see are more at routing (BGP/LDP/RSVP/whatever), recovery, and convergence.

Either way, the MPLS or IP problem already has an existing solution. If you like IP, you can keep it. If you like MPLS, you can keep it. So I'd be spending less time on the forwarding (of course, if there are ways to improve that and someone has the time, why not), and as you say, work on fixing the control plane and the signaling for efficiency and scale.

did we really learn so little from IP routing that we need to recreate analogous complexity and fragility in the MPLS control plane? ( sound of steam eminating from saku's ears :)

The path to SR-MPLS's inherent signaling carried in the IGP is an optimum solution, that even I have been wanting since inception. But, it's still too fresh, global deployment is terrible, and there is still much to be learned about how it behaves outside of the lab. For me, a graceful approach toward SR via LDPv6 makes sense. But, as always, YMMV.

and then there is buffering; which seems more serious than simple forwarding rate. get it there faster so it can wait in a queue? my principal impression of the Stanford/Google workshops was the parable of the blind men and the elephant. though maybe Matt had the main point: given scaling 4x, Moore's law can not save us and it will all become paced protocols. will we now have a decade+ of BBR evolution and tuning? if so, how do we engineer our networks for that?

This deserves a lot more attention than it's receiving. The problem is it doesn't sound sexy enough to compile into a PPT that you can project to suits whom you need to part with cash. It doesn't have that 5G or SRv6 or Controller or IoT ring to it :-). It's been a while since vendors that control a large portion of the market paid real attention to their geeky side. The buffer problem, for me, would fall into that category. Maybe a smaller, more agile, more geeky start-up, can take the lead with this one.

and up 10,000m, we watch vendor software engineers hand crafting in an assembler language with if/then/case/for, and running a chain of checking software to look for horrors in their assembler programs. it's the bleeping 21st century. why are the protocol specs not formal and verified, and the code formally generated and verified? and don't give me too slow given that the hardware folk seem to be able to do 10x in the time it takes to run valgrind a few dozen times.

And for today's episode of Jeopardy:     "What used to be the IETF?"

we're extracting ore with hammers and chisels, and then hammering it into shiny objects rather than safe and securable network design and construction tools.

Rush it out the factory, fast, even though it's not ready. Get all their money before they board the ship and sail for Mars. Mark.

But, today, people are seems to be using, so called, MPLS, with

explicitly configured flows, administration of which does not

scale and is annoying.

I am actually not sure what you are talking about here. The only per flow action in any MPLS deployments I have seen was mapping flow groups to specific TE-LSPs. In all other TDP or LDP cases flow == IP destination so it is exact based on the destination reachability. And such mapping is based on the LDP FEC to IGP (or BGP) match. Even worse, if route near the destination expected to pop the label

chain goes down, how can the source knows that the router goes down and choose alternative router near the destination?

In normal MPLS the src does not pick the transit paths. Transit is 100% driven by IGP and if you loose a node local connectivity restoration techniques (FRR or IGP convergence applies). If egress signalled implicit NULL it would signal it to any IGP peer. That is also possible with SR-MPLS too. No change ... no per flow state at all more then per IP destination routing. If you want to control your transit hops you can - but this is an option not w requirement. MPLS with hierarchical routing just does not scale. While I am not defending MPLS here and 100% agree that IP as transit is a much better option today and tomorrow I also would like to make sure we communicate true points. So when you say it does not scale - it could be good to list what exactly does not scale by providing a real network operational example. Many thx, R.

On Sat, Jun 20, 2020 at 11:08 AM Mark Tinka <mark.tinka@seacom.mu> wrote:

...

MPLS with hierarchical routing just does not scale.

With Internet in a VRF, I truly agree.

But if you run a simple global BGP table and no VRF's, I don't see an issue. This is what we do, and our scaling concerns are exactly the same whether we run plain IP or IP/MPLS.

Mark.

We run the Internet in a VRF to get watertight separation between management and the Internet. I do also have a CGN vrf but that one has very few routes in it (99% being subscriber management created, eg. one route per customer). Why would this create a scaling issue? If you collapse our three routing tables into one, you would have exactly the same number of routes. All we did was separate the routes into namespaces, to establish a firewall that prevents traffic to flow where it shouldn't. Regards, Baldur

On Sat, Jun 20, 2020 at 12:38 PM Mark Tinka <mark.tinka@seacom.mu> wrote:

On 20/Jun/20 11:27, Baldur Norddahl wrote:

...

We run the Internet in a VRF to get watertight separation between management and the Internet. I do also have a CGN vrf but that one has very few routes in it (99% being subscriber management created, eg. one route per customer). Why would this create a scaling issue? If you collapse our three routing tables into one, you would have exactly the same number of routes. All we did was separate the routes into namespaces, to establish a firewall that prevents traffic to flow where it shouldn't.

It may be less of an issue in 2020 with the current control planes and how far the code has come, but in the early days of l3vpn's, the number of VRF's you could have was directly proportional to the number of routes you had in each one. More VRF's, less routes for each. More routes per VRF, less VRF's in total.

I don't know if that's still an issue today, as we don't run the Internet in a VRF. I'd defer to those with that experience, who knew about the scaling limitations of the past.

I can't speak for the year 2000 as I was not doing networking at this level at that time. But when I check the specs for the base mx204 it says something like 32 VRFs, 2 million routes in FIB and 6 million routes in RIB. Clearly those numbers are the total of routes across all VRFs otherwise you arrive at silly numbers (64 million FIB if you multiply, 128k FIB if you divide by 32). My conclusion is that scale wise you are ok as long you do not try to have more than one VRF with a complete copy of the DFZ. More worrying is that 2 million routes will soon not be enough to install all routes with a backup route, invalidating BGP FRR. Regards, Baldur

On Sun, Jun 21, 2020 at 9:56 AM Mark Tinka <mark.tinka@seacom.mu> wrote:

On 20/Jun/20 22:00, Baldur Norddahl wrote:

I can't speak for the year 2000 as I was not doing networking at this level at that time. But when I check the specs for the base mx204 it says something like 32 VRFs, 2 million routes in FIB and 6 million routes in RIB. Clearly those numbers are the total of routes across all VRFs otherwise you arrive at silly numbers (64 million FIB if you multiply, 128k FIB if you divide by 32). My conclusion is that scale wise you are ok as long you do not try to have more than one VRF with a complete copy of the DFZ.

I recall a number of networks holding multiple VRF's, including at least 2x Internet VRF's, for numerous use-cases. I don't know if they still do that today, but one can get creative real quick :-).

Yes I once made a plan to have one VRF per transit provider plus a peering VRF. That way our BGP customers could have a session with each of those VRFs to allow them full control of the route mix. I would of course also need a Internet VRF for our own needs. But the reality of that would be too many copies of the DFZ in the routing tables. Although not necessary in the FIB as each of the transit VRFs could just have a default route installed. Regards, Baldur

On Sun, Jun 21, 2020 at 1:30 PM Mark Tinka <mark.tinka@seacom.mu> wrote:

On 21/Jun/20 12:45, Baldur Norddahl wrote:

Yes I once made a plan to have one VRF per transit provider plus a peering VRF. That way our BGP customers could have a session with each of those VRFs to allow them full control of the route mix. I would of course also need a Internet VRF for our own needs.

But the reality of that would be too many copies of the DFZ in the routing tables. Although not necessary in the FIB as each of the transit VRFs could just have a default route installed.

We just opted for BGP communities :-).

Not really the same. Lets say the best path is through transit 1 but the customer thinks transit 1 sucks balls and wants his egress traffic to go through your transit 2. Only the VRF approach lets every BGP customer, even single homed ones, make his own choices about upstream traffic. You would be more like a transit broker than a traditional ISP with a routing mix. Your service is to buy one place, but get the exact same product as you would have if you bought from top X transits in your area. Delivered as X distinct BGP sessions to give you total freedom to send traffic via any of the transit providers. This is also the reason you do not actually need any routes in the FIB for each of those transit VRFs. Just a default route because all traffic will unconditionally go to said transit provider. The customer routes would still be there of course. Regards, Baldur

Hi Baldur,

From memory mx204 FIB is 10M (v4/v6) and RIB 30M for each v4 and v6.

And remember the FIB is hierarchical -so it’s the next-hops per prefix you are referring to with BGP FRR. And also going from memory of past scaling testing, if pfx1+NH1 == x, then Pfx1+NH1+NH2 !== 2x, where x is used FIB space. adam From: NANOG <nanog-bounces+adamv0025=netconsultings.com@nanog.org> On Behalf Of Baldur Norddahl Sent: Saturday, June 20, 2020 9:00 PM I can't speak for the year 2000 as I was not doing networking at this level at that time. But when I check the specs for the base mx204 it says something like 32 VRFs, 2 million routes in FIB and 6 million routes in RIB. Clearly those numbers are the total of routes across all VRFs otherwise you arrive at silly numbers (64 million FIB if you multiply, 128k FIB if you divide by 32). My conclusion is that scale wise you are ok as long you do not try to have more than one VRF with a complete copy of the DFZ. More worrying is that 2 million routes will soon not be enough to install all routes with a backup route, invalidating BGP FRR.

On 20/Jun/20 14:41, Masataka Ohta wrote:

  There are many. So, our research group tried to improve RSVP.

I'm a lot younger than the Internet, but I read a fair bit about its history. I can't remember ever coming across an implementation of RSVP between a host and the network in a commercial setting. If I missed it, kindly share, as I'd be keen to see how that went.

Practically, the most serious problem of RSVP is, like OSPF, using unreliable link multicast to reliably exchange signalling messages between routers, making specification and implementations very complicated.

So, we developed SRSVP (Simple RSVP) replacing link multicast by, like BGP, link local TCP mesh (thanks to the CATENET model, unlike BGP, there is no scalability concern). Then, it was not so difficult to remove other problems.

Was "S-RSVP" ever implemented, and deployed?

However, perhaps, most people think show stopper to RSVP is lack of scalability of weighted fair queueing, though, it is not a problem specific to RSVP and MPLS shares the same problem.

QoS has nothing to do with MPLS. You can do QoS with or without MPLS. I should probably point out, also, that RSVP (or RSVP-TE) is not MPLS. They collaborate, yes, but we'd be doing the community a disservice by interchanging them for one another.

Obviously, weighted fair queueing does not scale because it is based on deterministic traffic model of token bucket model and, these days, people just use some ad-hoc ways for BW guarantee implicitly assuming stochastic traffic model. I even developed a little formal theory on scalable queueing with stochastic traffic model.

Maybe so, but I still don't see the relation to MPLS. All MPLS can do is convey IPP or DSCP values as an EXP code point in the core. I'm not sure how that creates a scaling problem within MPLS itself. If you didn't have MPLS, you'd be encoding those values in IPP or DSCP. So what's the issue?

So, we have specification and working implementation of hop-by-hop, scalable, stable unicast/multicast interdomain QoS routing protocol supporting routing hierarchy without clank back.

See

    http://www.isoc.org/inet2000/cdproceedings/1c/1c_1.htm

for rough description of design guideline.

If I understand this correctly, would this be the IntServ QoS model?  

I didn't attempt to standardize our result in IETF, partly because optical packet switching was a lot more interesting.

Still is, even today :-)?

That should be a reasonable way of practical operation, though I'm not very interested in OCS (optical circuit switching) of GMPLS

Design goals are often what they are, and then the real world hits you.

For IP layer, that should be enough. For ASON, so complicated GMPLS is actually overkill.

When I was playing with ATM switches, I established control plain network with VPI/VCI=0/0 and assign control plain IP addresses to ATM switches. To control other VCs, simple UDP packets are sent to switches from controlling hosts.

Similar technology should be applicable to ASON. Maintaining integrity between wavelength switches is responsibility of controllers.

Well, GMPLS and ASON is basically skinny OSPF, IS-IS and RSVP running in a DWDM node's control plane.

No, I just explained what was advertised to be MPLS by people around Cisco against Ipsilon.

According to the advertisements, you should call what you are using LS or GLS, not MPLS or GMPLS.

It takes a while for new technology to be fully understood, which is why I'm not rushing on to the SR bandwagon :-). I can't blame the sales droids or the customers of the day. It probably sounded like dark magic.

Assuming a central controller (and its collocated or distributed back up controllers), we don't need complicated protocols in the network to maintain integrity of the entire network.

Well, that's a point of view, I suppose. I still can't walk into a shop and "buy a controller". I don't know what this controller thing is, 10 years on. IGP's, BGP and label distribution protocols have proven themselves, in the interim.

What if, an inner label becomes invalidated around the destination, which is hidden, for route scalability, from the equipments around the source?

I can't say I've ever come across that scenario running MPLS since 2004. Do you have an example from a production network that you can share with us? I'd really like to understand this better.

No, as "the destination expected to pop the label" is located somewhere around the final destination end-host.

If, at the destination site, connectivity between a router to pop nested label and the fine destination end-host is lost, we are at a loss, unless source side changes inner label.

Maybe a diagram would help, as I still don't get this failure scenario. If a host lost connectivity with the service provider network, getting label switching to work is pretty low on the priority list. Again, unless I misunderstand.

If you are using intra-domain hierarchical routing for scalability within the domain, you still suffer from lack of scalability of MPLS.

And, VRF is, in a sense, a form of intra-domain hierarchical routing with a lot of flexibility, which means a lot of unnecessary complications.

I don't think stuffing your VRF's full of routes is an intrinsic problem of MPLS. MPLS works whether you run l3vpn's or not. That MPLS provides a forwarding paradigm for VRF's does not put it and the potential poor scalability VRF's in the same WhatsApp group. Mark.

On 21/Jun/20 12:10, Masataka Ohta wrote:

  It was implemented and some technology was used by commercial router from Furukawa (a Japanese vendor selling optical fiber now not selling routers).

I won't lie, never heard of it.

GMPLS, you are using, is the mechanism to guarantee QoS by reserving wavelength resource. It is impossible for GMPLS not to offer QoS.

That is/was the idea. In practice (at least in our Transport network), deploying capacity as an offline exercise is significantly simpler. In such a case, we wouldn't use GMPLS for capacity reservation, just path re-computation in failure scenarios. Our Transport network isn't overly meshed. It's just stretchy. Perhaps if one was trying to build a DWDM backbone into, out of and through every city in the U.S., capacity reservation in GMPLS may be a use-case. But unless someone is willing to pipe up and confess to implementing it in this way, I've not heard of it.

Moreover, as some people says they offer QoS with MPLS, they should be using some prioritized queueing mechanisms, perhaps not poor WFQ.

It would be a combination - PQ and WFQ depending on the traffic type and how much customers want to pay. But carrying an MPLS EXP code point does not make MPLS unscalable. It's no different to carrying a DSCP or IPP code point in plain IP. Or even an 802.1p code point in Ethernet.

They are different, of course. But, GMPLS is to reserve bandwidth resource.

In theory. What are people doing in practice? I just told you our story.

MPLS, in general, is to reserve label values, at least.

MPLS is the forwarding paradigm. Label reservation/allocation can be done manually or with a label distribution protocol. MPLS doesn't care how labels are generated and learned. It will just push, swap and pop as it needs to.

I didn't say scaling problem caused by QoS.

But, as you are avoiding to extensively use MPLS, I think you are aware that extensive use of MPLS needs management of a lot of labels, which does not scale.

Or, do I misunderstand something?

I'm not avoiding extensive use of MPLS. I want extensive use of MPLS. In IPv4, we forward in MPLS 100%. In IPv6, we forward in MPLS 80%. This is due to vendor nonsense. Trying to fix.

No. IntServ specifies format to carry QoS specification in RSVP packets without assuming any specific model of QoS.

Then I'm failing to understand your point, especially since it doesn't sound like any operator is deploying such a model, or if so, publicly suffering from it.

No. As experimental switches are working years ago and making it work >10Tbps is not difficult (switching is easy, generating 10Tbps packets needs a lot of parallel equipment), there is little remaining for research.

We'll get there. This doesn't worry me so much :-). Either horizontally or vertically. I can see a few models to scale IP/MPLS carriage.

    SDN, maybe. Though I'm not saying SDN scale, it should be no worse than MPLS.

I still can't tell you what SDN is :-). I won't suffer it in this decade, thankfully.

I did some retrospective research.

   https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching    History    1994: Toshiba presented Cell Switch Router (CSR) ideas to IETF BOF    1996: Ipsilon, Cisco and IBM announced label switching plans    1997: Formation of the IETF MPLS working group    1999: First MPLS VPN (L3VPN) and TE deployments    2000: MPLS traffic engineering    2001: First MPLS Request for Comments (RFCs) released

as I was a co-chair of 1994 BOF and my knowledge on MPLS is mostly on 1997 ID:

   https://tools.ietf.org/html/draft-ietf-mpls-arch-00

there seems to be a lot of terminology changes.

My comment to that was in reference to your text, below:     "What if, an inner label becomes invalidated around the     destination, which is hidden, for route scalability,     from the equipments around the source?" I've never heard of such an issue in 16 years.

I'm saying that, if some failure occurs and IGP changes, a lot of LSPs must be recomputed, which does not scale if # of LSPs is large, especially in a large network where IGP needs hierarchy (such as OSPF area).

That happens everyday, already. Links fail, IGP re-converges, LDP keeps humming. RSVP-TE too, albeit all that state does need some consideration especially if code is buggy. Particularly, where you have LFA/IP-FRR both in the IGP and LDP, I've not come across any issue where IGP re-convergence caused LSP's to fail. In practice, IGP hierarchy (OSPF Areas or IS-IS Levels) doesn't help much if you are running MPLS. FEC's are forged against /32 and /128 addresses. Yes, as with everything else, it's a trade-off. Mark.

On 21/Jun/20 15:48, Robert Raszuk wrote:

Actually when IGP changes LSPs are not recomputed with LDP or SR-MPLS (when used without TE :). 

"LSP" term is perhaps what drives your confusion --- in LDP MPLS there is no "Path" - in spite of the acronym (Labeled Switch *Path*). Labels are locally significant and swapped at each LSR - resulting essentially with a bunch of one hop crossconnects. 

In other words MPLS LDP strictly follows IGP SPT at each LSR hop.

Yep, which is what I tried to explain as well. With LDP, MPLS-enabled hosts simply push, swap and pop. There is not concept of an "end-to-end LSP" as such. We just use the term "LSP" to define an FEC. But really, each node in the FEC's path is making its own push, swap and pop decisions. The LFIB in each node need only be as large as the number of LDP-enabled routers in the network. You can get scenarios where FEC's are also created for infrastructure links, but if you employ filtering to save on FIB slots, you really just need to allocate labels to Loopback addresses only. Mark.

On 21/Jun/20 19:34, Robert Raszuk wrote:

That is true for P routers ... not so much for PEs. 

Please observe that label space in each PE router is divided for IGP and BGP as well as other label hungy services ... there are many consumers of local label block. 

So it is always the case that LFIB table (max 2^20 entries - 1M) on PEs is much larger then LFIB on P nodes.

I should point out that all of my input here is based on simple MPLS forwarding of IP traffic in the global table. In this scenario, labels are only assigned to BGP next-hops, which is typically an IGP Loopback address. Labels don't get assigned to BGP routes in a global table. There is no use for that. Of course, as this is needed in VRF's and other BGP-based VPN services, the extra premium customers pay for that priviledge may be considered warranted :-). Mark.

On 21/Jun/20 22:21, Robert Raszuk wrote:

Well this is true for one company :) Name starts with j .... 

Other company name starting with c - at least some time back by default allocated labels for all routes in the RIB either connected or static or sourced from IGP. Sure you could always limit that with a knob if desired.

Juniper allocates labels to the Loopback only. Cisco allocates labels to all IGP and interface routes. Neither allocate labels to BGP routes for the global table.

The issue with allocating labels only for BGP next hops is that your IP/MPLS LFA breaks (or more directly is not possible) as you do not have a label to PQ node upon failure.  Hint: PQ node is not even running BGP :).

Wouldn't T-LDP fix this, since LDP LFA is a targeted session? Need to test.

Sure selective folks still count of "IGP Convergence" to restore connectivity. But I hope those will move to much faster connectivity restoration techniques soon.

We are happy :-). Mark.

On 21/Jun/20 23:01, Robert Raszuk wrote:

Nope. You need to get to PQ node via potentially many hops. So you need to have even ordered or independent label distribution to its loopback in place.

I have some testing I want to do with IS-IS only announcing the Loopback from a set of routers to the rest of the backbone, and LDP allocating labels for it accordingly, to solve a particular problem. I'll test this out and see what happens re: LDP LFA. Mark.

On 19/Jun/20 17:13, Robert Raszuk wrote:

So I think Ohta-san's point is about scalability services not flat underlay RIB and FIB sizes. Many years ago we had requests to support 5M L3VPN routes while underlay was just 500K IPv4.

Ah, if the context, then, was l3vpn scaling, yes, that is a known issue. Apart from the global table vs. VRF parity concerns I've always had (one of which was illustrated earlier this week, on this list, with RPKI in a VRF), the other reason I don't do Internet in a VRF is because it was always a trade-off:     - More routes per VRF = fewer VRF's.     - More VRF's  = fewer routes per VRF. Going forward, I believe the l3vpn pressures (for pure VPN services, not Internet in a VRF) should begin to subside as businesses move on-prem workloads to the cloud, bite into the SD-WAN train, and generally, do more stuff over the public Internet than via inter-branch WAN links formerly driven by l3vpn. Time will tell, but in Africa, bar South Africa, l3vpn's were never a big thing, mostly because Internet connectivity was best served from one or two major cities, where most businesses had a branch that warranted connectivity. But even in South Africa (as the rest of our African market), 98% of our business is plain IP. The other 2% is mostly l2vpn. l3vpn's don't really feature, except for some in-house enterprise VoIP carriage + some high-speed in-band management. Even with the older South African operators that made a killing off l3vpn's, these are falling away as their customers either move to the cloud and/or accept SD-WAN thingies.

Last - when I originally discussed just plain MPLS with customers with single application of hierarchical routing (no BGP in the core) frankly no one was interested. Till L3VPN arrived which was game changer and run for new revenue streams ...

The BGP-free core has always sounded like a dark art. More so in the days when hardware was precious, core routers doubled as inline route reflectors and the size of the IPv4 DFZ wasn't rapidly exploding like it is today, and no one was even talking about the IPv6 DFZ. Might be useful speaking with them again, in 2020 :-). Mark.

On 21/Jun/20 21:15, adamv0025@netconsultings.com wrote:

I wouldn't say it's known to many as not many folks are actually limited by only up to ~1M customer connections, or next level up, only up to ~1M customer VPNs.

It's probably less of a problem now than it was 10 years ago. But, yes, I don't have any real-world experience.

Well yeah, things work differently in VRFs, not a big surprise. And what about an example of bad flowspec routes/filters cutting the boxes off net -where having those flowspec routes/filters contained within an Internet VRF would not have such an effect. See, it goes either way. Would be interesting to see a comparison of good vs bad for the Internet routes in VRF vs in Internet routes in global/default routing table.

Well, the global table is the basics, and VRF's is where sexy lives :-).

No, that's just a result of having a finite FIB/RIB size -if you want to cut these resources into virtual pieces you'll naturally get your equations above. But if you actually construct your testing to showcase the delta between how much FIB/RIB space is taken by x prefixes with each in a VRF as opposed to all in a single default VRF (global routing table) the delta is negligible. (Yes negligible even in case of per prefix VPN label allocation method -which I'm assuming no one is using anyways as it inherently doesn't scale and would limit you to ~1M VPN prefixes though per-CE/per-next-hop VPN label allocation method gives one the same functionality as per-prefix one while pushing the limit to ~1M PE-CE links/IFLs which from my experience is sufficient for most folks out there).

Like I said, with today's CPU's and memory, probably not an issue. But it's not an area I play in, so those with more experience - like yourself - would know better. Mark.

On 19/Jun/20 17:40, Masataka Ohta wrote:  

As the first person to have proposed the forwarding paradigm of label switching, I have been fully aware from the beginning that:

   https://tools.ietf.org/html/draft-ohta-ip-over-atm-01

   Conventional Communication over ATM in a Internetwork Layer

   The conventional communication, that is communication that does not    assume connectivity, is no different from that of the existing IP, of    course.

special, prioritized forwarding should be done only by special request by end users (by properly designed signaling mechanism, for which RSVP failed to be) or administration does not scale.

I could be wrong, but I get the feeling that you are speaking about RSVP in its original form, where hosts were meant to make calls (CAC) into the network to reserve resources on their behalf. As we all know, that never took off, even though I saw some ideas about it being proposed for mobile phones as well. I don't think there ever was another attempt to get hosts to reserve resources within the network, since the RSVP failure.

Not. Even without MPLS, fine tuning of BGP does not scale.

We all know this, and like I said, that is a current concern.

However, just as using plain IP router costs less than using MPLS capable IP routers, BGP-only administration costs less than BGP and MPLS administration.

For better networking infrastructure, extra cost should be spent for L1, not MPLS or very complicated technologies around it.

In the early 2000's, I would have agreed with that. Nowadays, there is a very good chance that a box you require a BGP DFZ on inherently supports MPLS, likely without extra licensing. Mark.