On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote:

...

In short, the idea is that optical networks are wasteful and routers do a better job making more use of a network's capacity than ROADMs. Take the extra router hop (or 3 or 8) versus short-cutting it with an optical network because the silicon is so low-latency anyway that it hardly makes a difference now. Putting more GBs per second on fewer strands means saving a lot of money on infrastructure costs.

This is a very convoluted way of backing into the ole packet-switched vs. circuit switched decision.

I don't follow. While ROADMs can be thought of as circuit-switchers, the number of concurrent clients and switching latency put ROADMs on a different operational level than packet switchers, right? Cheers, Etienne On Tue, May 2, 2023 at 4:29 PM Izaac <izaac@setec.org> wrote:

On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote:

...

In short, the idea is that optical networks are wasteful and routers do a better job making more use of a network's capacity than ROADMs. Take the extra router hop (or 3 or 8) versus short-cutting it with an optical network because the silicon is so low-latency anyway that it hardly makes a difference now. Putting more GBs per second on fewer strands means saving a lot of money on infrastructure costs.

This is a very convoluted way of backing into the ole packet-switched vs. circuit switched decision.

-- . ___ ___ . . ___ . \ / |\ |\ \ . _\_ /__ |-\ |-\ \__

-- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

So right Jared....magic has been in the NPU capacity increase that's driven the cost per 100G down on 1RU routers; and the integration of DSPs and more into QSFP-DD form factors at much lower power than expected. The standards for optical links are maturing as well, but we still have work to do on the management side for the electrical interfaces. Eve On Tue, May 2, 2023 at 3:33 PM Jared Mauch <jared@puck.nether.net> wrote:

...

On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG < nanog@nanog.org> wrote:

...

In short, the idea is that optical networks are wasteful and routers do a better job making more use of a network's capacity than ROADMs. Take

On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote: the

...

extra router hop (or 3 or 8) versus short-cutting it with an optical network because the silicon is so low-latency anyway that it hardly makes a difference now. Putting more GBs per second on fewer strands means saving a lot of money on infrastructure costs.

This is a very convoluted way of backing into the ole packet-switched vs. circuit switched decision.

I don't follow. While ROADMs can be thought of as circuit-switchers, the number of concurrent clients and switching latency put ROADMs on a different operational level than packet switchers, right?

I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that.

I know of a few companies that have looked at solutions like this, and can expect there to be some interesting solutions that would appear as a result. Optical line systems tend to have pretty low power requirements compared to a router, but some of the routers are getting pretty low power as well when it comes to the power OPEX/bit, and if you have the ability to deliver services as an integrated packet optical you could see reduced costs and simplified components/sparing.

I’ll also say that I’ve not yet seen the price compression that I had expected in the space yet, but I figure that’s coming. We are seeing the bits/watt ratio improve though, so for the same or less power consumption you get more bits. Some of this technology stuff is truly magical.

- Jared

I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that.

Jared, I understand your point in the above statement to be that directionality is cost-effectively implemented through label-switched paths, rather than (ROADM-enabled) optical path switching. Do I understand right? Thank you. Etienne On Tue, May 2, 2023 at 9:33 PM Jared Mauch <jared@puck.nether.net> wrote:

...

On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG < nanog@nanog.org> wrote:

...

In short, the idea is that optical networks are wasteful and routers do a better job making more use of a network's capacity than ROADMs. Take

On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote: the

...

extra router hop (or 3 or 8) versus short-cutting it with an optical network because the silicon is so low-latency anyway that it hardly makes a difference now. Putting more GBs per second on fewer strands means saving a lot of money on infrastructure costs.

This is a very convoluted way of backing into the ole packet-switched vs. circuit switched decision.

I don't follow. While ROADMs can be thought of as circuit-switchers, the number of concurrent clients and switching latency put ROADMs on a different operational level than packet switchers, right?

I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that.

I know of a few companies that have looked at solutions like this, and can expect there to be some interesting solutions that would appear as a result. Optical line systems tend to have pretty low power requirements compared to a router, but some of the routers are getting pretty low power as well when it comes to the power OPEX/bit, and if you have the ability to deliver services as an integrated packet optical you could see reduced costs and simplified components/sparing.

I’ll also say that I’ve not yet seen the price compression that I had expected in the space yet, but I figure that’s coming. We are seeing the bits/watt ratio improve though, so for the same or less power consumption you get more bits. Some of this technology stuff is truly magical.

- Jared

-- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

On 5/2/23 21:32, Jared Mauch wrote:

I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device.

My memory is rather fuzzy, but didn't Juniper attempt something like this in their PTX's after they picked up BTI? I think the plan was to co-locate the ROADM at the bottom of the PTX chassis, or something along those lines. I know Cisco (and Juniper) tried by integrating GMPLS into their code as a starting point, but that didn't go very far with customers. It just seemed impossible for the Transport teams to allow the IP/MPLS teams that level of access into their line system :-).

As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that.

A rather high barrier to entry for most operators, but we have to start from somewhere.

I know of a few companies that have looked at solutions like this, and can expect there to be some interesting solutions that would appear as a result. Optical line systems tend to have pretty low power requirements compared to a router, but some of the routers are getting pretty low power as well when it comes to the power OPEX/bit, and if you have the ability to deliver services as an integrated packet optical you could see reduced costs and simplified components/sparing.

The main problem is distance. If you need to move that kind of capacity more than 50km, it's hard to avoid DWDM.

I’ll also say that I’ve not yet seen the price compression that I had expected in the space yet, but I figure that’s coming. We are seeing the bits/watt ratio improve though, so for the same or less power consumption you get more bits. Some of this technology stuff is truly magical.

I think for long spans, DWDM will not only be cheaper, but the only feasible solution. For the metro, it will come down to what motivates the business... plenty of features, or plenty of speed. Also, DWDM vendors are adding speed and distance faster and cheaper than the IP/MPLS vendors can. So they will always be one step ahead in that respect; and we have the submarine cable systems to thank for that. Mark.

Very helpful observations, Matt, thank you. How comfortably does the phrase "routed optical networks over Ethernet without ROADMs" sit with you? I mean: would you accept a limitation of "optical network" to the case of a network without optical layer switching (of the type done by add-drop multiplexers)? Cheers, Etienne On Mon, May 1, 2023 at 10:57 PM Matt Erculiani <merculiani@gmail.com> wrote:

Hi Etienne

In short, the idea is that optical networks are wasteful and routers do a better job making more use of a network's capacity than ROADMs. Take the extra router hop (or 3 or 8) versus short-cutting it with an optical network because the silicon is so low-latency anyway that it hardly makes a difference now. Putting more GBs per second on fewer strands means saving a lot of money on infrastructure costs.

400G ZR comes to mind as a foundational technology since it basically made active optical muxponder equipment obsolete in the metro. The savings here means telcos/enterprises can afford more router ports, which we've already established can utilize paths more efficiently anyway. Otherwise, this is more of a concept and can be executed with a variety of pre-existing technologies, or someone's new secret sauce that bakes everything together like SD-WAN did to its constituent technologies.

-Matt

On Mon, May 1, 2023 at 12:30 PM Etienne-Victor Depasquale via NANOG < nanog@nanog.org> wrote:

...

Hello folks,

Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not?

Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks?

I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey.

Cheers,

Etienne

-- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

-- Matt Erculiani ERCUL-ARIN

-- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

On 5/2/23 07:28, Vasilenko Eduard via NANOG wrote:

The incumbent carrier typically has enough fiber strands to avoid any colored interfaces (that are 3x expensive compare to gray) in the Metro.

Metro ring typically has 8-10 nodes (or similar). 16-20 strands of fiber were not possible to construct anyway – any cable is bigger.

It is the same cost to lay down fiber on 16 strands or 32.

Hence, PTT just does not need DWDM in Metro, not at all. Hence, the DWDM optimization that you are talking about below is not needed too.

This may or may not always be the case. Especially for large carriers, where there could be a requirement to sell some of those dark fibre pairs to large customers (think the content folk coming into town, e.t.c.), they may no longer have the priviledge of having plenty of free fibre in the metro. Or if they did, the rate of traffic expansion means they burn through those fibre pairs pretty quick. 10Gbps isn't a lot nowadays, and 100Gbps may start to push the limits depending on the size of the operator, the scope of the Metro-E ring and the level of service that needs to be maintained during a re-route (two available paths in the ring could balance 100Gbps of traffic, but if one half of that ring breaks, the remaining path may need to carry a lot more than 100Gbps, and then packets start to fall flat on the floor). At that scale, DWDM in the metro will make sense, at least more sense than 400G-ZR, at the moment.

If you rent a single pair of fiber then you need colored interfaces to multiplex 8-10 nodes into 1 pair on the ring.

Then the movement of transponders from DWDM into the router would eliminate 2 gray interfaces on every node (4 per link): one on the router side, and another on the DWDM side.

Overall, it is about a 25% cost cut of the whole “router+DWDM”.

Some operators would also be selling Transport services in or along the metro, and customers paying for that may require that they do not cross a router device.

It is still 2x more expensive compare to using additional fiber strands on YOUR fiber.

There are plenty of DWDM pizza boxes that cost next to nothing. At scale, the price of these is not a stumbling block. And certainly, the price of these would be far lower than a router line card.

By the way, about “well-defined stack of technologies”:

NMS (polished by SDN our days) should be cross-layer: it should manage at the same time: ROADM/OADM in DWDM and colored laser in Router.

It is a vendor lock up to now (no multi-vendor). Hence, 25% cost savings would go to the vendor that has such NMS, not to the carrier.

Technology still does not make sense because no multivendor support between the NMS of one vendor and the router or DWDM of another.

Looking at the NMS history, it would probably never be multi-vendor. For that reason, I am pessimistic about the future of the colored interfaces in routers (and alien lambdas in DWDM). Despite a potential 25% cost advantage in eliminating gray interfaces.

OpenROADM is a good initiative. But it seems it's to be to Transport equipment vendors what IPv6 and DNSSEC is to the IP world :-). Mark.

On 5/3/23 08:20, Vasilenko Eduard wrote:

I would risk to say a little more on this.

Indeed, maybe the situation (in many countries) when the Carrier sells a lot of TDM services.

But in general, packet services are enough these days for many carriers/regions.

There aren't enough TDM services to warrant DWDM, nowadays. The reason for DWDM is mainly being driven by Ethernet, and IP. At any reasonable scale, it's actually pretty hard to buy a TDM service, in most markets.

Additionally, I am sure that in many countries/Metro it is cheaper to lay down a new fiber than to provision DWDM, even if it is a pizza box.

I disagree. Existing fibre may be cheap because it was laid down a decade or more ago, en masse, by several operators. So the market would be experiencing a glut, not because it is cheap to open up the roads and plant more fibre, but because there is so much of it to begin with. At worst, there is still enough duct space that the operator can blow more fibre. But when that duct gets full, and there are no more free ducts available, or another route needs to get built for whatever reason, it is a rather costly affair to open up the roads and trunk some fibre, in any market. So no, DWDM is not more expensive, if you are delivering services at scale. It is actually cheaper. It is only more expensive if you are small scale, because in some markets, the fibre glut means you can buy dark fibre for cheaper than you can light it with DWDM. But this is a situation unique to small operators, not large ones.

The colored interface is still very expensive.

This only matters for the line side. For client-facing, it's not a drama. And you typically buy more optics for the client side than you do the line side.

Of course, there are some Cities (not “towns”) where it is very expensive or maybe even impossible to lay down a new fiber.

Yes, in the majority of cases, it is cheaper to lay down fiber.

I think what you mean to say is that in the majority of cases where there is fibre glut, and dark fibre is a market option, buying fibre is cheaper than lighting it with DWDM. This is true. But I think that on a global scale, this is the exception, not the rule. In general, you are not likely to be able to buy dark fibre, cheaply or otherwise, if you look at all markets in the world.

Hence, the importance of DWDM for the Metro is overestimated.

Again, only if you are small scale. If you are a large scale operator with as many IP/Ethernet customers as you have Transport, DWDM is essential.

Use only routers. Provision enough fiber. Have always 1 router hop to the aggregation (hub-spoke topology), no routers chaining in the ring.

If fiber is not enough – then use normal DWDM with an external transponder. Routers would be still in hub-spoke topology.

Yeah, you sound like an equipment vendor whose main customers are incumbent telco's in a few rich markets :-). The life of the average operator, around the world, is far less glamorous. Mark.

On 5/4/23 11:40, Denis Fondras wrote:

You may also take into account the time to deliver. Laying fiber takes much more time than plugging a colored optic.

Indeed - part of the expense of running new fibre is the time it takes to start making money from it. Mark.

On 5/4/23 15:03, Jared Mauch wrote:

I’m a bit shocked that I now need a 288F cable on some of my routes to support future expansion, but that fiber cost is still small compared to the labor.

Yes, labour is generally the cost. And then way-leaves add cost in terms of time and lost opportunities. Mark.

On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote:

Well, ISP is typically plan something for a year. It is more than enough for both.

The real world is much less certain, especially in these economic times.

Funny, that with the current lead times for electronics, Fiber could be faster. Of course, it is a temporary glitch.

Even in the same market, no two lays of fibre can be guaranteed to be completed in the same time. Mark.

Many Big PTTs have a lot of ducts in many places, it is easy for them to lay fibers, especially in cities. Ge

Le 4 mai 2023 à 14:27, Vasilenko Eduard via NANOG <nanog@nanog.org> a écrit :

I had an experience in one big PTT. Fiber was easy in the majority of Metro places. Even faster than DWDM or router commissioning. It is just 1 PTT. Hence, an example could not be counted. Eduard -----Original Message----- From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei.com@nanog.org] On Behalf Of Mark Tinka Sent: Thursday, May 4, 2023 2:11 PM To: nanog@nanog.org Subject: Re: Routed optical networks

On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote:

...

Well, ISP is typically plan something for a year. It is more than enough for both.

The real world is much less certain, especially in these economic times.

...

Funny, that with the current lead times for electronics, Fiber could be faster. Of course, it is a temporary glitch.

Even in the same market, no two lays of fibre can be guaranteed to be completed in the same time.

Mark.

Anyone publicly traded doesn't plan longer than the current quarter. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mark Tinka" <mark@tinka.africa> To: nanog@nanog.org Sent: Thursday, May 4, 2023 6:10:36 AM Subject: Re: Routed optical networks On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote:

Well, ISP is typically plan something for a year. It is more than enough for both.

The real world is much less certain, especially in these economic times.

Funny, that with the current lead times for electronics, Fiber could be faster. Of course, it is a temporary glitch.

Even in the same market, no two lays of fibre can be guaranteed to be completed in the same time. Mark.

Well, ISP is typically plan something for a year. It is more than enough for both.

s/more/should be/ The economics are such these days that in many circumstances, bean counters don't want to hear about payback in years, they want to hear it in quarters. Short term financial thinking is dominant. On Thu, May 4, 2023 at 6:59 AM Vasilenko Eduard via NANOG <nanog@nanog.org> wrote:

Well, ISP is typically plan something for a year. It is more than enough for both.

Funny, that with the current lead times for electronics, Fiber could be faster. Of course, it is a temporary glitch.

Ed/ -----Original Message----- From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei.com@nanog.org] On Behalf Of Denis Fondras Sent: Thursday, May 4, 2023 12:41 PM To: nanog@nanog.org Subject: Re: Routed optical networks

Le Wed, May 03, 2023 at 06:20:48AM +0000, Vasilenko Eduard via NANOG a écrit :

...

Additionally, I am sure that in many countries/Metro it is cheaper to

lay down a new fiber than to provision DWDM, even if it is a pizza box. The colored interface is still very expensive.

...

Of course, there are some Cities (not “towns”) where it is very expensive or maybe even impossible to lay down a new fiber. Yes, in the majority of cases, it is cheaper to lay down fiber.

You may also take into account the time to deliver. Laying fiber takes much more time than plugging a colored optic.

On 5/2/23 16:01, Eve Griliches wrote:

Hi Etienne, Below is our (Cisco) definition of the Routed Optical Network. The goal, metro or long haul or subsea, is to reduce the number of control planes. By migration TDM traffic using CEM or PLE to the IP layer, you eliminate the OTN control plane and management. Eventually, when standards are settled the ultimate goal is to have a single control plane for the network. I'm not trying to be a commercial here, but you can read more in the resources section on this page: https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-netw... HTH, Eve

Routed optical networking, is an architecture that delivers improved operational efficiencies and simplicity. The solution works by merging IP and private line services onto a single layer where all the switching is done at Layer 3. Routers are connected with standardized 400G ZR/ZR+ coherent pluggable optics.

With a single service layer based upon IP, flexible management tools can leverage telemetry and model-driven programmability to streamline lifecycle operations. This simplified architecture integrates open data models and standard APIs, enabling a provider to focus on automation initiatives for a simpler topology.

To be honest, I've been hearing about this since as long as I can remember. IPoDWDM was another attempt at trying to make the above a reality. But for some reason, operators prefer to keep these networks separate, and many customers, especially very large ones, prefer to bypass routers for their Transport services. I think the effort will be appreciated, but if history is anything to go by, vendors are going to struggle to strip operators and customers away from some degree of separation. Mark.

On 5/4/23 19:32, Phil Bedard wrote:

It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks.  There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit.

I think every time the IP space gets close to running an all-packet network, the Transport folk come out with an easier way to do it, that it's too hard to ignore. Based on that, I think they will always be one step ahead, with the key advantage being reliability of capacity over the distance, for the cost. The farther your fibre has to run, the costlier it gets to do it without DWDM. I mean, it's only now that 100G-ZR is becoming a reality for packet networks, and we are talking thousands of US$ for optics to get us 80km - 120km distance. Meanwhile, DWDM vendors can get you 800Gbps per wavelength in the same distance (or 30X that distance) far less cheaply. I get the appeal of not needing DWDM gear to underlay your packet network... it's neater and offers fewer points of failure. But unless you are dealing with very short distances and can ride a reasonable balance between service features and switching/forwarding capacity in your router/switch, it's going to be hard to ignore the DWDM gear if you are trying to be a serious operation, at that scale, over a wide area. Mark.

On 5/8/23 21:53, Phil Bedard wrote:

I guess let’s not confuse two things.  The optical network is made up of the photonic portion and then the transponder/muxponder portion.   A single term like “DWDM” can be confusing since it can refer to both.

Indeed. I am short-handing to mean DWDM on the line side and grey on the client side.

   It will take a long time (maybe never) to remove the photonic switching part of the network.

There was a time when my intention was to do just that. But that was prior to expecting to ever run links larger than 10Gbps :-).

  However, it’s always been cheap to deploy because optical vendors tended to subsidize that network using sales of the other portion, the transponders, which you buy more and more over time.  Those photonic components are expensive.

It depends on the application... either you want performance, space and power cost optimization or functional integration of previously disparate system-level features. This will determine whether you focus on pluggables or embeddeds, with pluggables promoting cost reduction, while embeddeds push performance.

On the DWDM signal portion, I’m not talking about 100ZR compared to 100G on a transponder or DWDM line system.  100ZR has had to deal with the power limitations of QSFP28 ports, which QDD/OSFP do not suffer from.

Right - 100ZR is short-reach (80km). It's really for the metro edge where 400Gbps is not needed.

  There are quite a few QDD pluggables in production today capable of supporting 100G signals over 1000s of km or 400G near 1500km.

I think the more interesting QSPF-DD (or OSFP) development is OpenZR+, which is an MSA project to standardize 400ZR+. The plan is to be able to support 100Gbps up to 5,800km, and 400Gbps up to 480km (EDFA) or about 1,000km (EDFA + Raman). With a 4x 100Gbps mode supported on QSFP28 router ports, you can have one muxponder talking to 4x routers at 100Gbps each.

  Now that’s not what you can get out of some external transponders, so those will still have their place in high performance applications.   When you move to 800G, 1.2Tbps single channel they also have their own distance limitations.  So it really depends on the application and the network.

800Gbps and 1.2Tbps applications are really for long haul use-cases, especially if you used 400Gbps pluggables before and run out of distance (so less than 1,000km). They are also preferred for submarine use-cases. I can't wait to see what happens when the CMOS gets down to 5nm and 3nm. Mark.

On 5/9/23 20:37, Phil Bedard wrote:

[phil] These are already available today and have been for some time and in use in production networks for over a year now.  This is with 400G links running up to 600km in routers with QDD ports.  400G-16QAM using 60Gbaud (the OpenZR+ standard) can reach around 1300km.   These optics are being used in both routers and xponders on the line side.

Right... I've not been keeping in touch with this development on the terrestrial side, as our focus is mainly on the submarine end of things. But this is good to know. In our case, the majority of our customers are still seeking 10Gbps and 100Gbps services over long distances (so 200km and over). Neither they (nor us) have a requirement for 400Gbps in the metro. That said, we do have the capability to do this on terrestrial spans where we are able to carry 600Gbps per channel across 700km or so. We are looking forward to the next generation of tech. with our vendor that could get us to about 700Gbps per channel for the same distance.

[phil] 800Gbps and 1.2Tbps are not really positioned for long haul use cases yet.  It requires lowering the modulation to something like QPSK which at those speeds requires baud rates which are not yet commercially available.

I meant more in terms of the capabilities of the 800G/1.2T embedded option to improve spectral efficiency for more capacity over long haul use-cases, and not necessarily that they can actually get to 800Gbps or 1.2Tbps for that length of span. The next generation of 5nm CMOS with the potential for close to 150 Gbaud is quite intriguing, although I'm not expecting anything more than 15% - 18% improvement in performance compared to the current 7nm 100 Gbaud systems. Will keep you posted, especially on our submarine application of this. Mark.

Disclaimer: Metaverse has not changed Metro traffic yet. Then ... I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one "2" for redundancy, another "2" for capacity) in the majority of cases worldwide. How many BRASes serve more than 40000/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits - it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change - it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei.com@nanog.org] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale <edepa@ieee.org>; NANOG <nanog@nanog.org> Subject: Re: Routed optical networks It's not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that's not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that's over a ROADM based optical network or not, especially at metro/regional distances. There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn't exceed the capabilities of those optics. It's been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment. It's my personal opinion we aren't to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren't a good fit. Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices. Thanks, Phil From: NANOG <nanog-bounces+bedard.phil=gmail.com@nanog.org<mailto:nanog-bounces+bedard.phil=gmail.com@nanog.org>> on behalf of Etienne-Victor Depasquale via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Date: Monday, May 1, 2023 at 2:30 PM To: NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

You are right, my “Metro” definition is about ISP/Carriers. Mobile or Fixed, despite pure Mobiles would like to call it MBH – it has much less traffic (Mobile subscribers would always have 7x less than fixed). It is still the place where the majority of port capacity lives. Because all content and cache are after this link. 50GE is better just because it is half of the cost of 100GE and it is enough now for the great majority of cases. Money is very important these days for this industry. 100GE single mode is more expensive than the best router port itself. Routers have been deprecated 10x for the decade (almost 100x for 2 decades). Pluggable optics is not that much deprecated. I do not think that content provider guys call their DCI “Metro”, not very often. I agree that 100GE for DCI is the minimum, 400GE is probably already needed in some places. IMHO: it is a different story. Very interested too. PS: By the way, even if some ISP has 50% of revenue from Enterprise services (it is probably the biggest number, typically 30%-40%), it is still just 5% compare to residential traffic. Traffic to enterprises is still sold 4x-10x (depending on the country). Hence, Enterprise does not make sense to mention in the traffic discussion. It is a “rounding error”. Enterprise business created a huge demand for oversubscribed ports to connect Enterprises. And QoS/QoE. Not traffic. Eduard From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei.com@nanog.org] On Behalf Of Mark Tinka Sent: Friday, May 5, 2023 11:13 AM To: nanog@nanog.org Subject: Re: Routed optical networks On 5/5/23 07:57, Vasilenko Eduard via NANOG wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 40000/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. So, it depends on what "metro" means to you. For an ISP selling connectivity to enterprise customers, it can be a bunch of Metro-E routers deployed in various commercial buildings within a city. For a content provider, it could be DCI. For a telco, it could interconnecting their Active-E/GPON/DSLAM/CMTS network. Whatever the case, the need for 100Gbps is going to be driven by the cost of optics over the distance required. Some operators run 2x 10Gbps for resilience/redundancy, while some others run 4x 10Gbps for the same. It all depends on the platform you are using. At some point, that capacity runs out, especially when you account for fibre outages, and you need something larger on one side of the ring mainly to provide sufficient bandwidth during failure events on the other side of the ring, and not necessarily because you are growing by that much. Also, if the optics are available and are reasonably priced, why muck around with 40Gbps when you can just go straight to 100Gbps? The equipment usually can support either. I'm unaware of any popularity around 50Gbps interfaces, but I also probably don't pay too much attention to such nuance :-). So, it's not that we are seeing organic growth that justifies 100Gbps over anything smaller. It's more that the optics are available, they are cheap, they can go the distance, and the routers/switches can do the speed. At least, for us anyway, that is what is driving the next phase of our Metro-E network... going straight from 10Gbps to 100Gbps links, not because that is the growth we are seeing from an organic traffic standpoint, but because the routers can do it, and it offers us peace of mind that we can handle any traffic re-route when one half of the ring fails, without dropping packets. The 400Gbps market will be restricted to mainly content folk linking up data centres, as well as some large telco's, for the time being. It is not likely to be the norm for the majority of operators who run some kind of metro network. Mark.

Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG <nanog@nanog.org> wrote:

Disclaimer: Metaverse has not changed Metro traffic yet. Then …

I am puzzled when people talk about 400GE and Tbps in the Mero context.

For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide.

How many BRASes serve more than 40000/1.5=27k users in the busy hour?

It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth.

Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful).

We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics.

Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau.

How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research.

IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS)

Additionally, who would pay for this traffic growth? It also limits traffic at some point.

I hope it would happen after we would get our 22h/4k/12bit/120hz.

Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed.

Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth.

Ed/

*From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei.com@nanog.org] *On Behalf Of *Phil Bedard *Sent:* Thursday, May 4, 2023 8:32 PM *To:* Etienne-Victor Depasquale <edepa@ieee.org>; NANOG <nanog@nanog.org> *Subject:* Re: Routed optical networks

It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations.

The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based optical network or not, especially at metro/regional distances.

There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn’t exceed the capabilities of those optics. It’s been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment.

It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit.

Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices.

Thanks,

Phil

*From: *NANOG <nanog-bounces+bedard.phil=gmail.com@nanog.org> on behalf of Etienne-Victor Depasquale via NANOG <nanog@nanog.org> *Date: *Monday, May 1, 2023 at 2:30 PM *To: *NANOG <nanog@nanog.org> *Subject: *Routed optical networks

Hello folks,

Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not?

Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks?

I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey.

Cheers,

Etienne

--

Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta

Web. https://www.um.edu.mt/profile/etiennedepasquale

-- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos

But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates YYYY traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.taht@gmail.com] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com> Cc: Phil Bedard <bedard.phil@gmail.com>; Etienne-Victor Depasquale <edepa@ieee.org>; NANOG <nanog@nanog.org> Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 40000/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei.com@nanog.org<mailto:huawei.com@nanog.org>] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based optical network or not, especially at metro/regional distances. There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn’t exceed the capabilities of those optics. It’s been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment. It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit. Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices. Thanks, Phil From: NANOG <nanog-bounces+bedard.phil=gmail.com@nanog.org<mailto:nanog-bounces+bedard.phil=gmail.com@nanog.org>> on behalf of Etienne-Victor Depasquale via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Date: Monday, May 1, 2023 at 2:30 PM To: NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos

I did investigate traffic for every Carrier while dealing with it as a consultant (repeated many dozens of times). I have seen over a decade how traffic growth dropped year-over-year (from 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). Sometimes I talk to old connections and they confirm that it is even less now. In rear cases, It is typically possible to find this information on the public Internet (I remember the case when Google disclosed traffic for Pakistan at the conference with the explanation that 30% is attributed to new subscribers, and an additional +30% is to more heavy content per subscriber). But mostly, it was confidential information from a discussion with Carriers – they all know very well their traffic growth. In general, traffic stat is pretty confidential. I did not have the motivation to aggregate it. Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future. Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it. VNI is claiming so many things. Please show where exactly they show traffic growth (I am not interested in prediction speculations). Is it possible to understand CAGR for the 5 last years? Is it declining or growing? (traffic itself is for sure still growing) Of course, the disruption could come at any year and add a new S-curve (Metaverse?). But disruption is by definition not predictable. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:edepa@ieee.org] Sent: Thursday, May 11, 2023 12:48 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com> Cc: Dave Taht <dave.taht@gmail.com>; Phil Bedard <bedard.phil@gmail.com>; NANOG <nanog@nanog.org> Subject: Re: Routed optical networks Eduard, academics cite the VNI (and the Sandvine Global reports). Do you know of alternative sources that show traffic growth data you're more comfortable with? Cheers, Etienne On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> wrote: But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates YYYY traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.taht@gmail.com<mailto:dave.taht@gmail.com>] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> Cc: Phil Bedard <bedard.phil@gmail.com<mailto:bedard.phil@gmail.com>>; Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 40000/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei.com@nanog.org<mailto:huawei.com@nanog.org>] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based optical network or not, especially at metro/regional distances. There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn’t exceed the capabilities of those optics. It’s been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment. It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit. Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices. Thanks, Phil From: NANOG <nanog-bounces+bedard.phil=gmail.com@nanog.org<mailto:nanog-bounces+bedard.phil=gmail.com@nanog.org>> on behalf of Etienne-Victor Depasquale via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Date: Monday, May 1, 2023 at 2:30 PM To: NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

To clarify the table I linked to in the previous email: Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with: • endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”). Both the mobile access network and the fixed access network are considered. Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA). In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic. Cheers, Etienne On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale <edepa@ieee.org> wrote:

Historically, this is what VNI has claimed <https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing> .

Cheers,

Etienne

On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard < vasilenko.eduard@huawei.com> wrote:

...

I did investigate traffic for every Carrier while dealing with it as a consultant (repeated many dozens of times).

I have seen over a decade how traffic growth dropped year-over-year (from 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel).

Sometimes I talk to old connections and they confirm that it is even less now.

In rear cases, It is typically possible to find this information on the public Internet (I remember the case when Google disclosed traffic for Pakistan at the conference with the explanation that 30% is attributed to new subscribers, and an additional +30% is to more heavy content per subscriber).

But mostly, it was confidential information from a discussion with Carriers – they all know very well their traffic growth.

In general, traffic stat is pretty confidential. I did not have the motivation to aggregate it.

Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future.

Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it.

VNI is claiming so many things. Please show where exactly they show traffic growth (I am not interested in prediction speculations). Is it possible to understand CAGR for the 5 last years? Is it declining or growing? (traffic itself is for sure still growing)

Of course, the disruption could come at any year and add a new S-curve (Metaverse?). But disruption is by definition not predictable.

PS: Everything above and below in this thread is just my personal opinion.

Eduard

*From:* Etienne-Victor Depasquale [mailto:edepa@ieee.org] *Sent:* Thursday, May 11, 2023 12:48 PM *To:* Vasilenko Eduard <vasilenko.eduard@huawei.com> *Cc:* Dave Taht <dave.taht@gmail.com>; Phil Bedard <bedard.phil@gmail.com>; NANOG <nanog@nanog.org> *Subject:* Re: Routed optical networks

Eduard, academics cite the VNI (and the Sandvine Global reports).

Do you know of alternative sources that show traffic growth data you're more comfortable with?

Cheers,

Etienne

On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard < vasilenko.eduard@huawei.com> wrote:

But it is speculation, not a trend yet.

I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen.

Now we see presentations that AI would be talking to AI that generates YYYY traffic.

Maybe some technology would push traffic next S-curve, maybe not. It is still speculation.

The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet.

It is almost finished for both trends. We are close to the plateau of these S-curves.

For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR.

Everything else up to now was “marketing” to say it mildly.

Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when.

PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget.

Eduard

*From:* Dave Taht [mailto:dave.taht@gmail.com] *Sent:* Tuesday, May 9, 2023 11:41 PM *To:* Vasilenko Eduard <vasilenko.eduard@huawei.com> *Cc:* Phil Bedard <bedard.phil@gmail.com>; Etienne-Victor Depasquale < edepa@ieee.org>; NANOG <nanog@nanog.org> *Subject:* Re: Routed optical networks

Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling.

On the other hand, predictions such as 640k should be enough for everyone did not pan out.

On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month:

https://lwn.net/Articles/930939/

Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so...

On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG < nanog@nanog.org> wrote:

Disclaimer: Metaverse has not changed Metro traffic yet. Then …

I am puzzled when people talk about 400GE and Tbps in the Mero context.

For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide.

How many BRASes serve more than 40000/1.5=27k users in the busy hour?

It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth.

Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful).

We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics.

Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau.

How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research.

IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS)

Additionally, who would pay for this traffic growth? It also limits traffic at some point.

I hope it would happen after we would get our 22h/4k/12bit/120hz.

Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed.

Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth.

Ed/

*From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei.com@nanog.org] *On Behalf Of *Phil Bedard *Sent:* Thursday, May 4, 2023 8:32 PM *To:* Etienne-Victor Depasquale <edepa@ieee.org>; NANOG <nanog@nanog.org> *Subject:* Re: Routed optical networks

It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations.

The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based optical network or not, especially at metro/regional distances.

There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn’t exceed the capabilities of those optics. It’s been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment.

It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit.

Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices.

Thanks,

Phil

*From: *NANOG <nanog-bounces+bedard.phil=gmail.com@nanog.org> on behalf of Etienne-Victor Depasquale via NANOG <nanog@nanog.org> *Date: *Monday, May 1, 2023 at 2:30 PM *To: *NANOG <nanog@nanog.org> *Subject: *Routed optical networks

Hello folks,

Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not?

Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks?

I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey.

Cheers,

Etienne

--

Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta

Web. https://www.um.edu.mt/profile/etiennedepasquale

--

Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/

Dave Täht CSO, LibreQos

--

Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta

Web. https://www.um.edu.mt/profile/etiennedepasquale

-- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

-- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

Hi Jared, Could I make a conclusion from your comments: "only Carrier itself understand the traffic - see many examples in the text". I would very agree to this. Eduard -----Original Message----- From: Jared Mauch [mailto:jared@puck.nether.net] Sent: Thursday, May 11, 2023 3:16 PM To: Etienne-Victor Depasquale <edepa@ieee.org> Cc: Vasilenko Eduard <vasilenko.eduard@huawei.com>; NANOG <nanog@nanog.org> Subject: Re: Routed optical networks

On May 11, 2023, at 7:45 AM, Etienne-Victor Depasquale via NANOG <nanog@nanog.org> wrote:

To clarify the table I linked to in the previous email:

Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with:

• endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”).

Both the mobile access network and the fixed access network are considered.

Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA).

In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic.

I think there’s a lot of problems here. While places like my employer will periodically disclose our traffic numbers, and DDoS providers, mitigation platforms and otherwise will disclose the peaks they see, much of this data is a bit opaque, and tools like AI that do in-metro or cross-metro datacenter-datacenter remote DMA type activities, those all count differently. We have seen a continued trend of the privatization of traffic and localization of that over time. I’ve watched all the big carriers retreat from their global network reaches to be more of regionalized networks. A decade ago you would have seen European national incumbents peering and with market in Asia, and the complete global networks continue to shrink. Meanwhile you have a mix of the content and cloud providers continue to build their business-purpose networks expanding into markets that the uppercase Internet may not need to reach. You can look at the proposals in the EU about fees, and I have dual thoughts on this which are MY OWN and don’t represent my employer or otherwise, but if you read this post from Petra Arts - https://blog.cloudflare.com/eu-network-usage-fees/ - it speaks around major interconnection points like Frankfurt, which are important but double as problematic. The number of people that need to go to the near market (eg: Chicago, while I’m in Detroit area) for good connectivity is an issue, meanwhile there’s a robust need to keep traffic within the state of Michigan and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on the board). There need to be some aggregation points, so not everyone needs to be in Detroit, but also not everyone needs to be in Frankfurt - and content localization needs to continue to happen, but is also very regionalized in popularity. How to do this all and not have it all route via Chicago or Frankfurt is a challenge, but also not everyone will be in Berlin, Munich or these other markets. This is where having a robust optical network capability (or backbone) can come into play, that you can deliver deeper from those hub points, but at the same time, I’ve been in meetings where companies have their own challenges accepting that content in those downstream locations as their network was also built to get to/from the major hub cities, or IP space wasn’t allocated in a way that can provide consistent routing results or behaviors. (This is where IPv6 can be super helpful, it gives the chance to possibly Greenfield, aka not screw it up - at least initially). There’s huge volumes of IP traffic exchanged, but the largest volumes are being moved over private interconnects or a localized IX to those eyeball networks with the historical global backbones playing more of the long-distance carrier role, which is critical as you want a path to deliver those bits, without it following the ITU-style sender pays model, as the majority of IP traffic is actually requested by the customer of the end-user network. (All of it if you remove network scans, ddos, web bots/crawlers). Most networks have no SLA once things cross an unpaid boundary (SFI, or even private peering) - and if they are a customer and that path is congested, it’s up to the customer to upgrade that path. - Jared (many hats)

On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote:

Hi Jared, Could I make a conclusion from your comments: "only Carrier itself understand the traffic - see many examples in the text". I would very agree to this.

I wouldn't say only carriers understand the traffic as much as I would say carrier's traffic is more transparent, and perhaps, even more readily available. I just think that it is not relevant to try and lump network and content traffic into one growth pattern. For all intents & purposes, there are two Internets running side-by-side between network and content. They converge at some point, but really, they are very different. Mark.

On 5/11/23 17:26, Jared Mauch wrote:

And as I’ve seen, they continue to become a bit more divergent. As someone who has an access network I see where the majority of my bits go, which is to the content folks. There’s some to the other part, but mostly people want their MTV, but since it’s 2023, it’s not MTV, but people want their TikTok, Metaverse, Game downloads, Streaming service, Email and cloud ramps (enterprise).

100% - and if the trend continues, Telegeography et al will have less raw growth to report on unless the content folk willingly open their skirts up to them for a peek - which they won't do. So yes, predicting stable or negative growth for the public Internet is not without merit. But that does not translate to what the content folk are recording. It might make more sense to start reporting on traffic growth in the edge, specifically, the peering edge, as content networks continue to become major on-ramp/off-ramp paths for telco's. But that prediction can be extrapolated from submarine cable builds, submarine cable upgrades as well as DWDM vendor sales, with some degree of reliability. Mark.

Hi Etienne, Look carefully what you have shown to me. It is a only speculation again (“predictions”). It is just a table with a collection of all predictions in the past. Moreover, averaged between years. I was asking for real data from the past 5 years. Are you sure that VNI has it? If you would find real historical data in VNI, then we would be capable to check the table that you have shown: was the guessing right? I strongly suspect an answer. Eduard From: Etienne-Victor Depasquale [mailto:edepa@ieee.org] Sent: Thursday, May 11, 2023 2:46 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com> Cc: Dave Taht <dave.taht@gmail.com>; Phil Bedard <bedard.phil@gmail.com>; NANOG <nanog@nanog.org> Subject: Re: Routed optical networks To clarify the table I linked to in the previous email: Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with: • endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”). Both the mobile access network and the fixed access network are considered. Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA). In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic. Cheers, Etienne On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>> wrote: Historically, this is what VNI has claimed<https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>. Cheers, Etienne On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> wrote: I did investigate traffic for every Carrier while dealing with it as a consultant (repeated many dozens of times). I have seen over a decade how traffic growth dropped year-over-year (from 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). Sometimes I talk to old connections and they confirm that it is even less now. In rear cases, It is typically possible to find this information on the public Internet (I remember the case when Google disclosed traffic for Pakistan at the conference with the explanation that 30% is attributed to new subscribers, and an additional +30% is to more heavy content per subscriber). But mostly, it was confidential information from a discussion with Carriers – they all know very well their traffic growth. In general, traffic stat is pretty confidential. I did not have the motivation to aggregate it. Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future. Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it. VNI is claiming so many things. Please show where exactly they show traffic growth (I am not interested in prediction speculations). Is it possible to understand CAGR for the 5 last years? Is it declining or growing? (traffic itself is for sure still growing) Of course, the disruption could come at any year and add a new S-curve (Metaverse?). But disruption is by definition not predictable. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:edepa@ieee.org<mailto:edepa@ieee.org>] Sent: Thursday, May 11, 2023 12:48 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> Cc: Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>>; Phil Bedard <bedard.phil@gmail.com<mailto:bedard.phil@gmail.com>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks Eduard, academics cite the VNI (and the Sandvine Global reports). Do you know of alternative sources that show traffic growth data you're more comfortable with? Cheers, Etienne On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> wrote: But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates YYYY traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.taht@gmail.com<mailto:dave.taht@gmail.com>] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> Cc: Phil Bedard <bedard.phil@gmail.com<mailto:bedard.phil@gmail.com>>; Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 40000/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei.com@nanog.org<mailto:huawei.com@nanog.org>] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based optical network or not, especially at metro/regional distances. There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn’t exceed the capabilities of those optics. It’s been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment. It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit. Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices. Thanks, Phil From: NANOG <nanog-bounces+bedard.phil=gmail.com@nanog.org<mailto:nanog-bounces+bedard.phil=gmail.com@nanog.org>> on behalf of Etienne-Victor Depasquale via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Date: Monday, May 1, 2023 at 2:30 PM To: NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

Why you are so sure that they have access to traffic data for many access networks? Their BRASes market share is far from 100%. DSLAMs finished 20 years ago. For the cases where they support BRASes they could collect statistics. But are they doing it? Carriers have not given permission (not even requested). I do not have a clue about VNI arrangement – it is magic. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:edepa@ieee.org] Sent: Thursday, May 11, 2023 6:03 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com> Cc: Dave Taht <dave.taht@gmail.com>; Phil Bedard <bedard.phil@gmail.com>; NANOG <nanog@nanog.org> Subject: Re: Routed optical networks Eduard, you know the answer as well as I do, right :) ? Here's my answer: I think that Cisco can only estimate (let's not say speculate, it has pretty bad connotations) what comes out of access networks. No offence meant, I hope none is taken. Cheers, Etienne On Thu, May 11, 2023 at 3:47 PM Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> wrote: Hi Etienne, Look carefully what you have shown to me. It is a only speculation again (“predictions”). It is just a table with a collection of all predictions in the past. Moreover, averaged between years. I was asking for real data from the past 5 years. Are you sure that VNI has it? If you would find real historical data in VNI, then we would be capable to check the table that you have shown: was the guessing right? I strongly suspect an answer. Eduard From: Etienne-Victor Depasquale [mailto:edepa@ieee.org<mailto:edepa@ieee.org>] Sent: Thursday, May 11, 2023 2:46 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> Cc: Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>>; Phil Bedard <bedard.phil@gmail.com<mailto:bedard.phil@gmail.com>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks To clarify the table I linked to in the previous email: Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with: • endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”). Both the mobile access network and the fixed access network are considered. Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA). In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic. Cheers, Etienne On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>> wrote: Historically, this is what VNI has claimed<https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>. Cheers, Etienne On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> wrote: I did investigate traffic for every Carrier while dealing with it as a consultant (repeated many dozens of times). I have seen over a decade how traffic growth dropped year-over-year (from 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). Sometimes I talk to old connections and they confirm that it is even less now. In rear cases, It is typically possible to find this information on the public Internet (I remember the case when Google disclosed traffic for Pakistan at the conference with the explanation that 30% is attributed to new subscribers, and an additional +30% is to more heavy content per subscriber). But mostly, it was confidential information from a discussion with Carriers – they all know very well their traffic growth. In general, traffic stat is pretty confidential. I did not have the motivation to aggregate it. Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future. Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it. VNI is claiming so many things. Please show where exactly they show traffic growth (I am not interested in prediction speculations). Is it possible to understand CAGR for the 5 last years? Is it declining or growing? (traffic itself is for sure still growing) Of course, the disruption could come at any year and add a new S-curve (Metaverse?). But disruption is by definition not predictable. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:edepa@ieee.org<mailto:edepa@ieee.org>] Sent: Thursday, May 11, 2023 12:48 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> Cc: Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>>; Phil Bedard <bedard.phil@gmail.com<mailto:bedard.phil@gmail.com>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks Eduard, academics cite the VNI (and the Sandvine Global reports). Do you know of alternative sources that show traffic growth data you're more comfortable with? Cheers, Etienne On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> wrote: But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates YYYY traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.taht@gmail.com<mailto:dave.taht@gmail.com>] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard <vasilenko.eduard@huawei.com<mailto:vasilenko.eduard@huawei.com>> Cc: Phil Bedard <bedard.phil@gmail.com<mailto:bedard.phil@gmail.com>>; Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 40000/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei.com@nanog.org<mailto:huawei.com@nanog.org>] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale <edepa@ieee.org<mailto:edepa@ieee.org>>; NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Routed optical networks It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based optical network or not, especially at metro/regional distances. There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn’t exceed the capabilities of those optics. It’s been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment. It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit. Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices. Thanks, Phil From: NANOG <nanog-bounces+bedard.phil=gmail.com@nanog.org<mailto:nanog-bounces+bedard.phil=gmail.com@nanog.org>> on behalf of Etienne-Victor Depasquale via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Date: Monday, May 1, 2023 at 2:30 PM To: NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale

On 5/11/23 13:45, Etienne-Victor Depasquale via NANOG wrote:

To clarify the table I linked to in the previous email:

Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with:

• endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”).

Both the mobile access network and the fixed access network are considered.

Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA).

So either Cisco think/though that the only operators worth surveying/predicting were the large ones (NTT, Telia, Tata, Lumen, Cogent, e.t.c.), or that on-net MPLS/VPN traffic was more significant than public IP Transit both in terms of revenue and strategic direction of the operators they surveyed/predicated. Either way, I'd imagine any results based on those data points would be incomplete, at least from a real-world standpoint.

In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic.

In the IP space, this traffic type is quickly exceeding any historically significant MPLS/VPN traffic, if it hasn't already. Mark.

On Thu, May 11, 2023 at 7:32 AM Mark Tinka <mark@tinka.africa> wrote:

On 5/11/23 13:25, Vasilenko Eduard via NANOG wrote:

Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future.

Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it.

And QUIC is making the the DPI model so loved by MNO's quickly irrelevant.

<plug> What we have been doing with libreqos is add in FQ + AQM (sch_cake), while pioneering some nifty new eBPF traffic monitoring abilities like being able to do "mice and elephants" plots, and more importantly sample TCP RTT statistics using a variant of kathie nichols' "passive ping" (pping) utility. This latter feature is proving very useful in diagnosing problems deeper in the network. Quic does respond to good ole queuing delays, packet drop, and in some cases ecn, but I am going to miss pping as it deploys more. It is the FQ that helps the most on videoconferencing and voip traffic. Libreqos is free software, working as a bridge, you can plug it in between any two points on your network, and on cheap (350 bucks off of ebay) xeon gold hardware easily cracks 25Gbits while shaping with a goal of cracking 100Gbits one day soon. demo here: https://payne.taht.net , running a variety of flent based stress tests, tcp up and downloads, the rrul test, etc. The interesting part to me, is that *real* traffic actually looks nothing like that on a per subscriber basis. Here´s a screen-movie example of real-world netflix queuing depth and delay: https://www.youtube.com/watch?v=C-2oSBr2200 Our new "piano roll" mice and elephants plotter is pretty neat, if I can say so myself. ... What we see from those ISPs that have shared their data from their Libreqos deployment is that the principal usage of major bandwidth is movie streaming, and there is essentially no difference in average usage between a 50Mbit residential plan and a gbit plan and only barely discernible at 25 vs 50. Given how new this codebase is we do not have any long term statistics for traffic growth or decline, or other patterns. But I kind of hope more folk here fire it up in their labs, at least. Please drop in on our chat channel #libreqos:matrix.org if you need help getting it setup. Sampling as we can at 10ms is a very different view of the net from 5 minute averages. </plug> Specifically plugging because I would like to understand CDN traffic patterns better, and do not have much data as yet on that side, collected this way.

Mark.

-- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos

Changing the topic... On Fri, May 12, 2023 at 7:11 AM Mark Tinka <mark@tinka.africa> wrote:

On 5/12/23 15:03, Dave Taht wrote:

...

Libreqos is free software, working as a bridge, you can plug it in between any two points on your network, and on cheap (350 bucks off of ebay) xeon gold hardware easily cracks 25Gbits while shaping with a goal of cracking 100Gbits one day soon.

This is fantastic!

:blush: We have done a couple podcasts about it, like this one: https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-... and have perhaps made a mistake by using matrix chat, rather than a web forum, to too-invisibly, do development and support in, but it has been a highly entertaining way to get a better picture of the real problems caring ISPs have. I see you are in Africa? We have a few ISPs playing with this in kenya...

I also found your post about it here:

https://www.reddit.com/r/HomeNetworking/comments/11pmc9a/a_latency_on_the_in...

If you could throw more hardware at it, could it do several 100's of Gbps?

We do not know. Presently our work is supported by equinix´s open source program, with four servers in their Dallas DC, and they are 25Gbit ports. Putting together enough dough to get to 100Gbit or finding someone willing to send traffic through more bare metal at that data center or elsewhere is on my mind. In other words, we can easily spin up the ability to L2 route some traffic through a box in their DCs, if only we knew where to find it. :) If you assume linearity to cores (which is a lousy assumption, ok?), 64 Xeon cores could do about 200Gbit, running flat out. I am certain it will not scale linearly and we will hit multiple bottlenecks on a way to that goal. Limits we know about: A) Trying to drive 10s of gbits of realistic traffic through this requires more test clients and servers than we have, or someone with daring and that kind of real traffic in the first place. For example one of our most gung-ho clients has 100Gbit ports, but not anywhere near that amount of inbound traffic. (they are crazy enough to pull git head, try it for a few minutes in production, and then roll back or leave it up) B) A brief test of a 64 core AMD + Nvidia ethernet was severely outperformed by our current choice of a 20 core xeon gold + intel 710 or 810 card. It is far more the ethernet card that is the dominating factor. I would kill if I could find one that did a LPM -> CPU mapping... (e.g. instead of a LPM->route mapping, LPM to what cpu to interrupt). We also tried an 80 core arm to inconclusive results early on. Tests of the latest ubuntu release are ongoing. I am not prepared to bless that or release any results yet. C) A single cake instance on one of the more high end Xeons can *almost* push 10Gbit/sec while eating a core. D) Our model is one cake instance per subscriber + the ability to establish trees emulating links further down the chain. One ISP is modeling 10 mmwave hops. Another is just putting in multiple boxes closer to the towers. So in other words, 100s of gbits is achievable today if you throw boxes at it, and more cost effective to do that way. We will of course, keep striving to crack 100gbit native on a single box with multiple cards. It is a nice goal to have. E) In our present, target markets, 10k typical residential subscribers only eat 11Gbit/sec at peak. That is a LOT of the smaller ISPs and networks that fit into that space, so of late we have been focusing more on analytics and polish than pushing more traffic. Some of our new R/T analytics break down at 10k cake instances (that is 40 million fq_codel queues, ok?), and we cannot sample at 10ms rates, falling back to (presently) 1s conservatively. We are nearing putting out a v1.4-rc7 which is just features and polish, you can get a .deb of v1.4-rc6 here: https://github.com/LibreQoE/LibreQoS/releases/tag/v1.4-rc6 There is an optional, and anonymized reporting facility built into that. In the last two months, 44404 cake shaped devices shaping .19Tbits that we know of have come online. Aside from that we have no idea how many ISPs have picked it up! a best guess would be well over 100k subs at this point. Putting in libreqos is massively cheaper than upgrading all the cpe to good queue management, (it takes about 8 minutes to get it going in monitor mode, but exporting shaping data into it requires glue, and time) but better cpe remains desirable - especially that the uplink component of the cpe also do sane shaping natively. "And dang, it, ISPs of the world, please ship decent wifi!?", because we can see the wifi going south in many cases from this vantage point now. In the past year mikrotik in particular has done a nice update to fq_codel and cake in RouterOS, eero 6s have got quite good, much of openwifi/openwrt, evenroute is good... It feels good, after 14 years of trying to fix the internet, to be seeing such progress, on fixing bufferbloat, and in understanding and explaining the internet better. joooooiiiiiiiin us..

Also, when you say "bridge", if the server dies, does it become a wire, or would that require specialized hardware builds?

What we do now is put it inline with ospf/olsr/bgp with a low cost, and a wire with a higher cost, if it fails. Things have stablized a lot in the last few months, the last crash I can remember was in january. (in rust we trust!). You have to watch out for breaking spanning tree in that case. The most common install bug is someone flipping inbound and outbound interfaces in the setup. Among other things we replaced the linux native bridge code with about 600 lines of ebpf C. The enormous speedup from that is getting us closer to what dpdk could do, but dpdk cannot queue worth a darn, just forward willy nilly. I hope, in particular, far, far more folk start leveraging variants of doing inband measurements with pping. The stand alone code for that is here: https://github.com/thebracket/cpumap-pping

Mark.

-- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos

On Sat, May 13, 2023 at 12:28 AM Mark Tinka <mark@tinka.africa> wrote:

On 5/12/23 17:59, Dave Taht wrote:

...

:blush:

We have done a couple podcasts about it, like this one:

https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-...

and have perhaps made a mistake by using matrix chat, rather than a web forum, to too-invisibly, do development and support in, but it has been a highly entertaining way to get a better picture of the real problems caring ISPs have.

I see you are in Africa? We have a few ISPs playing with this in kenya...

DM me, please, the ones you are aware about that would be willing to share their experiences. I'd like to get them to talk about what they've gathered at the upcoming SAFNOG meeting in Lusaka.

We have a fairly large network in Kenya, so would be happy to engage with the operators running the LibreQoS there.

I forwarded your info. Slide 40 here, has an anonymized libreqos report of observed latencies in Africa. The RTTs there are severely bimodal (30ms vs 300ms), which mucks with a default assumption in sch_cake of a default 100ms RTT. http://www.taht.net/~d/Misunderstanding_Residential_Bandwidth_Latency.pdf There are two ways to deal with this, right now we are recommending cake rtt 200ms setting to keep throughput up. The FQ component dominates for most traffic, anyway. With a bit more work we hope to come up with a way of more consistent queuing delay. Or, we could just wait for more CDNs to and IXPs to deploy there. /me hides A note about our public plots: We had a lot of people sharing screenshots, so we added a "klingon mode" to consistently transliterate the more private data to that language. Another fun fact was by deploying this stuff several folk found sufficient non-paying clients on their network to pay for the hardware inside of a month or two.

...

We do not know. Presently our work is supported by equinix´s open source program, with four servers in their Dallas DC, and they are 25Gbit ports. Putting together enough dough to get to 100Gbit or finding someone willing to send traffic through more bare metal at that data center or elsewhere is on my mind. In other words, we can easily spin up the ability to L2 route some traffic through a box in their DCs, if only we knew where to find it. :)

If you assume linearity to cores (which is a lousy assumption, ok?), 64 Xeon cores could do about 200Gbit, running flat out. I am certain it will not scale linearly and we will hit multiple bottlenecks on a way to that goal.

Limits we know about:

A) Trying to drive 10s of gbits of realistic traffic through this requires more test clients and servers than we have, or someone with daring and that kind of real traffic in the first place. For example one of our most gung-ho clients has 100Gbit ports, but not anywhere near that amount of inbound traffic. (they are crazy enough to pull git head, try it for a few minutes in production, and then roll back or leave it up)

B) A brief test of a 64 core AMD + Nvidia ethernet was severely outperformed by our current choice of a 20 core xeon gold + intel 710 or 810 card. It is far more the ethernet card that is the dominating factor. I would kill if I could find one that did a LPM -> CPU mapping... (e.g. instead of a LPM->route mapping, LPM to what cpu to interrupt). We also tried an 80 core arm to inconclusive results early on.

Tests of the latest ubuntu release are ongoing. I am not prepared to bless that or release any results yet.

C) A single cake instance on one of the more high end Xeons can *almost* push 10Gbit/sec while eating a core.

D) Our model is one cake instance per subscriber + the ability to establish trees emulating links further down the chain. One ISP is modeling 10 mmwave hops. Another is just putting in multiple boxes closer to the towers.

So in other words, 100s of gbits is achievable today if you throw boxes at it, and more cost effective to do that way. We will of course, keep striving to crack 100gbit native on a single box with multiple cards. It is a nice goal to have.

E) In our present, target markets, 10k typical residential subscribers only eat 11Gbit/sec at peak. That is a LOT of the smaller ISPs and networks that fit into that space, so of late we have been focusing more on analytics and polish than pushing more traffic. Some of our new R/T analytics break down at 10k cake instances (that is 40 million fq_codel queues, ok?), and we cannot sample at 10ms rates, falling back to (presently) 1s conservatively.

We are nearing putting out a v1.4-rc7 which is just features and polish, you can get a .deb of v1.4-rc6 here:

https://github.com/LibreQoE/LibreQoS/releases/tag/v1.4-rc6

There is an optional, and anonymized reporting facility built into that. In the last two months, 44404 cake shaped devices shaping .19Tbits that we know of have come online. Aside from that we have no idea how many ISPs have picked it up! a best guess would be well over 100k subs at this point.

Putting in libreqos is massively cheaper than upgrading all the cpe to good queue management, (it takes about 8 minutes to get it going in monitor mode, but exporting shaping data into it requires glue, and time) but better cpe remains desirable - especially that the uplink component of the cpe also do sane shaping natively.

"And dang, it, ISPs of the world, please ship decent wifi!?", because we can see the wifi going south in many cases from this vantage point now. In the past year mikrotik in particular has done a nice update to fq_codel and cake in RouterOS, eero 6s have got quite good, much of openwifi/openwrt, evenroute is good...

It feels good, after 14 years of trying to fix the internet, to be seeing such progress, on fixing bufferbloat, and in understanding and explaining the internet better. joooooiiiiiiiin us..

All sounds very exciting.

I am happier about this than I have been since free.fr deployed fq_codel (to ultimately 3m devices) in 2012.

I'll share this with some friends at Cisco who are actively looking at ways to incorporate such tech. in their routers in response to QUIC. They might find it interesting.

I had hoped they were paying attention, in particular, that Cisco AFD would deploy more widely. Is anyone using that? It doesn´t do ECN (I think) but seemed promising at the time I encountered it. I kind of gave up on getting juniper to revisit their RED implementation, and started slapping cake on mikrotik in front of it. Here´s an example of why... https://blog.cerowrt.org/post/juniper/ hardware-wise, not middlebox, running native, right now it is just the low/middle end of the market fq_codeled - the cambiums, ubnts (smart queues are pretty universal there), openwrt and derivatives like openwifi (I am pleased to see tip openwifi gaining traction), opensense, freebsd, openbsd, linux, apple, and mikrotiks, the eeros, riverbeds, peplinks, evenroute, firewalla, now a pretty long list!, with only a few calling out the underlying algorithm specifically. Peplink has a "mitigate bufferbloat" checkbox, ubnt (smart queues). Despite my advocacy here of libreqos, cake is now also in paraqum´s product. They are claiming 100Gbit support, and I do not know how that works!! Also fq_codel has long been an integral part of preseem, which I think is the leading middlebox in the wisp world. As best as I can tell, only bequant (licensed by cambium) is doing a more dpi-oriented middlebox approach still. I do not know much about other major DPI players nowadays. ? These companies actually have business models, which thus far is sadly lacking for libreqos. Donations and feature bounties are not cutting it. Without equinix´s support and a few fervent believers we would not be where we are today, and I worry about exponential growth overwhelming our chat room´s all-volunteer support department. Yesterday we had 44044 shaped devices, this morning 46864 shaped devices... I think the install process is now easy enough for most folk to get it going without any support, but...

...

What we do now is put it inline with ospf/olsr/bgp with a low cost, and a wire with a higher cost, if it fails. Things have stablized a lot in the last few months, the last crash I can remember was in january. (in rust we trust!). You have to watch out for breaking spanning tree in that case. The most common install bug is someone flipping inbound and outbound interfaces in the setup.

Among other things we replaced the linux native bridge code with about 600 lines of ebpf C. The enormous speedup from that is getting us closer to what dpdk could do, but dpdk cannot queue worth a darn, just forward willy nilly.

I hope, in particular, far, far more folk start leveraging variants of doing inband measurements with pping. The stand alone code for that is here: https://github.com/thebracket/cpumap-pping

Thank you for doing this work. Because of the scope of our network, it's

Honestly, I want to finally quit doing bufferbloat in the next year or three, and work on other things. farplay.io is pretty neat, so is jacktrip.org. I have spent a lot of time in the past year trying to shift USA BEAD planning in more of the right directions.

not something that we would deploy in this current form (which is why I'd like to see what Cisco think about it, even if we don't really use them much anymore). But I do see the utility in it, especially for the smaller-to-medium sized ISP's, and will be sure let the community know about this.

Thanks for being willing to share!

Mark.

-- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos

On 5/11/23 09:33, Vasilenko Eduard via NANOG wrote:

But it is speculation, not a trend yet.

I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen.

Now we see presentations that AI would be talking to AI that generates YYYY traffic.

Maybe some technology would push traffic next S-curve, maybe not. It is still speculation.

The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet.

It is almost finished for both trends. We are close to the plateau of these S-curves.

For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR.

Everything else up to now was “marketing” to say it mildly.

Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when.

PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget.

I think the place where we are seeing continued growth is in the content provider private network, for that DC-to-DC traffic, even across continents. For everything else, as you say, it's nice to predict. But what analysts and marketing departments say and what end users actually do are often vastly different. As long as the majority of the Internet's traffic is "invisible", we can't really ever know. We can only tell by how much kit the DWDM vendors are selling to non-telco customers (content folk), as well as how many submarine cables are being built by non-telco consortia (content folk). Mark.

On 5/12/23 22:14, Mike Hammett wrote:

"I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen"

Often the type of people making these kinds of predictions that a tire pressure sensor generates 20 gigabytes of traffic a day.

I like growing old... your BS detector becomes so slick, you know to ignore certain links, conferences, speakers, topics, meetings, slideware, e-mails, colleagues and announcements without fear of actually missing out on trends, because you know that in the end it will lead to nowhere real :-). Mark.