Re: constant FEC errors juniper mpc10e 400g
On Sat, 20 Apr 2024 at 14:35, Mark Tinka <mark@tinka.africa> wrote:
Even when our market seeks OTN from European backhaul providers to extend submarine access into Europe and Asia-Pac, it is often for structured capacity grooming, and not for OAM benefit.
It would be interesting to learn whether other markets in the world still make a preference for OTN in lieu of Ethernet, for the OAM benefit, en masse. When I worked in Malaysia back in the day (2007 - 2012), WAN-PHY was generally asked for for 10G services, until about 2010; when folk started to choose LAN-PHY. The reason, back then, was to get that extra 1% of pipe bandwidth :-).
Oh I don't think OTN or WAN-PHY have any large deployment future, the cheapest option is 'good enough' and whatever value you could extract from OTN or WAN-PHY, will be difficult to capitalise, people usually don't even capitalise the capabilities they already pay for in the cheaper technologies. Of course WAN-PHY is dead post 10GE, a big reason for it to exist was very old optical systems which simply could not regenerate ethernet framing, not any features or functional benefits. -- ++ytti
On 4/20/24 13:39, Saku Ytti wrote:
Oh I don't think OTN or WAN-PHY have any large deployment future, the cheapest option is 'good enough' and whatever value you could extract from OTN or WAN-PHY, will be difficult to capitalise, people usually don't even capitalise the capabilities they already pay for in the cheaper technologies.
A handful of OEM's still push OTN like it has just been invented, especially those still pushing "IPoDWDM" :-). Fair point, if you have a highly-meshed metro network with lots of drops to customers across a ring-mesh topology, there might be some value in OTN when delivering such services at low speeds (10G, 25G, 2.5G, 1G). But while the topology is valid, most networks aren't using high-end optical gear to drop low-speed services, nowadays. Even though on a per-bit basis, they might be cheaper than 1U IP/MPLS router looking to do the same job if all you are considering is traffic, and not additional services that want to eat packets.
Of course WAN-PHY is dead post 10GE, a big reason for it to exist was very old optical systems which simply could not regenerate ethernet framing, not any features or functional benefits.
In our market, we are trending toward a convergence between 10G and 100G orders intersecting for long haul and submarine asks. But pockets of 10G demand still exist in many African countries, and none of them have any WAN-PHY interest of any statistical significance. That said, I don't expect any subsea cables getting built in the next 3 years and later will have 10G as a product on the SLTE itself... it wouldn't be worth the spectrum. Mark.
hey,
That said, I don't expect any subsea cables getting built in the next 3 years and later will have 10G as a product on the SLTE itself... it wouldn't be worth the spectrum.
10G wavelengths for new builds died about 10 years ago when coherent 100G became available, submarine or not. Putting 10G into same system is not really feasible at all. -- tarko
On 4/20/24 18:19, Tarko Tikan wrote:
10G wavelengths for new builds died about 10 years ago when coherent 100G became available, submarine or not. Putting 10G into same system is not really feasible at all.
I was referring to 10G services (client-side), not 10G wavelengths (line side). Mark.
Assume that some carrier has 10k FBB subscribers in a particular municipality (without any hope of considerably increasing this number). 2Mbps is the current average per household in the busy hour, pretty uniform worldwide. You could multiply it by 8/7 if you like to add wireless -> not much would change. 2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to carry 10k subscribers. The optical ring may be less than 20 municipalities - it is very common. Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) makes sense for some carriers. It depends on the population density and the carrier market share. 10GE for the WAN side would not be dead in the next 5 years because 2Mbps per household would not grow very fast in the future - this logistic curve is close to a plateau. PS: It is probably not the case for Africa where new subscribers are connected to the Internet at a fast rate. Ed/ -----Original Message----- From: NANOG <nanog-bounces+vasilenko.eduard=huawei.com@nanog.org> On Behalf Of Tarko Tikan Sent: Saturday, April 20, 2024 19:19 To: nanog@nanog.org Subject: Re: constant FEC errors juniper mpc10e 400g hey,
That said, I don't expect any subsea cables getting built in the next 3 years and later will have 10G as a product on the SLTE itself... it wouldn't be worth the spectrum.
10G wavelengths for new builds died about 10 years ago when coherent 100G became available, submarine or not. Putting 10G into same system is not really feasible at all. -- tarko
On 4/22/24 09:47, Vasilenko Eduard via NANOG wrote:
Assume that some carrier has 10k FBB subscribers in a particular municipality (without any hope of considerably increasing this number). 2Mbps is the current average per household in the busy hour, pretty uniform worldwide. You could multiply it by 8/7 if you like to add wireless -> not much would change. 2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to carry 10k subscribers. The optical ring may be less than 20 municipalities - it is very common. Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) makes sense for some carriers. It depends on the population density and the carrier market share. 10GE for the WAN side would not be dead in the next 5 years because 2Mbps per household would not grow very fast in the future - this logistic curve is close to a plateau. PS: It is probably not the case for Africa where new subscribers are connected to the Internet at a fast rate.
As a function of how much Internet there is in Africa, there really aren't that many optical transport service providers. Some countries/cities/towns have more than they need, others have just one. But in general, you would say there is massive room for improvement if you surveyed the entire continent. Typically, it will be the incumbents, alongside 2 or 3 competitives. In fact, in some African countries, only the incumbent may be large enough to run an optical backbone, with all the competitives leasing capacity from them. It is not uncommon to find the closest competitor to an incumbent for terrestrial services being the mobile network operator, purely because they have some excess capacity left over from having to build the backbone for their core business, mobile. And, they are flush with cash, so a loss-making terrestrial backhaul business can be covered by the month's sales in SIM cards. Truly independent transport providers are few and far between because access to dark fibre is not easy (either its lack of availability, the incumbent refusing to sell it, or its high price). For the few independent transport providers that do spring up, they will focus on a limited set of hot routes, and because competition on those routes may be wanting, prices and capacity would not be terribly attractive. So the bulk of Africa's Internet really relies on a handful of key African wholesale IP Transit providers taking great effort into extending their network as deep into cities as they can, and using their size to negotiate the best prices for terrestrial backhaul from the few optical network operators that the market has. Those providers then sell to the local and regional ISP's, who don't have to worry about running a backbone. All this means is that for those operators that run an optical backbone, especially nationally, 10G carriers are very, very legacy. If they still have them, it'd be a spin-off off the main core to support some old SDH customers that are too comfortable to move to Ethernet. Metro backhaul and last mile FNO's (fibre network operators) who have invested in extending access into homes and businesses are a different story, with most countries having a reasonable handful of options customers can choose from. Like national backhaul, there is plenty of room for improvement - in some markets more than others - but unlike national backhaul, not as constrained for choice or price. Mark.
On 4/20/24 13:39, Saku Ytti wrote:
Oh I don't think OTN or WAN-PHY have any large deployment future, the cheapest option is 'good enough'...
And what we find with EU providers is that Ethernet and OTN services are priced similarly. It's a software toggle on a transponder, but even then, Ethernet still continues to be preferred over OTN. Mark.
LAN PHY dominates in the US too. Requests for WAN PHY were almost exclusively for terrestrial backhaul extending off of legacy subsea systems that still commonly had TDM-framed services. It’s been a couple of years since I’ve been in optical transport directly but these requests were essentially non-existent after 2018 or so. OTN became somewhat more common from 2014 onward as optical system interop improved, but actually was more common in the enterprise space as providers would generally go straight to fiber in most use cases, and with dark fiber opex costs coming down in many markets, I see OTN requests as winnowing here as well. Dave Cohen craetdave@gmail.com
On Apr 20, 2024, at 7:57 AM, Mark Tinka <mark@tinka.africa> wrote:
On 4/20/24 13:39, Saku Ytti wrote:
Oh I don't think OTN or WAN-PHY have any large deployment future, the cheapest option is 'good enough'...
And what we find with EU providers is that Ethernet and OTN services are priced similarly. It's a software toggle on a transponder, but even then, Ethernet still continues to be preferred over OTN.
Mark.
On 4/20/24 14:41, Dave Cohen wrote:
LAN PHY dominates in the US too. Requests for WAN PHY were almost exclusively for terrestrial backhaul extending off of legacy subsea systems that still commonly had TDM-framed services. It’s been a couple of years since I’ve been in optical transport directly but these requests were essentially non-existent after 2018 or so. OTN became somewhat more common from 2014 onward as optical system interop improved, but actually was more common in the enterprise space as providers would generally go straight to fiber in most use cases, and with dark fiber opex costs coming down in many markets, I see OTN requests as winnowing here as well.
What really changed the game was coherent detection, which breathed new life into legacy subsea cables that were built on dispersion-managed fibre. Post-2014 when uncompensated (and highly dispersed) fibre has been the standard for subsea builds (even for SDM cables), coherent optical systems are the mainstay. In fact, because linear dispersion can be accurately calculated for the cable span, uncompensated cables are a good thing because the dispersion compensation happens in very advanced coherent DSP's in the optical engine, rather than in the fibre itself. WAN-PHY did not extend to 40G or 100G, which can explain one of the reasons it lost favour. For 10G, its availability also depended on the type of device, its NOS, line card and/or pluggable at the time, which made it hard to find a standard around this if you built multi-vendor networks or purchased backhaul services from 3rd party providers that had non-standard support for WAN-PHY/OTN/G.709. In other words, LAN-PHY (and plain Ethernet) became the lowest common denominator in the majority of cases for customers. In 2024, I find that operators care more about bringing the circuit up than using its link properties to trigger monitoring, failover and reconvergence. The simplest way to do that is to ask for plain Ethernet services, particularly for 100G and 400G, but also for 10G. In practice, this has been reasonably reliable in the past 2 - 3 years when procuring 100G backhaul services. So for the most part, users of these services seem to be otherwise happy. Mark.
Erm, WAN-PHY did not extend into 40G because there was not much of those STM-256 deployment? (or customers didnt wanted to pay for those). WAN-PHY was designed so people could encapsulate Ethernet frames right into STM-64. Once world moved out of SDH/SONET stuff, there was no more need for WAN-PHY anymore. ---------- Original message ---------- From: Mark Tinka <mark@tinka.africa> To: Dave Cohen <craetdave@gmail.com> Cc: nanog@nanog.org Subject: Re: constant FEC errors juniper mpc10e 400g Date: Sat, 20 Apr 2024 17:50:04 +0200 WAN-PHY did not extend to 40G or 100G, which can explain one of the reasons it lost favour. For 10G, its availability also depended on the type of device, its NOS, line card and/or pluggable at the time, which made it hard to find a standard around this if you built multi-vendor networks or purchased backhaul services from 3rd party providers that had non-standard support for WAN-PHY/OTN/G.709. In other words, LAN-PHY (and plain Ethernet) became the lowest common denominator in the majority of cases for customers. Mark.
On 4/20/24 21:36, borg@uu3.net wrote:
Erm, WAN-PHY did not extend into 40G because there was not much of those STM-256 deployment? (or customers didnt wanted to pay for those).
With SONET/SDH, as the traffic rate increased, so did the number of overhead bytes. With every iteration of the data rate, the overhead bytes quadrupled. This was one of the key reasons we did not see field deployment of STM-256/OC-768 and STM-1024/OC-3072. For example, if SONET/SDH had to transport a 100G service, it would require 160Gbps of bandwidth. That clearly wasn't going to work. At the time when STM-256/OC-768 was being developed, DWDM and OTN had come a long way. The granularity SONET/SDH required to stand up a service had become too small for the growing data rate (primarily VC-3, VC-4 and VC-12). If you look at OTN, the smallest container is 1.25Gbps (ODU0), which was attractive for networks looking to migrate from E1's, E3's, STM-1's, STM-4's and STM-16's - carried over VC-12, VC-4 and VC-3 SDH circuits - to 1GE, for example, rather than trying to keep their PDH/SDH infrastructure going. DWDM and OTN permitted a very small control overhead, so as data rates increased, there wasn't the same penalty you got with SONET/SDH.
WAN-PHY was designed so people could encapsulate Ethernet frames right into STM-64. Once world moved out of SDH/SONET stuff, there was no more need for WAN-PHY anymore.
Technically, what you are describing is EoS (Ethernet over SONET, Ethernet over SDH), which is not the same as WAN-PHY (although the working groups that developed these nearly confused each other in the process, ANSI/ITU for the former vs. IEEE for the latter). WAN-PHY was developed to be operated across multiple vendors over different media... SONET/SDH, DWDM, IP/MPLS/Ethernet devices and even dark fibre. The goal of WAN-PHY was to deliver a low-cost Ethernet interface that was SONET/SDH-compatible, as EoS interfaces were too costly for operators and their customers. As we saw in real life, 10GE ports out-sold STM-64/OC-192 ports, as networks replaced SONET/SDH backbones with DWDM and OTN. Mark.
On Sun, 21 Apr 2024 at 09:05, Mark Tinka <mark@tinka.africa> wrote:
Technically, what you are describing is EoS (Ethernet over SONET, Ethernet over SDH), which is not the same as WAN-PHY (although the working groups that developed these nearly confused each other in the process, ANSI/ITU for the former vs. IEEE for the latter).
WAN-PHY was developed to be operated across multiple vendors over different media... SONET/SDH, DWDM, IP/MPLS/Ethernet devices and even dark fibre. The goal of WAN-PHY was to deliver a low-cost Ethernet interface that was SONET/SDH-compatible, as EoS interfaces were too costly for operators and their customers.
As we saw in real life, 10GE ports out-sold STM-64/OC-192 ports, as networks replaced SONET/SDH backbones with DWDM and OTN.
Key difference being, WAN-PHY does not provide synchronous timing, so it's not SDH/SONET compatible for strict definition for it, but it does have the frame format. And the optical systems which could regenerate SONET/SDH framing, didn't care about timing, they just wanted to be able to parse and generate those frames, which they could, but they could not do it for ethernet frames. I think it is pretty clear, the driver was to support long haul regeneration, so it was always going to be a stop-gap solution. Even though I know some networks, who specifically wanted WAN-PHY for its error reporting capabilities, I don't think this was majority driver, majority driver almost certainly was 'thats only thing we can put on this circuit'. -- ++ytti
On 4/21/24 08:12, Saku Ytti wrote:
Key difference being, WAN-PHY does not provide synchronous timing, so it's not SDH/SONET compatible for strict definition for it, but it does have the frame format. And the optical systems which could regenerate SONET/SDH framing, didn't care about timing, they just wanted to be able to parse and generate those frames, which they could, but they could not do it for ethernet frames.
Correct. In those days, WAN-PHY was considered "SONET/SDH-Lite".
I think it is pretty clear, the driver was to support long haul regeneration, so it was always going to be a stop-gap solution. Even though I know some networks, who specifically wanted WAN-PHY for its error reporting capabilities, I don't think this was majority driver, majority driver almost certainly was 'thats only thing we can put on this circuit'.
SONET/SDH did have similar reach as OTN back then, just less bandwidth for the distance. It had FEC support for STM-16, STM-64 and STM-256. I really think the bigger driver was interface cost, because EoS had already been selling for 1GE alongside STM-16 for 2.5G. In those days, if you needed more than 1G but less than 10G, it was a toss-up between 2x 1G EoSDH vs. 1x STM-16. Often times, you took the 2x 1G EoSDH because 2x 1GE ports were cheaper than 1x STM-16 port, even though you ended up losing about 405Mbps of capacity in the process, which was a huge deal. The backbone providers did not like selling EoSDH services, because it was too much admin. for them (VC container management), and they ended up paying more for transponders on their side than their customers did for Ethernet ports on theirs :-). But by and large, the majority of networks in our market maintained SDH services long after coherent became commercially available. It was a perception thing, that SDH was more superior to Ethernet, even if that Ethernet was transported over a DWDM network. In the end, SDH port costs were too hard to ignore due to router vendors maintaining their mark-up on them despite dying demand, which then led to the migration from SDH to EoDWDM growing significantly from about 2016. Optical vendors also began de-prioritizing SDH transponder ports, which had a massive impact on the SLTE (submarine) side in making the decision to encourage customers away from SDH for wet services. Mark.
participants (6)
-
borg＠uu3.net -
Dave Cohen -
Mark Tinka -
Saku Ytti -
Tarko Tikan -
Vasilenko Eduard