I'm being sloppy with my verbiage, it's just been a long time since I thought about this in detail, sorry. The MAC layer hands bits to the Media Independent Interface, which connects the MAC to the PHY. The PHY converts the digital 1/0 into the form required by the media transmission type; the 'what goes over the wire' L1 stuff. The method of encoding will always add SOME number of bits as overhead. Ex, 64b/66b means that for every 64 bits of data to transmit, 2 bits are added, so 66 actual bits are transmitted. This encoding overhead is what I meant when I said 'ethernet control frame juju'. This starts getting into the weeds on symbol/baud rates and stuff as well, which I dont want to do now cause I'm even rustier there. When FEC is enabled, the number of overhead bits added to the transmission increases. For 400G-FR4 for example, you start with 256b/257b , which is doubled to 512b/514b for ($reason I cannot remember), then RS-FEC(544,514) is applied, adding 30 more bits for FEC. Following the example, this means 544 bits are transmitted for every 512 bits of payload data. So , more overhead. Those additional bits can correct up to 15 corrupted bits of the payload. All of these overhead bits are added in the PHY on the way out, and removed on the way in. So you'll never see them on a packet capture unless you're using something that's actually grabbing the bits off the wire. ( Pretty sure this is right, anyone please correct me if I munged any of it up.) On Thu, Apr 18, 2024 at 2:45 PM Aaron Gould <aaron1@gvtc.com> wrote:
Thanks. What "all the ethernet control frame juju" might you be referring to? I don't recall Ethernet, in and of itself, just sending stuff back and forth. Does anyone know if this FEC stuff I see concurring is actually contained in Ethernet Frames? If so, please send a link to show the ethernet frame structure as it pertains to this 400g fec stuff. If so, I'd really like to know the header format, etc.
-Aaron On 4/18/2024 1:17 PM, Tom Beecher wrote:
FEC is occurring at the PHY , below the PCS.
Even if you're not sending any traffic, all the ethernet control frame juju is still going back and forth, which FEC may have to correct.
I *think* (but not 100% sure) that for anything that by spec requires FEC, there is a default RS-FEC type that will be used, which *may* be able to be changed by the device. Could be fixed though, I honestly cannot remember.
On Thu, Apr 18, 2024 at 1:35 PM Aaron Gould <aaron1@gvtc.com> wrote:
Not to belabor this, but so interesting... I need a FEC-for-Dummies or FEC-for-IP/Ethernet-Engineers...
Shown below, my 400g interface with NO config at all... Interface has no traffic at all, no packets at all.... BUT, lots of FEC hits. Interesting this FEC-thing. I'd love to have a fiber splitter and see if wireshark could read it and show me what FEC looks like...but something tells me i would need a 400g sniffer to read it, lol
It's like FEC (fec119 in this case) is this automatic thing running between interfaces (hardware i guess), with no protocols and nothing needed at all in order to function.
-Aaron
{master} me@mx960> show configuration interfaces et-7/1/4 | display set
{master} me@mx960>
{master} me@mx960> clear interfaces statistics et-7/1/4
{master} me@mx960> show interfaces et-7/1/4 | grep packet Input packets : 0 Output packets: 0
{master} me@mx960> show interfaces et-7/1/4 | grep "put rate" Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps)
{master} me@mx960> show interfaces et-7/1/4 | grep rror Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Bit errors 0 Errored blocks 0 Ethernet FEC statistics Errors FEC Corrected Errors 28209 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 2347 FEC Uncorrected Errors Rate 0
{master} me@mx960> show interfaces et-7/1/4 | grep packet Input packets : 0 Output packets: 0
{master} me@mx960> show interfaces et-7/1/4 | grep "put rate" Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps)
{master} me@mx960> show interfaces et-7/1/4 | grep rror Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Bit errors 0 Errored blocks 0 Ethernet FEC statistics Errors FEC Corrected Errors 45153 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 29 FEC Uncorrected Errors Rate 0
{master} me@mx960> show interfaces et-7/1/4 | grep packet Input packets : 0 Output packets: 0
{master} me@mx960> show interfaces et-7/1/4 | grep "put rate" Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps)
{master} me@mx960> show interfaces et-7/1/4 | grep rror Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Bit errors 0 Errored blocks 0 Ethernet FEC statistics Errors FEC Corrected Errors 57339 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 2378 FEC Uncorrected Errors Rate 0
{master} me@mx960>
On 4/18/2024 7:13 AM, Mark Tinka wrote:
On 4/17/24 23:24, Aaron Gould wrote:
Well JTAC just said that it seems ok, and that 400g is going to show 4x more than 100g "This is due to having to synchronize much more to support higher data."
We've seen the same between Juniper and Arista boxes in the same rack running at 100G, despite cleaning fibres, swapping optics, moving ports, moving line cards, e.t.c. TAC said it's a non-issue, and to be expected, and shared the same KB's.
It's a bit disconcerting when you plot the data on your NMS, but it's not material.
Mark.
-- -Aaron
-- -Aaron