On 4/Apr/20 19:16, John St.Martin wrote:

Mark, Thanks for sharing your experiences with FRR. I don't work with IS-IS, but have found the development to be very active and fixing reproducible bugs quickly.

It look like FRR put a patch in for the bug you referenced and have a test build from 3/21 available which allows for up to 16k MTU @ https://ci1.netdef.org/browse/FRR-FRRPULLREQ-11364/artifact

I hope this helps and please continue to share your progress with the community.

Thanks for the feedback, John. It's great to hear this issue is being fixed. So I'm not really trying to get IS-IS to run at a high MTU, as that doesn't really do anything for the data plane. What I am really looking for is a way to tell IS-IS to run a specific MTU. In my network, even though we can carry up to 9,192 bytes in the data plane, we manually set IS-IS to operate at 8,000 bytes. This is due to VMware's limitation to address an MTU larger than 9,000, and we use it to run CSR1000v for our route reflector. So globally, our IS-IS domain runs 8,000 bytes. I am just looking for a way to tell IS-IS, in FRR, to run at 8,000 bytes, which can be done without needing to fix the current bug in 7.2. I just can't seem to find any configuration settings to do this, or anyone that knows what they are. Mark.

On 5/Apr/20 21:24, Randy Bush wrote:

ok, if IS-IS is kinda working on FRR, at least enough to get loopbacks and external interfaces around a pop, i gotta ask.

Not for me. I can't get it to even start, i.e., no link adjacencies due to an inability to agree on MTU. So if anyone has IS-IS on FRR running at all, please reach out. Mark.

On Sun, Apr 5, 2020 at 10:52 PM Saku Ytti <saku@ytti.fi> wrote:

The only thing that is larger in your network is hellos, and I'm not even sure how that works, considering 802.3 cannot signal larger frames than 1500B.

Probably this method: https://en.wikipedia.org/wiki/EtherType#Jumbo_frames

On Mon, Apr 6, 2020, at 07:51, Saku Ytti wrote:

I'm not sure what 'globally our IS-IS domain runs 8000 bytes' means. Your LSP MTU is like 1492B, there isn't a mechanism to fragment and reassemble LSP in-transit. ISIS network doesn't support different MTU sizes and I've not heard anyone being brave enough to increase LSP MTU above 1492B.

I won't speak for Mark, but NO, when you're carrying somebody's else's traffic you do your best to have the MTU on each and every backbone link "high enough" : preferably in the 9200(bytes) range, so you can easily transport 9000(bytes) client packets, and by no means so small that you need to fragment 1500(IP)/1514(Eth) byte packets. If things are really-really bad, 1600 bytes towards the edge.

The only thing that is larger in your network is hellos, and I'm not even sure how that works, considering 802.3 cannot signal larger frames than 1500B.

Ethernet cannot signal MTU. But if you have equipment at both sides of a P2P link, you don't need any signalling. You check the MTU suits your needs and put it in statically. Same for NNIs : the MTU is signalled in a document called "contract" or "agreement". And no, the guy that is being woken up at 3am for an issue, if he/she doesn't know that we run Jumbo, then he/she should start updating the CV. Back to the original question, I would expect FRR to be able to manually specify a MTU/frame-size, like any other decent NOS (even if it's not a full NOS).

On 6/Apr/20 10:18, Saku Ytti wrote:

And to your original question, no, nothing in Mark's ISIS network is above 1500, for the same reason.

Ummh, not really. As mentioned, we run 9,178 bytes on IOS and IOS XE, and 9,192 bytes on Junos and IOS XR, in our network. The 8,000 bytes we run is done only on the IOS, IOS XE and CSR1000v boxes. It doesn't mean that we are trying to transport IS-IS PDU's at 8,000 bytes. It just means we can, and are capping it to just about what VMware ESXi can support. Mark.

On 6/Apr/20 11:25, Saku Ytti wrote:

Quite and then you specified but for ISIS you run 8000. I'm only talking about your ISIS here, not rest. And that 8k doesn't do anything useful,

Except for our CSR1000v on ESXi. We only ever needed it when we went that route. We didn't need it prior to that.

so you can run 1500B to your FRR and not lose anything.

Which is my point - there is nowhere in FRR to set this, that is documented or has worked from experiment/general knowledge.

As I explained this is not true, by standard ISIS cannot do this for

...

1500B MTU. As ISIS is 802.3 and 802.3 cannot send larger frames. Cisco has proprietary hack to do EthernetII for hellos, but even that doesn't guarantee anything, because ISIS will still come up, even if it receives subMTU hello as 802.3.

Again, this is only required on our CSR1000v ESXi deployments, because our experience was that IS-IS would not establish because neighbors were running at a different MTU from the RR's, and IS-IS on the RR's was trying to setup adjacencies at a physically higher MTU than ESXi could support. So lowering it fixed it, and we just picked 8,000. We could have picked 2,000, or 1,500, or whatever. The actual number doesn't matter, as long as it's the same on the boxes that need it. We spent about 6hrs troubleshooting this back in 2014, so we didn't put it in just for fun. Mark.

On Mon, Apr 6, 2020, at 10:58, Mark Tinka wrote:

Within our core, we run 9,178 bytes (which translates to 9,192 bytes on Junos and IOS XR), to support the transport of Jumbo frames for

Beware of bad dog^H^H^H NCS model. On NCS5000 (don't know about 5500 - those arrived months after me leaving $job[-1]) you will get the nasty surprise of discovering that they have some limits to 9150(IP)/9164(eth), even if you can set the mtu higher (to an unusable 9192 or 92zz bytes).

On 6/Apr/20 07:51, Saku Ytti wrote:

I'm not sure what 'globally our IS-IS domain runs 8000 bytes' means. Your LSP MTU is like 1492B, there isn't a mechanism to fragment and reassemble LSP in-transit. ISIS network doesn't support different MTU sizes and I've not heard anyone being brave enough to increase LSP MTU above 1492B. The only thing that is larger in your network is hellos, and I'm not even sure how that works, considering 802.3 cannot signal larger frames than 1500B.

I mean that every router and switch interface in our network runs at 9,178 bytes. Ordinarily, we wouldn't need to do anything to IS-IS as it would just take that and run with it. However, VMware ESXi does not support anything larger than 9,000 bytes, and that is where we run our RR's. So to make it all harmonious, while the actual data plane in our routers and switches can forward up to 9,178 bytes, we set IS-IS MTU to 8,000 bytes on all of them, as a lowest common denominator so that our RR's can participate in the IS-IS domain. It's got nothing to do with trying to transport CLNS frames or fragment them, since as you rightly point out, fragmentation is not supported at Layer 2. Mark.

On 6/Apr/20 11:05, Saku Ytti wrote:

And the only thing this 'ISIS MTU' (think you mean CLNS MTU)...

Yes, I mean "clns mtu".

does, is for some cases make ISIS hellos larger. If you don't pad ISIS hellos, or if you have standard compliant ISIS, it doesn't do anything past 1500B.

Agreed. But in the network devices, IS-IS will infer the interface MTU to setup adjacencies. It doesn't matter what it is, it just needs to either be the same or low enough for the interface to support it.

More accurately ISIS doesn't have a way to fragment LSP into two LSP, so the whole ISIS network needs the same LSP-MTU, which has nothing to do with 'clns mtu', LSP-MTU does not increase with increasing CLNS MTU. There is no utility of having a harmonised CLNS MTU,...

Do you run CSR1000v on ESXi?

so if you want to speak to FRR, you can just do 1500B and it's fine.

Which is what this whole thread is about. How do I set that, manually, without changing the physical interface MTU? Mark.

On 6/Apr/20 11:28, Saku Ytti wrote:

You want to run your physical at high MTU and you also like ISIS to come up?

The services running on the server benefit from the high MTU. That's the whole point of the server... to run the services, not to run a routing protocol. So I don't want to lower the physical MTU because that will impact the performance of the services on the server that depend on it. For OSPF on Quagga, we just run "ip ospf mtu-ignore" and "ipv6 ospf6 mtu-ignore" to get around this, and this works fine.

FRR doesn't seem to support Cisco proprietary EthernetII based ISIS, so it'll only do 802.3 standard. Have you tried to just use what ever interface MTU, and NOT pad on far-end cisco? If I read the code right, this should be then standard max_size 802.3 and it should come up.

Hello Padding is disabled by default in our IS-IS core, so the other side isn't doing it already. The problem you have is IS-IS on FreeBSD won't initialize because it sees the physical interface running at 9,000 bytes, and yet its code can only go up to 8,192 bytes. Fair point, that bug has been fixed and just needs to be committed to upcoming code. But again, if I can manually tell IS-IS to run at a lower MTU than what it is trying to infer from the interface, we shall be golden. How do I do that, is the question?

If you force Cisco to pad, it'll use the EthernetI hack, which is not supported by FRR, and it won't work. I noticed in lab just now, if I set 'pad sometimes' on IOS-XR, it won't pad at all for >1500B MTUs, so seems like 'sometimes' stops Cisco from using the EthernetII hack.

    "no hello padding" is default for our core. Mark.

From your original post:

2020/03/21 03:12:36 ISIS: isis_send_pdu_bcast: sock_buff size 8192 is less than output pdu size 9014 on circuit em0 2020/03/21 03:12:36 ISIS: [EC 67108865] ISIS-Adj (1): Send L2 IIH on em0 failed

Why is PDU 9014, if you don't have padding? I wonder what FRR even does at >1500B, I don't see '8870' in source code, so I don't think it supports the EthernetII hack. I wonder did someone verify the fix, to see what it even does? To me it looks like what it'll do, it'll use what ever ethernetII matches byte-size of frame, which is decidedly, not correct and would not interoperate after 'fix'. -- ++ytti

On 6/Apr/20 12:00, Saku Ytti wrote:

Aah found it, it does do Cisco hack.

https://github.com/FRRouting/frr/blob/58980443821edf95719984e01f31720bd1dc7f...

But as long as you don't pad, your PDU shouldn't exceed 1500B.

Good man, you gave me an idea. Ran through the documentation and found out how to deal with Hello Padding in FRR: interface em0  ip router isis 1  ipv6 router isis 1  no isis hello padding Looks like IS-IS now initializes: 2020/04/06 10:09:21 ISIS: Opened BPF device /dev/bpf0 2020/04/06 10:09:21 ISIS: BPF buffer len = 4096 2020/04/06 10:09:21 ISIS: failed to set BPF buffer len (4096 to 9000) 2020/04/06 10:09:21 ISIS: circuit already connected 2020/04/06 10:09:21 ISIS: circuit already connected 2020/04/06 10:09:21 ISIS: circuit already connected Let me continue with the configuration and report back. Mark.

On 6/Apr/20 12:31, Saku Ytti wrote:

So FRR should have an addition of LSP-MTU which should default to 1492B to avoid interoperability issues when it must generate large LSP PDU.

A couple of weeks ago, my Google-fu led to me some kind of "lsp-mtu" command for FRR. I tried it everywhere but it wasn't supported.

So better not make config where FRR needs to inject larger LSP PDU, might be more excitement than what people would like. Someone can test what happens when you redistribute more prefixes than can fit in 1492B LSP PDU and if those LSPs propagate to Ciscos and Juniper, blackhole or crash the network.

On these servers, I'm pushing only 2 routes into the IS-IS domain. Mark.

On 6/Apr/20 12:47, Saku Ytti wrote:

Good good, if you are particularly concerned, match 8870 etype, len>1500 and drop on upstream router :). So should someone do something funky on your FRR, at least that's not the moment you test what your rest of the network think of large LSPs.

:-). First, let me get the adjacencies up. FRR seems to have its own ideas about what HMAC-MD5 is vs. the upstream IOS XE number. Mark.

On 6/Apr/20 11:48, Saku Ytti wrote:

Ok, and because this particular FRR VM does more than just ISIS, you want the extra mtu between 9k and 8192?

Yes sir. FRR is just another service running on the box, mainly to pull traffic toward it.

Change MTU after starting ISIS? :>

Hehe, I'm old now - I try to look for excitement outside of network operations :-). Mark.

On 6/Apr/20 12:57, Dale Shaw wrote:

Since vsphere67u3 (ESXi 6.7 U3), you can set an MTU of up to 9,190 bytes:

So we moved from 6.0 to 6.7 last year, when we refreshed the servers. Not sure why they hid the fact that they can breach the 9,000 mark:     https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc...     https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.networking....

The host UI doesn't let you go >9000 bytes so (for now) you need to use the CLI.

Yeaaaaah... RR's are critical infrastructure. I'd prefer not to mess with them beyond what is necessary :-).

Are your FRR VM vNICs attached to standard or distributed vswitch port groups?

So we only worry about this on our RR's. Other VM's that may be running are not a huge concern. Despite our network being able to carry 9,192 bytes, we only guarantee 9,000 for services and customers, end-to-end. Same applies to internal services we operate. Mark.