'show chassis fpc' might also be useful (or, at least easier :-)) W On Wed, May 04, 2022 at 4:43 PM, Saku Ytti <saku@ytti.fi> wrote:

Actually is this DPCE? 'show jtree N summary'

On Wed, 4 May 2022 at 23:39, Saku Ytti <saku@ytti.fi> wrote:

'show route summary' 'start shell pfe network fpcX' 'show jnh N pool summary' 'show jnh N pool usage'

On Wed, 4 May 2022 at 23:31, Nehul Patel <nehul.patel@gmail.com> wrote:

Hi NANOG,

We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis

If Someone has seen these errors before please suggest how to resolve it

May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid) May 4 12:42:01 last message repeated 4 times May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576 May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0 May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576 May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0 May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0 May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid) May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0 May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed) May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583 May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed

May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()

Any suggestions will be helpful

Please do let me know if you have any questions.

Regards and thanks, Nehul

-- ++ytti

-- ++ytti

Your line cards (not RE's) are running out of route-storage memory. As a short-term mitigation, you could try borrowing from segment 1, normally dedicated to filters, set chassis memory-enhanced route but this option may not exist in the version of JunOS you're running, which as already mentioned is very old. If the command is accepted, and it lets you commit, you'll then need to restart each of the FPC's, one at a time, by slot number, which will take each out of service for a few minutes, so you probably want to wait until a scheduled maintenance period, and start with less-important FPC slots first: request chassis fpc restart slot X On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:

Thank you Saku and the warren Here is the requested output

show route summary

inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1 hidden) Direct: 9 routes, 8 active Local: 8 routes, 8 active OSPF: 928 routes, 925 active BGP: 878686 routes, 878678 active Static: 2 routes, 2 active Aggregate: 15 routes, 12 active

inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden) LDP: 718 routes, 718 active

Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) Local: 1 routes, 1 active

mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden) MPLS: 3 routes, 3 active LDP: 387 routes, 387 active

inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0 hidden) Direct: 13 routes, 9 active Local: 10 routes, 10 active OSPF3: 16 routes, 15 active BGP: 286060 routes, 143030 active Static: 1 routes, 1 active

show chassis fpc Temp CPU Utilization (%) Memory Utilization (%) Slot State (C) Total Interrupt DRAM (MB) Heap Buffer 0 Online 31 11 0 1024 37 29 1 Online 31 11 0 1024 45 29 2 Online 30 4 0 1024 36 29

request pfe execute target fpc0 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x44817bb0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715144 bytes used GOT: 56880 bytes available (7168 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32519 pages used (2568 pages used in page alloc) GOT: 235 pages partially used GOT: 14 pages free (max contiguous = 6) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 26256 0 GOT: 16 14320 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2560 GOT: 3-3 1536 GOT: 6-6 3072 GOT: GOT: Fragmentation Index = 0.946, (largest free = 3072) GOT: Counters: GOT: 2643777 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 1095040 frees GOT: 0 holds GOT: 7 pending frees(pending bytes 56) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x448997f0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589552 bytes used GOT: 12185384 bytes available (12183552 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 5 pages partially used GOT: 23796 pages free (max contiguous = 23793) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 1416 0 GOT: 16 80 0 GOT: 48 336 32

request pfe execute target fpc1 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x447cc698) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715840 bytes used GOT: 56184 bytes available (8192 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32533 pages used (2568 pages used in page alloc) GOT: 219 pages partially used GOT: 16 pages free (max contiguous = 5) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 25544 0 GOT: 16 13312 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2048 GOT: 2-2 1024 GOT: 5-5 5120 GOT: GOT: Fragmentation Index = 0.954, (largest free = 2560) GOT: Counters: GOT: 2645725 allocs (0 failed) GOT: 2 releases(partial 0) GOT: 1096891 frees GOT: 0 holds GOT: 0 pending frees(pending bytes 0) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x4484e2d8) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589504 bytes used GOT: 12185432 bytes available (12184576 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 3 pages partially used GOT: 23798 pages free (max contiguous = 23798) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 424 0 GOT: 16 96 0 GOT: 48 336 32 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 27-32768 12184576 GOT: GOT: Fragmentation Index = 0.000, (largest free = 12184576) GOT: Counters: GOT: 45 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 0 frees GOT: 0 holds

-- Jordan.

Try setting "keep-none" on your BGP neighbor (s) not sure if it'll still need the cards rebooted equally you can also just accept a default route or wait for TAC to take over :) Regards Paschal Masha | Engineering Skype ID: paschal.masha From: "Nehul Patel" <nehul.patel@gmail.com> To: "Jordan" <jnh@vt11.net> Cc: "nanog" <nanog@nanog.org> Sent: Thursday, May 5, 2022 06:08:05 AM Subject: DMARC ViolationDKIM ViolationRe: Strange behavior on the Juniper MX240 Ok, thank you all for the feedback we are going to start with the Junos OS upgrade first on it but have to open the ticket with JTAC since currently on the juniper support website they have the Junos 15.1 is available so not sure we can directly jump from 10.4 to 15.1 maybe we have to do step by step upgrade on it. Any other suggestions will be helpful as well By the way, the uptime on the Juniper MX chassis was 1589 Days on it. On Wed, May 4, 2022 at 7:11 PM Jordan < jnh@vt11.net > wrote:

Your line cards (not RE's) are running out of route-storage memory. As a short-term mitigation, you could try borrowing from segment 1, normally dedicated to filters,

set chassis memory-enhanced route

but this option may not exist in the version of JunOS you're running, which as already mentioned is very old.

If the command is accepted, and it lets you commit, you'll then need to restart each of the FPC's, one at a time, by slot number, which will take each out of service for a few minutes, so you probably want to wait until a scheduled maintenance period, and start with less-important FPC slots first:

request chassis fpc restart slot X

On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:

...

Thank you Saku and the warren Here is the requested output

...

show route summary

...

inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1 hidden) Direct: 9 routes, 8 active Local: 8 routes, 8 active OSPF: 928 routes, 925 active BGP: 878686 routes, 878678 active Static: 2 routes, 2 active Aggregate: 15 routes, 12 active

...

inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden) LDP: 718 routes, 718 active

...

Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) Local: 1 routes, 1 active

...

mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden) MPLS: 3 routes, 3 active LDP: 387 routes, 387 active

...

inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0 hidden) Direct: 13 routes, 9 active Local: 10 routes, 10 active OSPF3: 16 routes, 15 active BGP: 286060 routes, 143030 active Static: 1 routes, 1 active

...

show chassis fpc Temp CPU Utilization (%) Memory Utilization (%) Slot State (C) Total Interrupt DRAM (MB) Heap Buffer 0 Online 31 11 0 1024 37 29 1 Online 31 11 0 1024 45 29 2 Online 30 4 0 1024 36 29

...

request pfe execute target fpc0 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x44817bb0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715144 bytes used GOT: 56880 bytes available (7168 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32519 pages used (2568 pages used in page alloc) GOT: 235 pages partially used GOT: 14 pages free (max contiguous = 6) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 26256 0 GOT: 16 14320 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2560 GOT: 3-3 1536 GOT: 6-6 3072 GOT: GOT: Fragmentation Index = 0.946, (largest free = 3072) GOT: Counters: GOT: 2643777 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 1095040 frees GOT: 0 holds GOT: 7 pending frees(pending bytes 56) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x448997f0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589552 bytes used GOT: 12185384 bytes available (12183552 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 5 pages partially used GOT: 23796 pages free (max contiguous = 23793) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 1416 0 GOT: 16 80 0 GOT: 48 336 32

...

request pfe execute target fpc1 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x447cc698) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715840 bytes used GOT: 56184 bytes available (8192 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32533 pages used (2568 pages used in page alloc) GOT: 219 pages partially used GOT: 16 pages free (max contiguous = 5) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 25544 0 GOT: 16 13312 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2048 GOT: 2-2 1024 GOT: 5-5 5120 GOT: GOT: Fragmentation Index = 0.954, (largest free = 2560) GOT: Counters: GOT: 2645725 allocs (0 failed) GOT: 2 releases(partial 0) GOT: 1096891 frees GOT: 0 holds GOT: 0 pending frees(pending bytes 0) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x4484e2d8) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589504 bytes used GOT: 12185432 bytes available (12184576 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 3 pages partially used GOT: 23798 pages free (max contiguous = 23798) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 424 0 GOT: 16 96 0 GOT: 48 336 32 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 27-32768 12184576 GOT: GOT: Fragmentation Index = 0.000, (largest free = 12184576) GOT: Counters: GOT: 45 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 0 frees GOT: 0 holds

-- Jordan.

Almost always direct upgrade works. If you ask TAC, they will likely suggest a formal process and you'll be doing many upgrades, which itself isn't actually something that is guaranteed to work (like in WRL9 case, but that is vmhost RE, not yours). And like Jordan said, you are out of resources but can extend them with the command given, which should give you more run rate. You may want to look in more detail how long you can keep running DPCE until you're really out. On Thu, 5 May 2022 at 06:08, Nehul Patel <nehul.patel@gmail.com> wrote:

Ok, thank you all for the feedback we are going to start with the Junos OS upgrade first on it but have to open the ticket with JTAC since currently on the juniper support website they have the Junos 15.1 is available so not sure we can directly jump from 10.4 to 15.1 maybe we have to do step by step upgrade on it. Any other suggestions will be helpful as well

By the way, the uptime on the Juniper MX chassis was 1589 Days on it.

On Wed, May 4, 2022 at 7:11 PM Jordan <jnh@vt11.net> wrote:

...

Your line cards (not RE's) are running out of route-storage memory. As a short-term mitigation, you could try borrowing from segment 1, normally dedicated to filters,

set chassis memory-enhanced route

but this option may not exist in the version of JunOS you're running, which as already mentioned is very old.

If the command is accepted, and it lets you commit, you'll then need to restart each of the FPC's, one at a time, by slot number, which will take each out of service for a few minutes, so you probably want to wait until a scheduled maintenance period, and start with less-important FPC slots first:

request chassis fpc restart slot X

On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:

...

Thank you Saku and the warren Here is the requested output

show route summary

inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1 hidden) Direct: 9 routes, 8 active Local: 8 routes, 8 active OSPF: 928 routes, 925 active BGP: 878686 routes, 878678 active Static: 2 routes, 2 active Aggregate: 15 routes, 12 active

inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden) LDP: 718 routes, 718 active

Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) Local: 1 routes, 1 active

mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden) MPLS: 3 routes, 3 active LDP: 387 routes, 387 active

inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0 hidden) Direct: 13 routes, 9 active Local: 10 routes, 10 active OSPF3: 16 routes, 15 active BGP: 286060 routes, 143030 active Static: 1 routes, 1 active

show chassis fpc Temp CPU Utilization (%) Memory Utilization (%) Slot State (C) Total Interrupt DRAM (MB) Heap Buffer 0 Online 31 11 0 1024 37 29 1 Online 31 11 0 1024 45 29 2 Online 30 4 0 1024 36 29

request pfe execute target fpc0 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x44817bb0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715144 bytes used GOT: 56880 bytes available (7168 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32519 pages used (2568 pages used in page alloc) GOT: 235 pages partially used GOT: 14 pages free (max contiguous = 6) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 26256 0 GOT: 16 14320 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2560 GOT: 3-3 1536 GOT: 6-6 3072 GOT: GOT: Fragmentation Index = 0.946, (largest free = 3072) GOT: Counters: GOT: 2643777 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 1095040 frees GOT: 0 holds GOT: 7 pending frees(pending bytes 56) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x448997f0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589552 bytes used GOT: 12185384 bytes available (12183552 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 5 pages partially used GOT: 23796 pages free (max contiguous = 23793) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 1416 0 GOT: 16 80 0 GOT: 48 336 32

request pfe execute target fpc1 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x447cc698) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715840 bytes used GOT: 56184 bytes available (8192 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32533 pages used (2568 pages used in page alloc) GOT: 219 pages partially used GOT: 16 pages free (max contiguous = 5) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 25544 0 GOT: 16 13312 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2048 GOT: 2-2 1024 GOT: 5-5 5120 GOT: GOT: Fragmentation Index = 0.954, (largest free = 2560) GOT: Counters: GOT: 2645725 allocs (0 failed) GOT: 2 releases(partial 0) GOT: 1096891 frees GOT: 0 holds GOT: 0 pending frees(pending bytes 0) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x4484e2d8) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589504 bytes used GOT: 12185432 bytes available (12184576 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 3 pages partially used GOT: 23798 pages free (max contiguous = 23798) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 424 0 GOT: 16 96 0 GOT: 48 336 32 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 27-32768 12184576 GOT: GOT: Fragmentation Index = 0.000, (largest free = 12184576) GOT: Counters: GOT: 45 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 0 frees GOT: 0 holds

-- Jordan.

-- ++ytti

Ok got it saku we will got with the direct upgrade of it On Wed, May 4, 2022, 22:34 Saku Ytti <saku@ytti.fi> wrote:

Almost always direct upgrade works. If you ask TAC, they will likely suggest a formal process and you'll be doing many upgrades, which itself isn't actually something that is guaranteed to work (like in WRL9 case, but that is vmhost RE, not yours).

And like Jordan said, you are out of resources but can extend them with the command given, which should give you more run rate. You may want to look in more detail how long you can keep running DPCE until you're really out.

On Thu, 5 May 2022 at 06:08, Nehul Patel <nehul.patel@gmail.com> wrote:

...

Ok, thank you all for the feedback we are going to start with the Junos

OS upgrade first on it but have to open the ticket with JTAC since currently on the juniper support website they have the Junos 15.1 is available so not sure we can directly jump from 10.4 to 15.1 maybe we have to do step by step upgrade on it. Any other suggestions will be helpful as well

...

By the way, the uptime on the Juniper MX chassis was 1589 Days on it.

On Wed, May 4, 2022 at 7:11 PM Jordan <jnh@vt11.net> wrote:

...

Your line cards (not RE's) are running out of route-storage memory. As a short-term mitigation, you could try borrowing from segment 1, normally dedicated to filters,

set chassis memory-enhanced route

but this option may not exist in the version of JunOS you're running, which as already mentioned is very old.

If the command is accepted, and it lets you commit, you'll then need to restart each of the FPC's, one at a time, by slot number, which will take each out of service for a few minutes, so you probably want to wait until a scheduled maintenance period, and start with less-important FPC slots first:

request chassis fpc restart slot X

On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:

...

Thank you Saku and the warren Here is the requested output

show route summary

inet.0: 879635 destinations, 879649 routes (879634 active, 0

holddown, 1

...

...

...

hidden) Direct: 9 routes, 8 active Local: 8 routes, 8 active OSPF: 928 routes, 925 active BGP: 878686 routes, 878678 active Static: 2 routes, 2 active Aggregate: 15 routes, 12 active

inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden) LDP: 718 routes, 718 active

Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden) Local: 1 routes, 1 active

mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden) MPLS: 3 routes, 3 active LDP: 387 routes, 387 active

inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0 hidden) Direct: 13 routes, 9 active Local: 10 routes, 10 active OSPF3: 16 routes, 15 active BGP: 286060 routes, 143030 active Static: 1 routes, 1 active

show chassis fpc Temp CPU Utilization (%) Memory Utilization (%) Slot State (C) Total Interrupt DRAM (MB) Heap Buffer 0 Online 31 11 0 1024 37 29 1 Online 31 11 0 1024 45 29 2 Online 30 4 0 1024 36 29

request pfe execute target fpc0 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x44817bb0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715144 bytes used GOT: 56880 bytes available (7168 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32519 pages used (2568 pages used in page alloc) GOT: 235 pages partially used GOT: 14 pages free (max contiguous = 6) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 26256 0 GOT: 16 14320 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2560 GOT: 3-3 1536 GOT: 6-6 3072 GOT: GOT: Fragmentation Index = 0.946, (largest free = 3072) GOT: Counters: GOT: 2643777 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 1095040 frees GOT: 0 holds GOT: 7 pending frees(pending bytes 56) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x448997f0) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589552 bytes used GOT: 12185384 bytes available (12183552 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 5 pages partially used GOT: 23796 pages free (max contiguous = 23793) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 1416 0 GOT: 16 80 0 GOT: 48 336 32

request pfe execute target fpc1 command "show jtree 0 memory extensive" SENT: Ukern command: show jtree 0 memory extensive GOT: GOT: Jtree memory segment 0 (Context: 0x447cc698) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 16715840 bytes used GOT: 56184 bytes available (8192 bytes from free pages) GOT: 3024 bytes wasted GOT: 2168 bytes unusable GOT: 32768 pages total GOT: 32533 pages used (2568 pages used in page alloc) GOT: 219 pages partially used GOT: 16 pages free (max contiguous = 5) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 25544 0 GOT: 16 13312 0 GOT: 24 8352 2040 GOT: 32 352 0 GOT: 48 432 128 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 1-1 2048 GOT: 2-2 1024 GOT: 5-5 5120 GOT: GOT: Fragmentation Index = 0.954, (largest free = 2560) GOT: Counters: GOT: 2645725 allocs (0 failed) GOT: 2 releases(partial 0) GOT: 1096891 frees GOT: 0 holds GOT: 0 pending frees(pending bytes 0) GOT: 0 pending forced GOT: 0 times free blocked GOT: 0 sync writes GOT: Error Counters:- GOT: 0 bad params GOT: 0 failed frees GOT: 0 bad cookie GOT: GOT: Jtree memory segment 1 (Context: 0x4484e2d8) GOT: ------------------------------------------- GOT: Memory Statistics: GOT: 16777216 bytes total GOT: 4589504 bytes used GOT: 12185432 bytes available (12184576 bytes from free pages) GOT: 2248 bytes wasted GOT: 32 bytes unusable GOT: 32768 pages total GOT: 8967 pages used (8967 pages used in page alloc) GOT: 3 pages partially used GOT: 23798 pages free (max contiguous = 23798) GOT: GOT: Partially Filled Pages (In bytes):- GOT: Unit Avail Overhead GOT: 8 424 0 GOT: 16 96 0 GOT: 48 336 32 GOT: GOT: Free Page Lists(Pg Size = 512 bytes):- GOT: Page Bucket Avail(Bytes) GOT: 27-32768 12184576 GOT: GOT: Fragmentation Index = 0.000, (largest free = 12184576) GOT: Counters: GOT: 45 allocs (0 failed) GOT: 0 releases(partial 0) GOT: 0 frees GOT: 0 holds

-- Jordan.

-- ++ytti

On 5/5/22 04:11, Jordan wrote:

Your line cards (not RE's) are running out of route-storage memory. As a short-term mitigation, you could try borrowing from segment 1, normally dedicated to filters,

set chassis memory-enhanced route

but this option may not exist in the version of JunOS you're running, which as already mentioned is very old.

This feature was introduced in 10.4, so he should have it. And yes, it's only supported for DPC's (I-chip). Mark.

JUNOS Version JUNOS Base OS boot [10.4R9.2] JUNOS Base OS Software Suite [10.4R9.2] JUNOS Kernel Software Suite [10.4R9.2] JUNOS Crypto Software Suite [10.4R9.2] JUNOS Packet Forwarding Engine Support (M/T Common) [10.4R9.2] JUNOS Packet Forwarding Engine Support (MX Common) [10.4R9.2] JUNOS Online Documentation [10.4R9.2] JUNOS Voice Services Container package [10.4R9.2] JUNOS Border Gateway Function package [10.4R9.2] JUNOS Services AACL Container package [10.4R9.2] JUNOS Services LL-PDF Container package [10.4R9.2] JUNOS Services PTSP Container package [10.4R9.2] JUNOS Services Stateful Firewall [10.4R9.2] JUNOS Services NAT [10.4R9.2] JUNOS Services Application Level Gateways [10.4R9.2] On Wed, May 4, 2022 at 1:49 PM Paschal Masha <paschal.masha@ke.wananchi.com> wrote:

What JUNOS version are you running?

Regards Paschal Masha | Engineering Skype ID: paschal.masha

----- Original Message ----- From: "Nehul Patel" <nehul.patel@gmail.com> To: "nanog" <nanog@nanog.org> Sent: Wednesday, May 4, 2022 10:56:14 PM Subject: Strange behavior on the Juniper MX240

Hi NANOG,

We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis

If Someone has seen these errors before please suggest how to resolve it

May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid) May 4 12:42:01 last message repeated 4 times May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576 May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0 May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576 May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0 May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0 May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid) May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0 May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed) May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583 May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed

May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()

Any suggestions will be helpful

Please do let me know if you have any questions.

Regards and thanks, Nehul

----- On May 4, 2022, at 6:58 PM, Tony Wicks tony@wicks.co.nz wrote: Hi,

Dude, JunOS 10.4 end of support - 06/08/2014. You have an almost 8 years past end of Vendor support O/S still in production! No, just no.

Now I'm really interested in the uptime of that box... Thanks, Sabri

Nehul, He was running the 15 code train. I think 15.1R6.7. But don't take that as fact. I just know it was 15 for sure. ________________________________ From: Nehul Patel <nehul.patel@gmail.com> Sent: Thursday, May 5, 2022 6:40 PM To: Nick Olsen <Nick@141networks.com> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Strange behavior on the Juniper MX240 Hi Nick, Thank you for the feedback on it. Would you please let me know which Juno OS version he had installed on the MX Chassis that works with the extended memory command of it? On Thu, May 5, 2022 at 12:50 PM Nick Olsen <Nick@141networks.com<mailto:Nick@141networks.com>> wrote: Friend of mine had this issue recently on an MX chassis running DPC's and RE-2000's. The extend memory command others have mentioned worked for him. His instance drove us crazy for a bit. The device would learn a route, show that it was installed (show routes) but traffic to said prefix would bounce net unreachable. We even pushed a static just for S&G's and that still didn't resolve it. It was a single prefix that a customer had reported. Some things to consider, as others have mentioned. 1. IPv6 routes share the same space. And use more per-route. You can extend the life of this box (probably considerably) by dropping full tables for IPv6. Perhaps taking just a default (Same goes for v4). 2. It seems from your previous output that you're taking ~1 full v4 table. And 2x v6 tables. Do you really need a full table if you're only taking 1 v4 table? Consider switching to a default only? In my Colleagues case, he was taking 2 full tables of v4 and v6 until he hit the same issue. 3. While you're RE's could use a nice upgrade too. Your linecards are actually the problem here. If you move to anything > DPC you get the trio chipset with much more FIB space (2 Million routes I believe?). I'd consider new RE's and new line cards for this box. Which might also mean new switch fabric controllers.... Basically, we'd be talking a full overhaul sans the power supplies and chassis. 4. Consider taking a default + full routes. Then filtering > /24 (if you even have anything < /24 learned now) (/48 on IPv6). Start with the memory command first and see where that gets you. But keep a watchful eye out for this to happen again (as the DFZ grows). Eventually your only option will be to filter routes and rely on a default or upgrade. ________________________________ From: NANOG <nanog-bounces+nick=141networks.com@nanog.org<mailto:141networks.com@nanog.org>> on behalf of Nehul Patel <nehul.patel@gmail.com<mailto:nehul.patel@gmail.com>> Sent: Wednesday, May 4, 2022 3:56 PM To: nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Strange behavior on the Juniper MX240 Hi NANOG, We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis If Someone has seen these errors before please suggest how to resolve it May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid) May 4 12:42:01 last message repeated 4 times May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576 May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0 May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576 May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0 May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0 May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid) May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0 May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed) May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583 May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit() Any suggestions will be helpful Please do let me know if you have any questions. Regards and thanks, Nehul

On 5/5/22 21:50, Nick Olsen wrote:

His instance drove us crazy for a bit. The device would learn a route, show that it was installed (show routes) but traffic to said prefix would bounce net unreachable. We even pushed a static just for S&G's and that still didn't resolve it. It was a single prefix that a customer had reported.

Start with the memory command first and see where that gets you. But keep a watchful eye out for this to happen again (as the DFZ grows). Eventually your only option will be to filter routes and rely on a default or upgrade.

These are the reasons why I was saying that while there may be some commands to move FIB allocations around, it's a lot of admin. because the DFZ is very dynamic, and FIB programming issues due to lack of slots that affect different prefixes in different ways can have you chasing your tail for weeks before you figure things out. I think doing this should be a short-term solution as you make plans to get newer hardware. As a long-term strategy, it will tax day-to-day operations. Mark.

On 5/6/22 10:09, Saku Ytti wrote:

This seems like a strange position. The device has 16MB+16MB jtree segments. The first is IP, the second is filters (Broadly).

OP has 16MB of first used. OP has <5MB of second used.

What if the platform had originally shipped with a different balance between filters and IP, and OP would have never hit this problem?

It is easy to see in many scenarios filter growth is negligible toi 0, IP growth is not. OP would technically have 70% FIB growth left, so DFZ of about 1.7M, which puts him in the year >2030 (potentially far beyond, but at least that).

I view the recarving as fixing poorly dimensioned memory use. And had it shipped with more sensible carving this discussion didn't exist, and no one would suggest they are in any sort of tactical situation. Saying there is a problem is logical fallacy, what if your platform shipped carving of 1 prefix, and rest for filters, and you could do 50M+50M by config toggle. By your logic, this would be a tactical temporary fix. No, we need to understand what we are doing, what is the problem, what the solution is, we cannot categorically say this is a tactical fix.

My response is to be taken in the context of running a (large) network, and not the view of a single box. We have run into issues with platforms that have shipped with FIB's in favour of IPv4 and less for IPv6 and MPLS labels. Shifted around, you could give up whatever is left for IPv6 and ACL's to give more to IPv4, but you then end up losing native IPv6 scalability. And, of course, whatever other permutation you may think of that leaves you in a babysitting scenario for the protocol(s) assigned to peasantry. When considered against the backdrop of a (large) network, one has to also consider the FIB requirements for the IGP, MPLS label space, e.t.c. And not to mention that IPv6 will require more FIB space than IPv4, both for the IGP and BGP. I'd love to say people's ACL's are simple, but who knows what folk populate into every RADIUS PPPoE session that they think filters are a solution for? So yes, it is important to understand the limitations (or capabilities) of your specific platform, but also look at the overall picture of your entire backbone, and get a full understanding of what re-juggling FIB memory may mean in the short and long term; of course, bearing in mind that for some operators, short-term could also be 10 years or more. So all I'm saying is if there is a hack like this to help you delay moving to newer hardware, go for it. But know your hardware in the global context of your network, which will require a lot more attention to avoid getting caught out when you least expect it. I'd be remiss if I suggested that "implement, move on and forget" is a normal way to treat this hack. Mark.