I believe we turned bgp-error-tolerance on after this Vulnerability Note from CERT: https://www.kb.cert.org/vuls/id/347067 Rereading it, that sounds like what everyone is reporting here, or at least extremely similar. John Stitt ________________________________ From: Niels den Otter via NANOG <nanog@lists.nanog.org> Sent: Tuesday, May 20, 2025 8:39 AM To: nanog@lists.nanog.org <nanog@lists.nanog.org> Cc: Niels den Otter <niels.denotter@surf.nl> Subject: Re: BGP malformed update/attribute list Hello Simon, We have seen the same in our network (was a BGP update for a specific /40 (!) IPv4 prefix as far as we can see). In Juniper speak 'bgp-error-tolerance' keeps the BGP sessions up, but downstream networks might still suffer from this. Regards, Niels _______________________________________________ NANOG mailing list https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.nanog.org%2Farchives%2Flist%2Fnanog%40lists.nanog.org%2Fmessage%2FCDMNBBMC54JZZZOXNL7A26Q27BVTQMFO%2F&data=05%7C02%7Cjstitt%40hop-electric.com%7Cd8603c209d2640b57fc308dd97a524f0%7C7707c291b2534ee2bcd6557cdf0fea43%7C0%7C0%7C638833457754493208%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=1%2F1SMJ7qEbo0tmCCn66OyeRLYzNhWhf25YnPp3WhpGc%3D&reserved=0<https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/CDMNBBMC54JZZZOXNL7A26Q27BVTQMFO/> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you are not expecting this message contact the sender directly via phone/text to verify.

For what it's worth, while we didn't see sessions reset, I do recall that we have bgp-error-tolerance enabled on our Juniper routers, so it's possible that kept our sessions from resetting. Customer that saw sessions drop is running Arista. John Stitt ________________________________ From: Ryan Rawdon via NANOG <nanog@lists.nanog.org> Sent: Tuesday, May 20, 2025 8:46 AM To: North American Network Operators Group <nanog@lists.nanog.org> Cc: Ryan Rawdon <ryan@u13.net> Subject: Re: BGP malformed update/attribute list On 2025-05-20 09:31, Simon Lockhart via NANOG wrote:

Did anyone see BGP flaps this morning at about 07:01 UTC as a result of BGP malformed update?

I was just mentioning this with you on IRC, here's the details of what we saw. We were most-impacted in Virginia at our two locations there, 11 of 12 transit IPv4 sessions (3 uptream ASNs) failed. IPv6 was unaffected. First instance at May 20 07:01:51 Last instance of that message in our logs was at May 20 07:08:08 Other regions across our global locations saw it, but only one or two sessions/upstreams here and there.

It flapped one of our iBGP sessions:

May 20 08:01:51.150 BST: %BGP-3-NOTIFICATION: received from neighbor XXX.XXX.XXX.XXX 3/1 (update malformed) 31 bytes E0281C00 00000000 00000000 00000000 00

Another ISP saw the same thing...

code 3 (Update Message Error) subcode 1 (invalid attribute list), Data: e0 28 1c 00 00 00

Is there a new BGP rogue update out there?

Simon _______________________________________________ NANOG mailing list https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.nanog.org%2Farchives%2Flist%2Fnanog%40lists.nanog.org%2Fmessage%2FGQP6V6BONTN2BPD7XSGW27WLZE5F3L7K%2F&data=05%7C02%7Cjstitt%40hop-electric.com%7C3686ee1866d749ada05408dd97a4cbfe%7C7707c291b2534ee2bcd6557cdf0fea43%7C0%7C0%7C638833456248584078%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=qUlr65OWUt6HXJYNgHCbWIIH%2B4%2By8ciuKSfrFRmsz6k%3D&reserved=0<https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/GQP6V6BONTN2BPD7XSGW27WLZE5F3L7K/>

_______________________________________________ NANOG mailing list https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.nanog.org%2Farchives%2Flist%2Fnanog%40lists.nanog.org%2Fmessage%2FL7KYGTXIHKBS3RSGZFCCXB6KEFBMDRBC%2F&data=05%7C02%7Cjstitt%40hop-electric.com%7C3686ee1866d749ada05408dd97a4cbfe%7C7707c291b2534ee2bcd6557cdf0fea43%7C0%7C0%7C638833456248610067%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=1%2FXDLCU2W20dDZe1ktzC3V9hvj94DfEHMS1Puptd%2FRY%3D&reserved=0<https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/L7KYGTXIHKBS3RSGZFCCXB6KEFBMDRBC/> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you are not expecting this message contact the sender directly via phone/text to verify.

I’m happy “bgp-error-tolerance” nowadays is enabled by in modern Junos! :-)

On 2025-05-20 09:48, John Stitt wrote:

For what it's worth, while we didn't see sessions reset, I do recall that we have bgp-error-tolerance enabled on our Juniper routers, so it's possible that kept our sessions from resetting.

Customer that saw sessions drop is running Arista.

John Stitt

Arista appears to have made changes for bug 899981 to drop (at least) certain malformed attributes. Was anyone running EOS versions greater than the following, and had Arista EOS reset the sessions today? We are unsure whether the change made in these versions would have provided a cleaner response to today's malformed attribute. 4.28.11+ 4.29.8+ 4.30.6+ 4.31.2+

so interested in responses to this.

No snark intended here, but unclear what sort of response you would be looking for. RFC4271 : Sec 6, Error Handling

When any of the conditions described here are detected, a NOTIFICATION message, with the indicated Error Code, Error Subcode, and Data fields, is sent, and the BGP connection is closed (unless it is explicitly stated that no NOTIFICATION message is to be sent and the BGP connection is not to be closed). If no Error Subcode is specified, then a zero MUST be used.

Section 6.3 , Update Message Error Handling All errors detected while processing the UPDATE message MUST be

indicated by sending the NOTIFICATION message with the Error Code UPDATE Message Error. The error subcode elaborates on the specific nature of the error.

A standard BGP implementation that detects an error MUST close the session per the spec. This was updated by RFC7606, Revised Error Handling for BGP, which provides for alternative ways to handle these errors, depending on what they are. Not all require a session reset. https://datatracker.ietf.org/doc/html/rfc7606 If your vendor's BGP implementation isn't RFC7606 compliant, then it's likely in your best interests to ask them to support it. The possibility of receiving a malformed update is omnipresent, and 7606 support can dramatically reduce the impact of such things when they do occur. On Tue, May 20, 2025 at 12:28 PM Chris Costa via NANOG < nanog@lists.nanog.org> wrote:

Did see an impact on some 7280R3 switches running EOS 4.29.7, so interested in responses to this.

On Tue, May 20, 2025 at 7:32 AM Ryan Rawdon via NANOG < nanog@lists.nanog.org> wrote:

...

On 2025-05-20 09:48, John Stitt wrote:

...

For what it's worth, while we didn't see sessions reset, I do recall that we have bgp-error-tolerance enabled on our Juniper routers, so it's possible that kept our sessions from resetting.

Customer that saw sessions drop is running Arista.

John Stitt

Arista appears to have made changes for bug 899981 to drop (at least) certain malformed attributes. Was anyone running EOS versions greater than the following, and had Arista EOS reset the sessions today? We are unsure whether the change made in these versions would have provided a cleaner response to today's malformed attribute.

4.28.11+

4.29.8+

4.30.6+

4.31.2+ _______________________________________________ NANOG mailing list

https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/FXGV24EY...

...

_______________________________________________ NANOG mailing list

https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/KXFOVLDN...

-------- Original Message -------- *Subject: *BGP malformed update/attribute list *From: *Simon Lockhart *To: *North American Network Operators Group *Cc: *"Marco Davids (Private)" *Date: *Wed, 21 May 2025 09:53:03 +0100

...

Might be clearer to stick with UTC on global lists like this.

If you hadn't been so aggressive with your cropping, you would have noticed that I had, indeed, quoted the timestamp in UTC in the body of my message.

I stand corrected - thank you. -- Marco

Hallo Randy, That's interesting. At exact the same moment this is what our Juniper routers reported; --- May 20 07:01:51 router rpd[34930]: %DAEMON-4: bgp_read_v4_update:13937: NOTIFICATION sent to a.b.c.d (Internal AS xxx): code 3 (Update Message Error) subcode 131 (invalid), Data: 00 00 00 00 00 00 May 20 07:01:51 router rpd[34930]: %DAEMON-3: Received malformed update from a.b.c.d (Internal AS xxx) May 20 07:01:51 router rpd[34930]: %DAEMON-3: Family inet-vpn-unicast, prefix a.b.c.d:32767:156.230.0.0/40 (label 114) May 20 07:01:51 router rpd[34930]: %DAEMON-3: Malformed Attribute PREFIX_SID(40) flag 0x80 length 28 error 131 (TLV length error). --- Appears to be another prefix? Unfortunately we don't have a BMP dump of this packet. * Niels ________________________________ Van: Randy Bush via NANOG <nanog@lists.nanog.org> Verzonden: woensdag 21 mei 2025 22:47 Aan: Simon Lockhart via NANOG <nanog@lists.nanog.org> CC: Randy Bush <randy@psg.com> Onderwerp: Re: BGP malformed update/attribute list just to aol, and other posts did not show full nlri May 20 07:01:51 r2.f00 16869308: RP/0/RSP0/CPU0:May 20 07:01:51.437 : bgp[1059]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 123.45.67.89 (VRF: default) - message length 106 bytes, error flags 0x000c0000, action taken "DiscardAttr". Error details: "Error 0x00040000, Field "Attr-length", Attribute 40 (Flags 0xe0, Length 28), Data [e0281c]". NLRIs: [IPv4 Unicast] 45.198.25.0/24 randy _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/56PKKMWI...

Hi Chris, Thanks for your detail information! Regarding the following message:

Message data: 144 bytes FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 00900200 00007540 01010040 02220208 00000D1C 0000232A 00002458 000210AA 00024F1E 00021B5D 000003D8 00060A11 400304D5 F2494980 04040000 0000C008 1C0D1C00 020D1C00 160D1C00 640D1C00 7B0D1C01 F70D1C03 850D1C08 13E0281C 00000000 00000000 00000000 00000000 00000000 00000000 00000000 182DC6B8

We can find the BGP attribute data that causes the problem as follows:

E0281C 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Semantically, this is a "complete" BGP Path Attribute: Flags=0xE0 Type=0x28, it is defined in RFC8669. (https://datatracker.ietf.org/doc/rfc8669/) Length=0x1C The Value field, as indicated by Length, does indeed occupy 28 bytes, although they are all zeros. Some network operating systems may try to parse the TLV carried in this attribute per RFC8669 and find that there is no valid TLV, so resulting in an error. Other operating systems found that the attribute was semantically correct but the content was incorrect, so they ignored the attribute and no BGP session interruption occurred. I'm curious how this strange attribute was generated. Was it the result of a test initiated by someone? Was it an attempt to test the robustness of the BGP protocol on the Internet? Cheers, Shunwan

-----Original Message----- From: Chris Welti via NANOG [mailto:nanog@lists.nanog.org] Sent: Thursday, May 22, 2025 8:09 PM To: North American Network Operators Group <nanog@lists.nanog.org> Cc: Niels den Otter <niels.denotter@surf.nl>; Chris Welti <chris.welti@switch.ch> Subject: Re: BGP malformed update/attribute list

Hi Niels,

For what it's worth, thats what we saw here on our AS3356 uplink:

    Total Update messages received: 281003910     Malformed Update messages received: 6       First received: May 20 09:01:52.256       Last  received: May 20 09:02:12.529 (2d04h ago)     Memory allocation failures: 0       First failure: ---       Last  failure: --- (never)     Error-handling session resets: 0       First reset: ---       Last  reset: --- (never)     Discarded attributes: 6

    Since session establishment:       Update messages received: 37579519       Final actions:         None: 0,  DiscardMsg: 0,  Reset: 0         TreatAsWdrOrReset: 0,  TreatAsWdr: 0,  DiscardAttr: 6         LocalRepair: 0

    Malformed messages stored: 5 (current index: 0)

      Malformed message #1         Received: May 20 09:02:12.529         Error flags: 0x00080000         Discarded attributes: 1         Final action: DiscardAttr

        Error elements: 1         [1] Error 0x00080000, Field "Attr-length", Attribute 40 (Flags 0xe0, Length 28)             Error data: [e0281c00] (4 bytes)             Action: DiscardAttr

        NLRIs: "IPv4 Unicast"  <15 chars>                140.150.9.0/24

        Reset/notification information:           Reason "None", Postit type "Update malformed"           Notification code 3, sub-code 1           Notification data [e0281c00000000000000000000000000] (16 bytes)

        Message data: 136 bytes           FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF           00880200 00006D40 01010040 021A0206           00000D1C 0000232A 00002458 000210AA           00024F1E 00021B5D 400304D5 F2494980           04040000 0000C008 1C0D1C00 020D1C00           160D1C00 640D1C00 7B0D1C01 F70D1C03           850D1C08 13E0281C 00000000 00000000           00000000 00000000 00000000 00000000           00000000 188C9609

      Malformed message #2         Received: May 20 09:02:12.529         Error flags: 0x00080000         Discarded attributes: 1         Final action: DiscardAttr

        Error elements: 1         [1] Error 0x00080000, Field "Attr-length", Attribute 40 (Flags 0xe0, Length 28)             Error data: [e0281c00] (4 bytes)             Action: DiscardAttr

        NLRIs: "IPv4 Unicast"  <68 chars>                138.113.116.0/24 163.171.104.0/24 163.1                71.102.0/24 163.171.103.0/24

        Reset/notification information:           Reason "None", Postit type "Update malformed"           Notification code 3, sub-code 1           Notification data [e0281c00000000000000000000000000] (16 bytes)

        Message data: 152 bytes           FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF           00980200 00007140 01010040 021E0207           00000D1C 0000232A 00002458 000210AA           00024F1E 00021B5D 0000D6D2 400304D5           F2494980 04040000 0000C008 1C0D1C00           020D1C00 160D1C00 640D1C00 7B0D1C01           F70D1C03 850D1C08 13E0281C 00000000           00000000 00000000 00000000 00000000           00000000 00000000 188A7174 18A3AB68           18A3AB66 18A3AB67

      Malformed message #3         Received: May 20 09:02:10.106         Error flags: 0x00080000         Discarded attributes: 1         Final action: DiscardAttr

        Error elements: 1         [1] Error 0x00080000, Field "Attr-length", Attribute 40 (Flags 0xe0, Length 28)             Error data: [e0281c00] (4 bytes)             Action: DiscardAttr

        NLRIs: "IPv4 Unicast"  <109 chars>                103.87.71.0/24 103.160.154.0/24 103.87.                70.0/24 103.160.54.0/24 110.44.172.0/22                103.52.2.0/24 203.84.138.0/24...

        Reset/notification information:           Reason "None", Postit type "Update malformed"           Notification code 3, sub-code 1           Notification data [e0281c00000000000000000000000000] (16 bytes)

        Message data: 184 bytes           FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF           00B80200 00006D40 01010040 021A0206           00000D1C 0000232A 00002458 000210AA           00024F1E 00021B5D 400304D5 F2494980           04040000 0000C008 1C0D1C00 020D1C00           160D1C00 640D1C00 7B0D1C01 F70D1C03           850D1C08 13E0281C 00000000 00000000           00000000 00000000 00000000 00000000           00000000 18675747 1867A09A 18675746           1867A036 166E2CAC 18673402 18CB548A           18CB5489 18A014DE 1867A037 18CA38AC           186E2CAA 18673403

      Malformed message #4         Received: May 20 09:01:57.313         Error flags: 0x00080000         Discarded attributes: 1         Final action: DiscardAttr

        Error elements: 1         [1] Error 0x00080000, Field "Attr-length", Attribute 40 (Flags 0xe0, Length 28)             Error data: [e0281c00] (4 bytes)             Action: DiscardAttr

        NLRIs: "IPv4 Unicast"  <15 chars>                156.230.0.0/16

        Reset/notification information:           Reason "None", Postit type "Update malformed"           Notification code 3, sub-code 1           Notification data [e0281c00000000000000000000000000] (16 bytes)

        Message data: 139 bytes           FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF           008B0200 00007140 01010040 021E0207           00000D1C 0000232A 00002458 000210AA           00024F1E 00021B5D 000003D8 400304D5           F2494980 04040000 0000C008 1C0D1C00           020D1C00 160D1C00 640D1C00 7B0D1C01           F70D1C03 870D1C08 13E0281C 00000000           00000000 00000000 00000000 00000000           00000000 00000000 109CE6

      Malformed message #5         Received: May 20 09:01:57.312         Error flags: 0x00080000         Discarded attributes: 1         Final action: DiscardAttr

        Error elements: 1         [1] Error 0x00080000, Field "Attr-length", Attribute 40 (Flags 0xe0, Length 28)             Error data: [e0281c00] (4 bytes)             Action: DiscardAttr

        NLRIs: "IPv4 Unicast"  <16 chars>                45.198.184.0/24

        Reset/notification information:           Reason "None", Postit type "Update malformed"           Notification code 3, sub-code 1           Notification data [e0281c00000000000000000000000000] (16 bytes)

        Message data: 144 bytes           FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF           00900200 00007540 01010040 02220208           00000D1C 0000232A 00002458 000210AA           00024F1E 00021B5D 000003D8 00060A11           400304D5 F2494980 04040000 0000C008           1C0D1C00 020D1C00 160D1C00 640D1C00           7B0D1C01 F70D1C03 850D1C08 13E0281C           00000000 00000000 00000000 00000000           00000000 00000000 00000000 182DC6B8

Cheers, Chris

...

Hallo Randy,

That's interesting. At exact the same moment this is what our Juniper routers reported;

--- May 20 07:01:51 router rpd[34930]: %DAEMON-4: bgp_read_v4_update:13937: NOTIFICATION sent to a.b.c.d (Internal AS xxx): code 3 (Update Message Error) subcode 131 (invalid), Data: 00 00 00 00 00 00 May 20 07:01:51 router rpd[34930]: %DAEMON-3: Received malformed update from a.b.c.d (Internal AS xxx) May 20 07:01:51 router rpd[34930]: %DAEMON-3: Family inet-vpn-unicast, prefix a.b.c.d:32767:156.230.0.0/40 (label 114) May 20 07:01:51 router rpd[34930]: %DAEMON-3: Malformed Attribute PREFIX_SID(40) flag 0x80 length 28 error 131 (TLV length error). ---

Appears to be another prefix? Unfortunately we don't have a BMP dump of

On 22.05.2025 08:29, Niels den Otter via NANOG wrote: this packet.

...

* Niels

________________________________ Van: Randy Bush via NANOG <nanog@lists.nanog.org> Verzonden: woensdag 21 mei 2025 22:47 Aan: Simon Lockhart via NANOG <nanog@lists.nanog.org> CC: Randy Bush <randy@psg.com> Onderwerp: Re: BGP malformed update/attribute list

just to aol, and other posts did not show full nlri

May 20 07:01:51 r2.f00 16869308: RP/0/RSP0/CPU0:May 20 07:01:51.437 : bgp[1059]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE

message

...

received from neighbor 123.45.67.89 (VRF: default) - message length 106 bytes, error flags 0x000c0000, action taken "DiscardAttr". Error details: "Error 0x00040000, Field "Attr-length", Attribute 40 (Flags 0xe0, Length 28), Data [e0281c]". NLRIs: [IPv4 Unicast] 45.198.25.0/24

randy _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/56 PKKMWIL7WN5T2VQTDL7M23RFSZO6I3/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/JL S5CHUGXNY6C55ZA4SVQO6CJU6KBTG5/

---

NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/IGG5VK 7BADZMQLYRND6L7YKHK7FTHYAD/