We just had every single prefix in AS378 start being announced by AS2027. Every announcement by AS2027 is failing RPKI yet being propagated a bit. Is this yet another misbehaving device or an actual attack? Thanks, Hank
Same stuff (with our ASN and our prefixes) detected here, coming from AS2027 (Milkywan), for a short time... Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a écrit :
We just had every single prefix in AS378 start being announced by AS2027.
Every announcement by AS2027 is failing RPKI yet being propagated a bit. Is this yet another misbehaving device or an actual attack?
-- *Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l'émetteur*
Hank, all exact match for prefix length? Or longer subnets covering the whole? (Is this leakage of a optimizer or possibly censorship leakage?) On Sun, Oct 22, 2023, 1:03 PM Olivier Benghozi <olivier.benghozi@wifirst.fr> wrote:
Same stuff (with our ASN and our prefixes) detected here, coming from AS2027 (Milkywan), for a short time...
Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a écrit :
We just had every single prefix in AS378 start being announced by AS2027.
Every announcement by AS2027 is failing RPKI yet being propagated a bit. Is this yet another misbehaving device or an actual attack?
*Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l'émetteur*
From what bgp.tools can see it seems like they had a private asn in
Hey everybody, I run bgp.tools, (And had a extremely busy alerting engine for a few minutes) the path like so ``` 2027 4220270000 6696 6939 42615 212232 ``` This can be valid for a number of reasons, ( they might have been doing some homemade BGP confederation for example ), and I assume then that they had enabled some kind of private asn filter that had not quite done what they expected. I think what they are expecting was the part to look like this: ``` 2027 6696 6939 42615 212232 ``` However instead the private AS stripping function instead did this, and sent it to their customers/collector feeds: ``` 2027 ``` This then obviously made everything look like a BGP origin hijack to all of the route collectors and alerting systems. It's worth noting that bgp.tools saw this from more than MilkyWan directly, but also from what I can assume are their customers. But I don't see any indication this faulty routing information propagated anywhere else than that. ( To sort of backup the response that Vincent has already provided us) Hope this provides some interesting insight, and maybe some future heads up :) On Sun, Oct 22, 2023 at 10:04 PM Christopher Morrow <morrowc.lists@gmail.com> wrote:
Hank, all exact match for prefix length? Or longer subnets covering the whole? (Is this leakage of a optimizer or possibly censorship leakage?)
On Sun, Oct 22, 2023, 1:03 PM Olivier Benghozi <olivier.benghozi@wifirst.fr> wrote:
Same stuff (with our ASN and our prefixes) detected here, coming from AS2027 (Milkywan), for a short time...
Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a écrit :
We just had every single prefix in AS378 start being announced by AS2027.
Every announcement by AS2027 is failing RPKI yet being propagated a bit. Is this yet another misbehaving device or an actual attack?
Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l'émetteur
Hello everyone, I'm working for MilkyWan / AS2027 and I wanted to give you some explanations regarding this incident. Last week-end, during an upgrade on our network configuration, it appears that some prefixes were announced with an incorrect AS Path. Based on our analysis, none of these routes seem to have been announced anywhere, but to some route-collectors (RIPE RIS, BGP.Tools, Qrator, HE.net, NLNOG RING), and therefore didn't effectively end up in the DFZ. The issue was discovered quickly (in a few minutes) and corrected right away. The incident is now closed on our side; please reach out to us should you see anything proving otherwise. We deeply apologize for that and we can confirm it was not a BGP hijack attempt. Wishing you a very pleasant day. Vincent F. for Milkywan Team Le 2023-10-22 19:02, Olivier Benghozi a écrit :
Same stuff (with our ASN and our prefixes) detected here, coming from AS2027 (Milkywan), for a short time...
Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a écrit :
We just had every single prefix in AS378 start being announced by AS2027.
Every announcement by AS2027 is failing RPKI yet being propagated a bit. Is this yet another misbehaving device or an actual attack?
_Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l'émetteur_
Thanks for the transparency, Vincent. Are you able to share how the AS-Path became mangled to begin with? I’m assuming this was some kind of route optimizer, but maybe something else going on? On Mon, Oct 23, 2023 at 07:32 <vincent@milkywan.fr> wrote:
Hello everyone, I'm working for MilkyWan / AS2027 and I wanted to give you some explanations regarding this incident. Last week-end, during an upgrade on our network configuration, it appears that some prefixes were announced with an incorrect AS Path. Based on our analysis, none of these routes seem to have been announced anywhere, but to some route-collectors (RIPE RIS, BGP.Tools, Qrator, HE.net, NLNOG RING), and therefore didn't effectively end up in the DFZ. The issue was discovered quickly (in a few minutes) and corrected right away. The incident is now closed on our side; please reach out to us should you see anything proving otherwise.
We deeply apologize for that and we can confirm it was not a BGP hijack attempt.
Wishing you a very pleasant day.
Vincent F. for Milkywan Team
Le 2023-10-22 19:02, Olivier Benghozi a écrit :
Same stuff (with our ASN and our prefixes) detected here, coming from AS2027 (Milkywan), for a short time...
Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a écrit :
We just had every single prefix in AS378 start being announced by AS2027.
Every announcement by AS2027 is failing RPKI yet being propagated a bit. Is this yet another misbehaving device or an actual attack?
_Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l'émetteur_
Hello, Because we were migrating our default table containing the DFZ into a VRF, we had a BGP session between 2 routers terminating on one side in the main table and on the other in the VRF. We had to remove the no-export from our redistribution route-map because of this private eBGP peering. As we were concerned about reachability with transits and peerings on both sides, we tried to activate a route-leak between the main table and the newly installed VRF DFZ. However, as a route-leak doesn't retain BGP attributes, routes started to be learned in originate from the router containing the route-leak. So there was no hijacking on the DFZ. Moreover, my route collector listening on the DFZ did not identify any hijack during the entire migration. On the Transit and Peering side, we are subject to max-pref on the provider side, and we have a route-map in prefix-list specific to our AS + customer community. However, we consider Route Collectors as customers, who redistribute prefixes greater than or equal to /24 or /48, minus bogons, and based on no-export, we don't send our internal routes. Except that in addition to the route leak, the no-export was removed, which let through all IGP originate routes to our customers, and route collectors. The problem was quickly identified, we cut the route leak and stopped all transits and peering in main table to leave only the DFZ and work on the end of migration having only some LNS not configured in the good VRF, and finished the migration direction from the new VRF DFZ. So don't rely on the Route Collector, which is updated at the whim of the various operators, but compare what's also in the DFZ before firing up the mailing lists. What's more, route collectors are not necessarily configured in the same way as standard peers, depending on the operator. This remains a mistake on our part, which has only resulted in horrors on the monitoring route collectors and not on the DFZ routes. So Route Collectors don't behave at all like transit or peering, given the lack of max pref, prefix-list or RPKI. But hey, it's quicker to send a flaming mail before typing your show ip route, I agree ;) My 2 cents, Nicolas On 23/10/2023 16:21, Tyler Conrad wrote:
Thanks for the transparency, Vincent. Are you able to share how the AS-Path became mangled to begin with? I’m assuming this was some kind of route optimizer, but maybe something else going on?
On Mon, Oct 23, 2023 at 07:32 <vincent@milkywan.fr> wrote:
Hello everyone, I'm working for MilkyWan / AS2027 and I wanted to give you some explanations regarding this incident. Last week-end, during an upgrade on our network configuration, it appears that some prefixes were announced with an incorrect AS Path. Based on our analysis, none of these routes seem to have been announced anywhere, but to some route-collectors (RIPE RIS, BGP.Tools, Qrator, HE.net, NLNOG RING), and therefore didn't effectively end up in the DFZ. The issue was discovered quickly (in a few minutes) and corrected right away. The incident is now closed on our side; please reach out to us should you see anything proving otherwise.
We deeply apologize for that and we can confirm it was not a BGP hijack attempt.
Wishing you a very pleasant day.
Vincent F. for Milkywan Team
Le 2023-10-22 19:02, Olivier Benghozi a écrit : > Same stuff (with our ASN and our prefixes) detected here, coming from > AS2027 (Milkywan), for a short time... > > Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> > a écrit : > >> We just had every single prefix in AS378 start being announced by >> AS2027. >> >> Every announcement by AS2027 is failing RPKI yet being propagated a >> bit. >> Is this yet another misbehaving device or an actual attack? > > _Ce message et toutes les pièces jointes (ci-après le "message") > sont établis à l’intention exclusive des destinataires désignés. > Il contient des informations confidentielles et pouvant être > protégé par le secret professionnel. Si vous recevez ce message par > erreur, merci d'en avertir immédiatement l'expéditeur et de > détruire le message. Toute utilisation de ce message non conforme à > sa destination, toute diffusion ou toute publication, totale ou > partielle, est interdite, sauf autorisation expresse de l'émetteur_
participants (7)
-
Ben Cox -
Christopher Morrow -
Hank Nussbacher -
Nicolas VUILLERMET -
Olivier Benghozi -
Tyler Conrad -
vincent＠milkywan.fr