RTBH and Flowspec Measurements - Stop guessing when the attack will over
I think most here know (way better than me) the concepts of DDoS, anomaly detection, and reactions. Some of the reactions that can be implemented to reduce the impact of an attack are Remote-Triggered BlackHole and FlowSpec Filtering. In theory, using FlowSpec would be possible to de source the trigger of that FlowSpec announcement receives the measurements of the Flowspec-Enforcer-Box the measurements of those rules. But in fact, considering FlowSpec-Enforcement as-a-service, I've never seen that happens between FlowSpec-RulesGenerator-Box and FlowSpec-Enforcer-Box that are operated by different organizations. (If some company does, please let me know.) So, in practical actions, the FlowSpec-RulesGenerator-Box needs to play a guessing game of how long will take until the attack ceases. - First, send that FlowSpec Filtering for 3 minutes. - After that initial timer expires and removing the FlowSpec Filtering, measure the Flows of his own equipment. - If the attack is still there, re-announce the FlowSpec Filter Rule for more 15 minutes. - Wait to expire again, if the attack is still there re-announce for more 30 minutes, and keep this on an eternal loop. The same occurs with Remote-Triggered-Blackhole. I need to remove it and feel it is still there. And every time I do that, small partial outages occur at the destination network. Have you already imagined if those who implemented the RTBH or FlowSpec could give you some feedback of how is the usage of that BH or FlowSpecDrop? I really still don't know how to do this... Or even know if already there is a solution to that and I'm trying to invent the wheel. What do you think about that? Any Ideas? -- Douglas Fernando Fischer Engº de Controle e Automação
On Feb 2, 2021, at 00:34, Douglas Fischer <fischerdouglas@gmail.com> wrote: Or even know if already there is a solution to that and I'm trying to invent the wheel. Many flow telemetry export implementations on routers/layer3 switches report both passed & dropped traffic on a continuous basis for DDoS detection/classification/traceback. It's also possible to combine the detection/classification/traceback & flowspec trigger functions. [Full disclosure: I work for a vendor of such systems.] -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
OK, but do you know any company the sells de Flowspec as a service, in the way that the Attack Identifications are not made by their equipment, just receiving de BGP-FlowSpec and applying that rules on that equipments... And even then give back to the customer some way to access those statistics? I just know one or two that do that, and(sadly) they do it on fancy web reports or PDFs. Without any chance of using that as structured data do feedback the anomaly detection tools to determine if already it is the time to remove that Flowsperc rule. What I'm looking for is something like: A) XML/JSON/CSV files streamed to my equipment from the Flowspec Upstream Equipments saying "Heepend that, that, and that." Almost in real time. B) NetFlow/IPFIX/SFlow streamed to my equipment from the Upstream Equipment, restricted to the DST-Address that matches to the IP blocks that were involved to the Flowspec or RTBH that I Annouced to then. C) Any other idea that does the job of gives me the visibility of what is happening with FlowSpec-rules, or RTBH on theyr network. Em seg., 1 de fev. de 2021 às 22:07, Dobbins, Roland < Roland.Dobbins@netscout.com> escreveu:
-- Douglas Fernando Fischer Engº de Controle e Automação
Personally, I would absolutely, positively, never ever under any circumstances provide access to a 3rd party company to push a FlowSpec rule or trigger RTBH on my networks. No way. You would be handing over a nuclear trigger and saying "Please break me at my earliest inconvenience." On Tue, Feb 2, 2021 at 5:56 AM Douglas Fischer <fischerdouglas@gmail.com> wrote:
Well... That is a point of view! And I must respect that. Against this position, there are several companies, including some tier 1, that sells this as an $extra$. About the "Please break me at my earliest inconvenience." part: I believe that the same type of prefix filtering that applies to Downstream-BGP-Routes applies to RTBH and Flowspec. So, exactly as in common BGP Route-Filtering: - If the network operator does it correctly, it should work correctly. - If the network operator deals with that without the needed skills, expertise, attention+devotion, wrong things will come up. But, this still does not helps to find a solution do an organization A that sends some flowspec our RTBH to organization B(presuming organization B will accept that), and organization B do some reports of what is match with that flowspec or RTBH. That, in my opinion, is the only way to stop guessing how long will an attack will last, and start to define the end of a flowspec/RTBH action based on real information related to that. I want to close the feedback loop. Em ter., 2 de fev. de 2021 às 13:07, Tom Beecher <beecher@beecher.cc> escreveu:
-- Douglas Fernando Fischer Engº de Controle e Automação
Hi, here is a Flowspec best practices document that I helped write that will hopefully help folks from shooting themselves in the foot http://m3aawg.org/flowspec-BP. As you stated, route policies can be applied to restrict what type of flowspec rules can or can’t be accepted. For example, only allow a rule from the Flowspec controller if it specifies a /32 destination IP and is tagged with a particular community, reject all else. Douglas, I think what you are looking for is DOTS: https://tools.ietf.org/html/rfc8811 DOTS has a data channel which allows the DOTS client and server to communicate telemetry about the attack. The RFC is pretty new. I don’t think that there are any companies that have implemented it yet. Hopefully this protocol will be adopted by DDoS mitigation companies soon. -Rich Compton From: NANOG <nanog-bounces+rich.compton=charter.com@nanog.org> on behalf of Douglas Fischer <fischerdouglas@gmail.com> Date: Tuesday, February 2, 2021 at 10:10 AM To: Tom Beecher <beecher@beecher.cc> Cc: NANOG list <nanog@nanog.org> Subject: [EXTERNAL] Re: RTBH and Flowspec Measurements - Stop guessing when the attack will over CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Well... That is a point of view! And I must respect that. Against this position, there are several companies, including some tier 1, that sells this as an $extra$. About the "Please break me at my earliest inconvenience." part: I believe that the same type of prefix filtering that applies to Downstream-BGP-Routes applies to RTBH and Flowspec. So, exactly as in common BGP Route-Filtering: - If the network operator does it correctly, it should work correctly. - If the network operator deals with that without the needed skills, expertise, attention+devotion, wrong things will come up. But, this still does not helps to find a solution do an organization A that sends some flowspec our RTBH to organization B(presuming organization B will accept that), and organization B do some reports of what is match with that flowspec or RTBH. That, in my opinion, is the only way to stop guessing how long will an attack will last, and start to define the end of a flowspec/RTBH action based on real information related to that. I want to close the feedback loop. Em ter., 2 de fev. de 2021 às 13:07, Tom Beecher <beecher@beecher.cc> escreveu: Personally, I would absolutely, positively, never ever under any circumstances provide access to a 3rd party company to push a FlowSpec rule or trigger RTBH on my networks. No way. You would be handing over a nuclear trigger and saying "Please break me at my earliest inconvenience." On Tue, Feb 2, 2021 at 5:56 AM Douglas Fischer <fischerdouglas@gmail.com<mailto:fischerdouglas@gmail.com>> wrote: OK, but do you know any company the sells de Flowspec as a service, in the way that the Attack Identifications are not made by their equipment, just receiving de BGP-FlowSpec and applying that rules on that equipments... And even then give back to the customer some way to access those statistics? I just know one or two that do that, and(sadly) they do it on fancy web reports or PDFs. Without any chance of using that as structured data do feedback the anomaly detection tools to determine if already it is the time to remove that Flowsperc rule. What I'm looking for is something like: A) XML/JSON/CSV files streamed to my equipment from the Flowspec Upstream Equipments saying "Heepend that, that, and that." Almost in real time. B) NetFlow/IPFIX/SFlow streamed to my equipment from the Upstream Equipment, restricted to the DST-Address that matches to the IP blocks that were involved to the Flowspec or RTBH that I Annouced to then. C) Any other idea that does the job of gives me the visibility of what is happening with FlowSpec-rules, or RTBH on theyr network. Em seg., 1 de fev. de 2021 às 22:07, Dobbins, Roland <Roland.Dobbins@netscout.com<mailto:Roland.Dobbins@netscout.com>> escreveu: On Feb 2, 2021, at 00:34, Douglas Fischer <fischerdouglas@gmail.com<mailto:fischerdouglas@gmail.com>> wrote: Or even know if already there is a solution to that and I'm trying to invent the wheel. Many flow telemetry export implementations on routers/layer3 switches report both passed & dropped traffic on a continuous basis for DDoS detection/classification/traceback. It's also possible to combine the detection/classification/traceback & flowspec trigger functions. [Full disclosure: I work for a vendor of such systems.] -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com<mailto:roland.dobbins@netscout.com>> -- Douglas Fernando Fischer Engº de Controle e Automação -- Douglas Fernando Fischer Engº de Controle e Automação E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Hey Rich! I'm in love with this RFC... It is not an easy one, so I did not understand it completely yet. But It is almost what I was thinking... Does anyone saw any docs about deploying it? Any software that implements it? Em ter., 2 de fev. de 2021 às 15:53, Compton, Rich A < Rich.Compton@charter.com> escreveu:
-- Douglas Fernando Fischer Engº de Controle e Automação
There have been a great many things predicated on "if someone does it properly". While that is a noble goal, reality tells us that more commonly people WON'T do it properly, so designing for that eventual certainty, to me, is more important. To your second point, I think it's a reasonable question to say if an operator doesn't have requisite skills, expertise, attention+devotion to run a thing, should they be running that thing in the first place? On Tue, Feb 2, 2021 at 12:08 PM Douglas Fischer <fischerdouglas@gmail.com> wrote:
Yep... But I remember the first concept of security: There is no real security on a single layer. So, considering That, FlowSpec should never be accepted directly by the FlowSpec-Enforcer-Box. It should be announced to another box, running other software than that one on the Perimeter, and filtering and refiltering should be done on both layers. Em qua., 3 de fev. de 2021 às 02:43, Hank Nussbacher <hank@interall.co.il> escreveu:
-- Douglas Fernando Fischer Engº de Controle e Automação
In between the FS-Enforcer and the network there should be an arbiter that is able to report, analyse, approve, ignore or rollback rules that are being pushed. Not sure if this already exsists. Verzonden vanuit Outlook<http://aka.ms/weboutlook> ________________________________ Van: NANOG <nanog-bounces+peterf.deboer=hotmail.com@nanog.org> namens Douglas Fischer <fischerdouglas@gmail.com> Verzonden: woensdag 3 februari 2021 10:59 Aan: Hank Nussbacher <hank@interall.co.il> CC: NANOG <nanog@nanog.org> Onderwerp: Re: RTBH and Flowspec Measurements - Stop guessing when the attack will over Yep... But I remember the first concept of security: There is no real security on a single layer. So, considering That, FlowSpec should never be accepted directly by the FlowSpec-Enforcer-Box. It should be announced to another box, running other software than that one on the Perimeter, and filtering and refiltering should be done on both layers. Em qua., 3 de fev. de 2021 às 02:43, Hank Nussbacher <hank@interall.co.il<mailto:hank@interall.co.il>> escreveu: On 02/02/2021 19:08, Douglas Fischer wrote: Well... That is a point of view! And I must respect that. Against this position, there are several companies, including some tier 1, that sells this as an $extra$. About the "Please break me at my earliest inconvenience." part: I believe that the same type of prefix filtering that applies to Downstream-BGP-Routes applies to RTBH and Flowspec. So, exactly as in common BGP Route-Filtering: - If the network operator does it correctly, it should work correctly. - If the network operator deals with that without the needed skills, expertise, attention+devotion, wrong things will come up. You forgot to mention software bugs: https://kb.juniper.net/InfoCenter/index?page=content&id=JSA11101&cat=SIRT_1&actp=LIST Note what Juniper states: Workaround: There are no viable workarounds for this issue -Hank But, this still does not helps to find a solution do an organization A that sends some flowspec our RTBH to organization B(presuming organization B will accept that), and organization B do some reports of what is match with that flowspec or RTBH. That, in my opinion, is the only way to stop guessing how long will an attack will last, and start to define the end of a flowspec/RTBH action based on real information related to that. I want to close the feedback loop. Em ter., 2 de fev. de 2021 às 13:07, Tom Beecher <beecher@beecher.cc><mailto:beecher@beecher.cc> escreveu: Personally, I would absolutely, positively, never ever under any circumstances provide access to a 3rd party company to push a FlowSpec rule or trigger RTBH on my networks. No way. You would be handing over a nuclear trigger and saying "Please break me at my earliest inconvenience." On Tue, Feb 2, 2021 at 5:56 AM Douglas Fischer <fischerdouglas@gmail.com<mailto:fischerdouglas@gmail.com>> wrote: OK, but do you know any company the sells de Flowspec as a service, in the way that the Attack Identifications are not made by their equipment, just receiving de BGP-FlowSpec and applying that rules on that equipments... And even then give back to the customer some way to access those statistics? I just know one or two that do that, and(sadly) they do it on fancy web reports or PDFs. Without any chance of using that as structured data do feedback the anomaly detection tools to determine if already it is the time to remove that Flowsperc rule. What I'm looking for is something like: A) XML/JSON/CSV files streamed to my equipment from the Flowspec Upstream Equipments saying "Heepend that, that, and that." Almost in real time. B) NetFlow/IPFIX/SFlow streamed to my equipment from the Upstream Equipment, restricted to the DST-Address that matches to the IP blocks that were involved to the Flowspec or RTBH that I Annouced to then. C) Any other idea that does the job of gives me the visibility of what is happening with FlowSpec-rules, or RTBH on theyr network. Em seg., 1 de fev. de 2021 às 22:07, Dobbins, Roland <Roland.Dobbins@netscout.com<mailto:Roland.Dobbins@netscout.com>> escreveu: On Feb 2, 2021, at 00:34, Douglas Fischer <fischerdouglas@gmail.com<mailto:fischerdouglas@gmail.com>> wrote: Or even know if already there is a solution to that and I'm trying to invent the wheel. Many flow telemetry export implementations on routers/layer3 switches report both passed & dropped traffic on a continuous basis for DDoS detection/classification/traceback. It's also possible to combine the detection/classification/traceback & flowspec trigger functions. [Full disclosure: I work for a vendor of such systems.] -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com<mailto:roland.dobbins@netscout.com>> -- Douglas Fernando Fischer Engº de Controle e Automação -- Douglas Fernando Fischer Engº de Controle e Automação -- Douglas Fernando Fischer Engº de Controle e Automação
In this case, in my opinion, I saw as the best scenario the FlowSpec Rules being announced from ASN-Customer to ASN-Flowspec-Enforcer - Not on a BGP Border of ASN-Flowspec-Enforcer. - But on a Central RR-Cluster of ASN-Flowspec-Enforcer. Em qua., 3 de fev. de 2021 às 07:36, Peter F. de Boer < peterf.deboer@hotmail.com> escreveu:
-- Douglas Fernando Fischer Engº de Controle e Automação
On Feb 3, 2021, at 17:01, Douglas Fischer <fischerdouglas@gmail.com> wrote: It should be announced to another box, running other software than that one on the Perimeter, and filtering and refiltering should be done on both layers. This is how the inter-operator implementations of which I'm aware function, via a policy broker. -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com>
Interesting, Do I read it right that there is no workaround, but the solution is to upgrade to an updated version which include the fix? The solution is just above the workaround. From the same page posted. https://kb.juniper.net/InfoCenter/index?page=content <https://kb.juniper.net/InfoCenter/index?page=content&id=JSA11101&cat=SIRT_1&actp=LIST> &id=JSA11101&cat=SIRT_1&actp=LIST Solution: The following software releases have been updated to resolve this specific issue: Junos OS: 15.1R7-S8, 15.1X49-D240, 17.3R3-S10, 17.4R2-S12, 17.4R3-S4, 18.1R3-S12, 18.2R2-S8, 18.2R3-S6, 18.3R3-S4, 18.4R1-S8, 18.4R2-S6, 18.4R3-S6, 19.1R2-S2, 19.1R3-S3, 19.2R3-S1, 19.3R2-S5, 19.3R3-S1, 19.4R1-S3, 19.4R2-S3, 19.4R3, 20.1R2, 20.2R1-S3, 20.2R2, 20.3R1-S1, 20.3R2, 20.4R1, and all subsequent releases. Junos OS Evolved: 20.3R1-S1-EVO, 20.3R2-EVO, 20.4R1-EVO, and all subsequent releases. It has a cvss score of 10.0 which is the highest. Is Juniper still vulnerable or not? Thanks <https://www.engardesecurite.ca/wp-content/uploads/2018/11/main1-1-214x300.gif> Jean St-Laurent CISSP #634103 ddosTest me security inc tel: <tel:+14388069800> 438 806-9800 site: <https://ddostest.me/> https://ddostest.me email: <mailto:jean@ddostest.me> jean@ddostest.me From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Hank Nussbacher Sent: February 3, 2021 12:41 AM To: nanog@nanog.org Subject: Re: RTBH and Flowspec Measurements - Stop guessing when the attack will over You forgot to mention software bugs: https://kb.juniper.net/InfoCenter/index?page=content <https://kb.juniper.net/InfoCenter/index?page=content&id=JSA11101&cat=SIRT_1&actp=LIST> &id=JSA11101&cat=SIRT_1&actp=LIST Note what Juniper states: Workaround: There are no viable workarounds for this issue -Hank But, this still does not helps to find a solution do an organization A that sends some flowspec our RTBH to organization B(presuming organization B will accept that), and organization B do some reports of what is match with that flowspec or RTBH. That, in my opinion, is the only way to stop guessing how long will an attack will last, and start to define the end of a flowspec/RTBH action based on real information related to that. I want to close the feedback loop. Em ter., 2 de fev. de 2021 às 13:07, Tom Beecher <mailto:beecher@beecher.cc> <beecher@beecher.cc> escreveu: Personally, I would absolutely, positively, never ever under any circumstances provide access to a 3rd party company to push a FlowSpec rule or trigger RTBH on my networks. No way. You would be handing over a nuclear trigger and saying "Please break me at my earliest inconvenience." On Tue, Feb 2, 2021 at 5:56 AM Douglas Fischer <fischerdouglas@gmail.com <mailto:fischerdouglas@gmail.com> > wrote: OK, but do you know any company the sells de Flowspec as a service, in the way that the Attack Identifications are not made by their equipment, just receiving de BGP-FlowSpec and applying that rules on that equipments... And even then give back to the customer some way to access those statistics? I just know one or two that do that, and(sadly) they do it on fancy web reports or PDFs. Without any chance of using that as structured data do feedback the anomaly detection tools to determine if already it is the time to remove that Flowsperc rule. What I'm looking for is something like: A) XML/JSON/CSV files streamed to my equipment from the Flowspec Upstream Equipments saying "Heepend that, that, and that." Almost in real time. B) NetFlow/IPFIX/SFlow streamed to my equipment from the Upstream Equipment, restricted to the DST-Address that matches to the IP blocks that were involved to the Flowspec or RTBH that I Annouced to then. C) Any other idea that does the job of gives me the visibility of what is happening with FlowSpec-rules, or RTBH on theyr network. Em seg., 1 de fev. de 2021 às 22:07, Dobbins, Roland <Roland.Dobbins@netscout.com <mailto:Roland.Dobbins@netscout.com> > escreveu: On Feb 2, 2021, at 00:34, Douglas Fischer <fischerdouglas@gmail.com <mailto:fischerdouglas@gmail.com> > wrote: Or even know if already there is a solution to that and I'm trying to invent the wheel. Many flow telemetry export implementations on routers/layer3 switches report both passed & dropped traffic on a continuous basis for DDoS detection/classification/traceback. It's also possible to combine the detection/classification/traceback & flowspec trigger functions. [Full disclosure: I work for a vendor of such systems.] -------------------------------------------- Roland Dobbins <roland.dobbins@netscout.com <mailto:roland.dobbins@netscout.com> > -- Douglas Fernando Fischer Engº de Controle e Automação -- Douglas Fernando Fischer Engº de Controle e Automação
Do I read it right that there is no workaround, but the solution is to upgrade to an updated version which include the fix?
"Upgrade to fixed code" is the most common solution for every vendor. To answer 'are they still vulnerable', IF someone is running one of the listed versions, AND they have flowspec enabled, there is exposure. On Wed, Feb 3, 2021 at 5:32 AM Jean St-Laurent via NANOG <nanog@nanog.org> wrote:
participants (7)
-
Compton, Rich A
-
Dobbins, Roland
-
Douglas Fischer
-
Hank Nussbacher
-
Jean St-Laurent
-
Peter F. de Boer
-
Tom Beecher