I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things: 1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports. Do these mirror others' observations with security issues and how abuse desks respond? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore. Matt On 4/28/20 11:57 AM, Mike Hammett wrote:
I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things:
1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports.
Do these mirror others' observations with security issues and how abuse desks respond?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
On Tue, 28 Apr 2020, Matt Corallo via NANOG wrote:
Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore.
Thats what SBL is for. -Dan
On Tue, Apr 28, 2020 at 08:45:12PM -0700, Dan Hollis wrote:
On Tue, 28 Apr 2020, Matt Corallo via NANOG wrote:
Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore.
Thats what SBL is for.
Do you recommend that we use a DNS blacklist to check every SSH and HTTPS connection attempt, about whether it should be filtered or not? Ultimately if there is scanning happening from an IP address delegated to someone, isn't their abuse@ responsible for handling the complaints? What are "real" issues? We have scanning happening on ssh, https, SIP, SMTP submission ports everyday. fail2ban does a good job blocking many of these, but ultimately should the scanning problem be ignored? Is nobody ultimately responsible to stop these hosts from scanning? Mukund
DDoS, hijacker, botnet C&C, compromised hosts, sufficiently-hard-to-deal-with phishing, etc are all things that carry real risk to services that are otherwise well-maintained (primarily in that many of the latter lead to the former). Nothing wrong with using or monitoring fail2ban, but if you’re spamming abuse contacts in an automated fashion (a pattern of misbehavior may be different) just because of some scanning, I recommend you fire your CSO (or get one). Matt
On Apr 28, 2020, at 22:13, Mukund Sivaraman <muks@mukund.org> wrote:
On Tue, Apr 28, 2020 at 08:45:12PM -0700, Dan Hollis wrote:
On Tue, 28 Apr 2020, Matt Corallo via NANOG wrote: Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore.
Thats what SBL is for.
Do you recommend that we use a DNS blacklist to check every SSH and HTTPS connection attempt, about whether it should be filtered or not?
Ultimately if there is scanning happening from an IP address delegated to someone, isn't their abuse@ responsible for handling the complaints? What are "real" issues?
We have scanning happening on ssh, https, SIP, SMTP submission ports everyday. fail2ban does a good job blocking many of these, but ultimately should the scanning problem be ignored? Is nobody ultimately responsible to stop these hosts from scanning?
Mukund
Hi Matt On Tue, Apr 28, 2020 at 11:02:04PM -0700, Matt Corallo wrote:
DDoS, hijacker, botnet C&C, compromised hosts, sufficiently-hard-to-deal-with phishing, etc are all things that carry real risk to services that are otherwise well-maintained (primarily in that many of the latter lead to the former). Nothing wrong with using or monitoring fail2ban, but if you’re spamming abuse contacts in an automated fashion (a pattern of misbehavior may be different) just because of some scanning, I recommend you fire your CSO (or get one).
It a fair game, that we the victim hosts should manually scan hundreds of reports generated due to traffic from automated bots from IP address block, so that things are easy for abuse@ contacts? I haven't come across a false positive report from our fail2ban instances on various servers (which it so far emails to our internal email address). It appears extremely unlikely for its reports to be false postitives - its detection method by parsing logs is identical to what a human would manually do too. I wouldn't call emailing its reports automatically to an abuse contact as "spamming". It is exactly what a human would do, and programmers/sysadmins love to automate. If an abuse report is incorrect, then it is fair to complain. Mukund
Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system. If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config. Matt
On Apr 28, 2020, at 23:22, Mukund Sivaraman <muks@mukund.org> wrote:
Hi Matt
On Tue, Apr 28, 2020 at 11:02:04PM -0700, Matt Corallo wrote: DDoS, hijacker, botnet C&C, compromised hosts, sufficiently-hard-to-deal-with phishing, etc are all things that carry real risk to services that are otherwise well-maintained (primarily in that many of the latter lead to the former). Nothing wrong with using or monitoring fail2ban, but if you’re spamming abuse contacts in an automated fashion (a pattern of misbehavior may be different) just because of some scanning, I recommend you fire your CSO (or get one).
It a fair game, that we the victim hosts should manually scan hundreds of reports generated due to traffic from automated bots from IP address block, so that things are easy for abuse@ contacts?
I haven't come across a false positive report from our fail2ban instances on various servers (which it so far emails to our internal email address). It appears extremely unlikely for its reports to be false postitives - its detection method by parsing logs is identical to what a human would manually do too.
I wouldn't call emailing its reports automatically to an abuse contact as "spamming". It is exactly what a human would do, and programmers/sysadmins love to automate.
If an abuse report is incorrect, then it is fair to complain.
Mukund
On Tue, Apr 28, 2020 at 11:40:16PM -0700, Matt Corallo wrote:
Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.
If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.
Yes we use that, and PermitRootLogin no and an AllowUsers list. I asked in my first email, if with security practices as above and use of fail2ban to filter attempts, should we just ignore this problem and think that nobody is ultimately reponsible to get rid of this activity?
From our perspecive, a dumb kid's attempts look no different to a botnet's and we cannot distinguish. We don't know what kind of customer/end user is generating this more than the party who has the IP block. An exploit of a vulnerability whether it is performed by a dumb kid or a botnet has similar consequences.
If this is generally about etiqutte of emailing abuse@, look at it from our (target's) point of view. Assume "Joe Company"'s IP addresses send nefarious scanning queries to our hosts. If we respond to their abuse@ contact with automated reports of such activity for TCP traffic, let's assume 10% of those reports are false-positives. Which actor is responsible for the worse etiquette here? Joe Company, whose IP block is reponsible for the nefarious scanning, or us who who are reporting these attempts using a program? We are a small company with no CFO, CTO, CSO, CXO. We have little resources to scan every attempt. We can ignore these attempts and turn a blind eye, or we can automate. If there's a false positive report from us, then use the stick and that would be fair. Mukund
On Wed, Apr 29, 2020 at 12:24:01PM +0530, Mukund Sivaraman wrote:
On Tue, Apr 28, 2020 at 11:40:16PM -0700, Matt Corallo wrote:
Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.
If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.
Yes we use that, and PermitRootLogin no and an AllowUsers list.
I asked in my first email, if with security practices as above and use of fail2ban to filter attempts, should we just ignore this problem and think that nobody is ultimately reponsible to get rid of this activity?
In theory, no. In practice, unfortunately, yes. The typical service provider has so much low-level "noise" going on that if everyone reported everything to them, they'd semi-literally drown. Certainly, there's no possible way they could economically handle all that abuse reporting -- hiring all the people to examine, determine the veracity of, and act upon the reports would cost a fortune, because you better believe there's no One True Format for automated abuse notifications, nor will there ever likely be one, so it's all humans, all the time. Now, you could argue that they should clean up their network so they don't have that volume of abuse reports coming in -- and you'd be right, in theory. But there's a *lot* of low-level stuff that it isn't practical to stop, in and of itself. Consider your own reports. What is it, exactly, that you expect a provider to do with your report of a few failed SSH login attempts to stop the activity? Say it's a residential ISP, or an IaaS provider. They have only a few very large hammers at their disposal -- they can (maybe) filter the destination port, filter your destination IP, or disconnect the customer. Any of those will very possibly result in a support call, or lost customer. That's a very large cost you're expecting them to pay for something which has, let's face it, cost you practically nothing. Forcing disconnection for a port scan is also, by the way, a *great* way to create an absolute gold-plated A+ denial-of-service: send in a plausible-looking report of shenanigans to the ISP of someone you don't like, and *boom* their Internet connection's cut off. WINNAH! So what are you left with, action-wise? An ISP could keep a tally of abuse reports by customer, and take action on whoever's at the top of the pile, but that would then require a large and expensive army of humans to receive, check, classify, and record all incoming abuse reports. Do *you* want to pay $1,000/month for your home Internet connection to cover the cost of all those extra ISP staff? Because, as I said before, there's no One True Format for reporting abuse, and there never will be. Not that it would work, anyway -- any sort of "threshold" system for abuse ends up being gamed, anyway. You only need to look at how Twitter users with an axe to grind gang up to send in malicious reports about some other Twitter they don't like, which trips Twitter's "lots of reports => autoban" logic, to see how that would end if you started applying it to Internet abuse reporting. Finally, because nobody is ever convinced by rhetoric, here's an appeal to your self-interest: "crying wolf" is never a good idea. In the event that you *do* have a real problem that needs to be dealt with some time in the future, do you want to have your e-mail address, IP address, and whatever else associated with a thousand previous GWF ("goober with firewall") reports? Any abuse desk who has seen your hundreds of previous unactionable reports will almost certainly round-file that important one, and then you're *really* up the creek sans paddle. Far better to keep your powder dry and be ready for when you actually need assistance from whoever you're contacting. - Matt
"What is it, exactly, that you expect a provider to do with your report of a few failed SSH login attempts to stop the activity?... disconnect the customer." Yes. Comcast does it. My wife's aunt and uncle had a compromised box on their network. They don't check their e-mail, so they didn't see the warnings from Comcast. They shut them off until the problem was resolved. " Forcing disconnection for a port scan is also, by the way, a *great* way to create an absolute gold-plated A+ denial-of-service: " Surely they have flow records showing suspicious activity from that customer. They may not confirm the specific IP being attacked, but they will see massive numbers of SSH, SMTP, SIP, etc. connections going out. It's likely if there's outbound activity of that nature and *someone* complained about it, not only were they a victim of it, but the activity is probably undesired by anyone else receiving it as well. " cost you practically nothing." You're right. An insecure Internet doesn't cost any of us anything. " there's no One True Format for automated abuse notifications" So then "let's" make one? No one can follow it if it doesn't exist. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Matt Palmer" <mpalmer@hezmatt.org> To: nanog@nanog.org Sent: Wednesday, April 29, 2020 6:48:51 AM Subject: Re: Abuse Desks On Wed, Apr 29, 2020 at 12:24:01PM +0530, Mukund Sivaraman wrote:
On Tue, Apr 28, 2020 at 11:40:16PM -0700, Matt Corallo wrote:
Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.
If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.
Yes we use that, and PermitRootLogin no and an AllowUsers list.
I asked in my first email, if with security practices as above and use of fail2ban to filter attempts, should we just ignore this problem and think that nobody is ultimately reponsible to get rid of this activity?
In theory, no. In practice, unfortunately, yes. The typical service provider has so much low-level "noise" going on that if everyone reported everything to them, they'd semi-literally drown. Certainly, there's no possible way they could economically handle all that abuse reporting -- hiring all the people to examine, determine the veracity of, and act upon the reports would cost a fortune, because you better believe there's no One True Format for automated abuse notifications, nor will there ever likely be one, so it's all humans, all the time. Now, you could argue that they should clean up their network so they don't have that volume of abuse reports coming in -- and you'd be right, in theory. But there's a *lot* of low-level stuff that it isn't practical to stop, in and of itself. Consider your own reports. What is it, exactly, that you expect a provider to do with your report of a few failed SSH login attempts to stop the activity? Say it's a residential ISP, or an IaaS provider. They have only a few very large hammers at their disposal -- they can (maybe) filter the destination port, filter your destination IP, or disconnect the customer. Any of those will very possibly result in a support call, or lost customer. That's a very large cost you're expecting them to pay for something which has, let's face it, cost you practically nothing. Forcing disconnection for a port scan is also, by the way, a *great* way to create an absolute gold-plated A+ denial-of-service: send in a plausible-looking report of shenanigans to the ISP of someone you don't like, and *boom* their Internet connection's cut off. WINNAH! So what are you left with, action-wise? An ISP could keep a tally of abuse reports by customer, and take action on whoever's at the top of the pile, but that would then require a large and expensive army of humans to receive, check, classify, and record all incoming abuse reports. Do *you* want to pay $1,000/month for your home Internet connection to cover the cost of all those extra ISP staff? Because, as I said before, there's no One True Format for reporting abuse, and there never will be. Not that it would work, anyway -- any sort of "threshold" system for abuse ends up being gamed, anyway. You only need to look at how Twitter users with an axe to grind gang up to send in malicious reports about some other Twitter they don't like, which trips Twitter's "lots of reports => autoban" logic, to see how that would end if you started applying it to Internet abuse reporting. Finally, because nobody is ever convinced by rhetoric, here's an appeal to your self-interest: "crying wolf" is never a good idea. In the event that you *do* have a real problem that needs to be dealt with some time in the future, do you want to have your e-mail address, IP address, and whatever else associated with a thousand previous GWF ("goober with firewall") reports? Any abuse desk who has seen your hundreds of previous unactionable reports will almost certainly round-file that important one, and then you're *really* up the creek sans paddle. Far better to keep your powder dry and be ready for when you actually need assistance from whoever you're contacting. - Matt
Enforcing rate limiting comes to mind. And if there is a blatant problem then very strict rate limiting to make even surfing yahoo news a pain is a good idea. Not to mention conn tracking and limiting to allow a customer to fix their problem is much better than a plain cut-off. The Oh my gawd!!! I’m being port scanned ... pffft it’s moot. There is blatant abuse and internet fuzz coming from legitimate sec corps that believe they are making the internet better by scanning your equipment without you asking them too and hoping to sell you services, and those are all just bullshit. -- J. Hellenthal The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.
On Apr 29, 2020, at 07:35, Mike Hammett <nanog@ics-il.net> wrote:
"What is it, exactly, that you expect a provider to do with your report of a few failed SSH login attempts to stop the activity?... disconnect the customer."
Yes.
Comcast does it. My wife's aunt and uncle had a compromised box on their network. They don't check their e-mail, so they didn't see the warnings from Comcast. They shut them off until the problem was resolved.
"Forcing disconnection for a port scan is also, by the way, a *great* way to create an absolute gold-plated A+ denial-of-service: "
Surely they have flow records showing suspicious activity from that customer. They may not confirm the specific IP being attacked, but they will see massive numbers of SSH, SMTP, SIP, etc. connections going out. It's likely if there's outbound activity of that nature and *someone* complained about it, not only were they a victim of it, but the activity is probably undesired by anyone else receiving it as well.
"cost you practically nothing." You're right. An insecure Internet doesn't cost any of us anything.
"there's no One True Format for automated abuse notifications"
So then "let's" make one? No one can follow it if it doesn't exist.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
From: "Matt Palmer" <mpalmer@hezmatt.org> To: nanog@nanog.org Sent: Wednesday, April 29, 2020 6:48:51 AM Subject: Re: Abuse Desks
On Wed, Apr 29, 2020 at 12:24:01PM +0530, Mukund Sivaraman wrote:
On Tue, Apr 28, 2020 at 11:40:16PM -0700, Matt Corallo wrote:
Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.
If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.
Yes we use that, and PermitRootLogin no and an AllowUsers list.
I asked in my first email, if with security practices as above and use of fail2ban to filter attempts, should we just ignore this problem and think that nobody is ultimately reponsible to get rid of this activity?
In theory, no. In practice, unfortunately, yes.
The typical service provider has so much low-level "noise" going on that if everyone reported everything to them, they'd semi-literally drown. Certainly, there's no possible way they could economically handle all that abuse reporting -- hiring all the people to examine, determine the veracity of, and act upon the reports would cost a fortune, because you better believe there's no One True Format for automated abuse notifications, nor will there ever likely be one, so it's all humans, all the time.
Now, you could argue that they should clean up their network so they don't have that volume of abuse reports coming in -- and you'd be right, in theory. But there's a *lot* of low-level stuff that it isn't practical to stop, in and of itself.
Consider your own reports. What is it, exactly, that you expect a provider to do with your report of a few failed SSH login attempts to stop the activity? Say it's a residential ISP, or an IaaS provider. They have only a few very large hammers at their disposal -- they can (maybe) filter the destination port, filter your destination IP, or disconnect the customer. Any of those will very possibly result in a support call, or lost customer. That's a very large cost you're expecting them to pay for something which has, let's face it, cost you practically nothing.
Forcing disconnection for a port scan is also, by the way, a *great* way to create an absolute gold-plated A+ denial-of-service: send in a plausible-looking report of shenanigans to the ISP of someone you don't like, and *boom* their Internet connection's cut off. WINNAH!
So what are you left with, action-wise? An ISP could keep a tally of abuse reports by customer, and take action on whoever's at the top of the pile, but that would then require a large and expensive army of humans to receive, check, classify, and record all incoming abuse reports. Do *you* want to pay $1,000/month for your home Internet connection to cover the cost of all those extra ISP staff? Because, as I said before, there's no One True Format for reporting abuse, and there never will be.
Not that it would work, anyway -- any sort of "threshold" system for abuse ends up being gamed, anyway. You only need to look at how Twitter users with an axe to grind gang up to send in malicious reports about some other Twitter they don't like, which trips Twitter's "lots of reports => autoban" logic, to see how that would end if you started applying it to Internet abuse reporting.
Finally, because nobody is ever convinced by rhetoric, here's an appeal to your self-interest: "crying wolf" is never a good idea. In the event that you *do* have a real problem that needs to be dealt with some time in the future, do you want to have your e-mail address, IP address, and whatever else associated with a thousand previous GWF ("goober with firewall") reports? Any abuse desk who has seen your hundreds of previous unactionable reports will almost certainly round-file that important one, and then you're *really* up the creek sans paddle. Far better to keep your powder dry and be ready for when you actually need assistance from whoever you're contacting.
- Matt
On April 29, 2020 at 07:35 nanog@ics-il.net (Mike Hammett) wrote:
"What is it, exactly, that you expect a provider to do with your report of a few failed SSH login attempts to stop the activity?... disconnect the customer."
Yes.
What I've done in the past is tell the customer we have received complaints and if they continue will bill them $100 (pick a number) per complaint as we are obliged to respond to them. I actually had someone pay me about $1,000 once tho I'll admit the threat was usually enough. In a couple of egregious cases I billed them and shut them off explaining they didn't have sufficient credit with us so will only be turned back on when the complaint bills are paid and a deposit for future complaints received (pick a number.) TBH I usually never heard from those customers again but that was fine by me. -- -Barry Shein Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*
On Tue, 28 Apr 2020, Matt Corallo wrote:
Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.
compromised servers on your infrastructure hosting nigerian criminals look much the same as a script kiddie experimenting with nmap.
If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.
you either care about reports of potentially compromised hosts on your infrastructure or you don't. -Dan
Good thing I care, but that's missing the point here - the volume of abuse requests makes the entire abuse system unworkable. Not for me so much, I can deal with the volume (a few obnoxious individuals aside), but AWS/OVH/Hertzner appear to have decided they cannot, and that means I can't contact them if there's something more serious going on. I highly doubt so many folks "don't care" about potentially compromised hosts, in fact I know for sure several of them have deployed a number of full-time staff to build solutions to monitor for such things. The fact that those solutions often don't involve their abuse system should tell us something. Matt On 4/29/20 3:44 AM, Dan Hollis wrote:
On Tue, 28 Apr 2020, Matt Corallo wrote:
Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.
compromised servers on your infrastructure hosting nigerian criminals look much the same as a script kiddie experimenting with nmap.
If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.
you either care about reports of potentially compromised hosts on your infrastructure or you don't.
-Dan
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"? I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible. I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email? -- Chris Adams <cma@cmadams.net>
Perhaps some organization of Network Operators should come up with an objective standard of what constitutes “abuse” and a standard format for reporting it. If only there was such an organization. Sent from my iPhone
On Apr 29, 2020, at 11:14 AM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible.
I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email?
-- Chris Adams <cma@cmadams.net>
SRonan, If only such a standard were feasible :) -mel beckman
On Apr 29, 2020, at 8:25 AM, "sronan@ronan-online.com" <sronan@ronan-online.com> wrote:
Perhaps some organization of Network Operators should come up with an objective standard of what constitutes “abuse” and a standard format for reporting it.
If only there was such an organization.
Sent from my iPhone
On Apr 29, 2020, at 11:14 AM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible.
I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email?
-- Chris Adams <cma@cmadams.net>
The standards are perfectly feasible. That doesn't mean people will follow them, however it's much better to say "I ignored your notification because it didn't follow the objective standard" then it is to just say "I ignored your notification because I felt like it" On Wed, Apr 29, 2020, 11:37 AM Mel Beckman <mel@beckman.org> wrote:
SRonan,
If only such a standard were feasible :)
-mel beckman
On Apr 29, 2020, at 8:25 AM, "sronan@ronan-online.com" < sronan@ronan-online.com> wrote:
Perhaps some organization of Network Operators should come up with an objective standard of what constitutes “abuse” and a standard format for reporting it.
If only there was such an organization.
Sent from my iPhone
On Apr 29, 2020, at 11:14 AM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible.
I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email?
-- Chris Adams <cma@cmadams.net>
In fact, SRonan, the real risk of such a standard is that people would use it to send an increasingly massive flood of pointless abuse reports, which would require deployment of an equally massive AI-based data analytics to cull the flood, which would then be Skynet :) -mel beckman
On Apr 29, 2020, at 8:40 AM, mel@beckman.org wrote:
SRonan,
If only such a standard were feasible :)
-mel beckman
On Apr 29, 2020, at 8:25 AM, "sronan@ronan-online.com" <sronan@ronan-online.com> wrote:
Perhaps some organization of Network Operators should come up with an objective standard of what constitutes “abuse” and a standard format for reporting it.
If only there was such an organization.
Sent from my iPhone
On Apr 29, 2020, at 11:14 AM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible.
I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email?
-- Chris Adams <cma@cmadams.net>
A standard would be nice. In some of the auto-responders, I get requirements that conflict or are unreasonable. * We don't accept abuse complaints via e-mail, please submit via this site: Yeah, okay. That's not scaleable. * Network A wants time in GMT, while network B wants time in their local timezone. How do I know that ahead of time? Adjusting for that isn't scaleable Some are asking for my IP address. Okay, I get that if you have CGNAT running, you need to know that to check your logs. Now I gotta figure out how to get my IP address into the log. Many of the devices watched have more than one IP address. Having a standard would make generation of reports and processing of said reports a lot easier to automate. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: sronan@ronan-online.com To: "NANOG" <nanog@nanog.org> Sent: Wednesday, April 29, 2020 10:25:19 AM Subject: Re: Abuse Desks Perhaps some organization of Network Operators should come up with an objective standard of what constitutes “abuse” and a standard format for reporting it. If only there was such an organization. Sent from my iPhone
On Apr 29, 2020, at 11:14 AM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible.
I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email?
-- Chris Adams <cma@cmadams.net>
On Wed, 29 Apr 2020 11:25:19 -0400, sronan@ronan-online.com said:
Perhaps some organization of Network Operators should come up with an objective standard of what constitutes “abuse” and a standard format for reporting it.
If only there was such an organization.
A different organization beat you to it. 7203 An Incident Object Description Exchange Format (IODEF) Extension for Structured Cybersecurity Information. T. Takahashi, K. Landfield, Y. Kadobayashi. April 2014. (Format: TXT, HTML) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC7203)
On Wed, Apr 29, 2020 at 10:12:29AM -0500, Chris Adams wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible.
I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email?
So your theory is that it is necessary for there to be a threshold of abuse? Is there any reason to expect that a random server is going to be able to figure out that a large pool of a million compromised IoT devices on a million different IP addresses is slowly probing their server for the root password and that a SPECIFIC probe is a member of this set? The way this stuff is trending today, you don't have a single host that is banging on another single host for hours or days at a password per second, which I hope we would agree would be well beyond any reasonable threshold to consider abuse. On the flip side, is it so much to ask that an abuse desk maybe take a look at both the ingress and egress packet stream of their customer, to see if there seems to be something untoward happening? And which one of these is a less damaging strategy? I know we're in the minority here, but policy over here at SOL hasn't changed much in the last quarter century. If you are getting unwanted and unsolicited traffic from us, and you contact abuse@, we're willing to make it stop. If it didn't originate here (forged, etc) then there isn't much to be done -- the community has been trying to encourage BCP38 for years. It's probably jumping the gun a bit for a single connection attempt to result in an abuse@ message, but then again when I look at the stream of trash addressed at SOL's IP space, maybe not. Some of it is clearly trying to scan from large botnets. There's also a lot of room for computers to be doing the hard work of detecting and reporting, and helping to analyze, while letting a human look at what's actually transpired and see if it feels problematic. However, the general solution that seems to have been adopted by the majority of the industry is to hire Dave Null for abuse@ ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov
Joe, Is there any reason to have a root-enabled (or any) ssh server exposed to the bare Internet? Any at all? Can you name one? I can’t. That’s basically pilot error. -mel
On Apr 29, 2020, at 8:37 AM, Joe Greco <jgreco@ns.sol.net> wrote:
On Wed, Apr 29, 2020 at 10:12:29AM -0500, Chris Adams wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
I've typoed IP/FQDN before and gotten an SSH response, and taken several tries before I realized my error. Did I actually "abuse" someone's server? I didn't get in, and it's hard to say that the server resources I used with a few failed tries were anything more than negligible.
I've had users tripped up by fail2ban because they were trying to access a server they don't use often and took several tries to get the password right or had the wrong SSH key. Should that have triggered an abuse email?
So your theory is that it is necessary for there to be a threshold of abuse?
Is there any reason to expect that a random server is going to be able to figure out that a large pool of a million compromised IoT devices on a million different IP addresses is slowly probing their server for the root password and that a SPECIFIC probe is a member of this set?
The way this stuff is trending today, you don't have a single host that is banging on another single host for hours or days at a password per second, which I hope we would agree would be well beyond any reasonable threshold to consider abuse.
On the flip side, is it so much to ask that an abuse desk maybe take a look at both the ingress and egress packet stream of their customer, to see if there seems to be something untoward happening?
And which one of these is a less damaging strategy?
I know we're in the minority here, but policy over here at SOL hasn't changed much in the last quarter century. If you are getting unwanted and unsolicited traffic from us, and you contact abuse@, we're willing to make it stop. If it didn't originate here (forged, etc) then there isn't much to be done -- the community has been trying to encourage BCP38 for years.
It's probably jumping the gun a bit for a single connection attempt to result in an abuse@ message, but then again when I look at the stream of trash addressed at SOL's IP space, maybe not. Some of it is clearly trying to scan from large botnets.
There's also a lot of room for computers to be doing the hard work of detecting and reporting, and helping to analyze, while letting a human look at what's actually transpired and see if it feels problematic.
However, the general solution that seems to have been adopted by the majority of the industry is to hire Dave Null for abuse@
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov
On Wed, Apr 29, 2020 at 03:41:06PM +0000, Mel Beckman wrote:
Joe,
Is there any reason to have a root-enabled (or any) ssh server exposed to the bare Internet? Any at all? Can you name one? I can???t. That???s basically pilot error.
Mel, I think you're looking at it the wrong way. Blaming a potential victim doesn't solve the problem. I like to use a metric of "if everybody did this, would it be a good thing" often. If everybody Good thing? Didn't run SSHD on public Inet Yes Ran SSH scanners against the rest of the Inet No Ran SSH scanners against their own gear and used it to shut down unnecessary SSH Yes The problem is that you're talking about the first case, but the actual problem is the second case. If this trash is allowed to continue, there is a point where your server will just get swamped by a growing number of SSH probes. Also, exposing SSH to the Internet is, for better or for worse, the way many cloud services enable access to their cloud VM's/instances/droplets/ whatever. And, finally, yes, there are reasons to expose SSH servers to the Internet. A well-defended SSH server can do things such as allow other parties access to your server. I run a number of bastion SSH servers for various purposes. You do not need to do so in an obvious manner. That doesn't mean I'm inviting unauthorized parties to try to connect to them. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov
On 4/29/20 8:41 AM, Mel Beckman wrote:
Is there any reason to have a root-enabled (or any) ssh server exposed to the bare Internet? Any at all? Can you name one? I can’t. That’s basically pilot error.
Remember HeartBleed? That didn't require a rout-enabled SSH server. It didn't require SSH server. That said, I use TCPWRAPPER to limit access to SSH to specific IP addresses. I process my LogWatch messages manually. I pull the fire alarm for showshoe probes, and excessive number of probes (over 30 in a 24-hour period). No registered abuse@ address in the WHOIS? The offending netblock goes into my edge router ACL, because I have learned that ne'er-do-wells without working abuse@ usually have other bad habits. And I disclose this practice to all who use my network. (Blackmail emails are another set-and-forget trigger, but that's a subject for NANAE newsgroup.)
----- On Apr 29, 2020, at 9:08 AM, Stephen Satchell list@satchell.net wrote: Hi,
That said, I use TCPWRAPPER to limit access to SSH to specific IP addresses. I process my LogWatch messages manually. I pull the fire alarm for showshoe probes, and excessive number of probes (over 30 in a 24-hour period). No registered abuse@ address in the WHOIS? The offending netblock goes into my edge router ACL, because I have learned that ne'er-do-wells without working abuse@ usually have other bad habits.
I have a very simple method to deal with that: a server with no other purpose than to blackhole portscanning culprits. Send so much as a tcp syn to port 22 and your entire /24 goes to null0 for a month. I have a few exceptions for entities that I know are responsive to abuse@, but that's it. Highly effective. Thanks, Sabri
Sabri, A clever idea to be sure, but it seems open to abuse. What stops someone from forging a tcp syn from every /24 on the Internet, causing you to blackhole your access to everywhere? -mel
On Apr 29, 2020, at 2:24 PM, Sabri Berisha <sabri@cluecentral.net> wrote:
----- On Apr 29, 2020, at 9:08 AM, Stephen Satchell list@satchell.net wrote:
Hi,
That said, I use TCPWRAPPER to limit access to SSH to specific IP addresses. I process my LogWatch messages manually. I pull the fire alarm for showshoe probes, and excessive number of probes (over 30 in a 24-hour period). No registered abuse@ address in the WHOIS? The offending netblock goes into my edge router ACL, because I have learned that ne'er-do-wells without working abuse@ usually have other bad habits.
I have a very simple method to deal with that: a server with no other purpose than to blackhole portscanning culprits. Send so much as a tcp syn to port 22 and your entire /24 goes to null0 for a month. I have a few exceptions for entities that I know are responsive to abuse@, but that's it.
Highly effective.
Thanks,
Sabri
----- On Apr 29, 2020, at 3:15 PM, mel mel@beckman.org wrote: Hi Mel,
A clever idea to be sure, but it seems open to abuse. What stops someone from forging a tcp syn from every /24 on the Internet, causing you to blackhole your access to everywhere?
Fair point, and I lied a bit. My code relies on inet_ntoa(client_addr.sin_addr)) after accept(), so technically it requires a bit more than just a SYN. But the basic idea is that anyone connecting to IPs that they should not be connecting to, will be nullrouted from the network for 30 days. The bad guys automate scanning, I automate blocking. In the old days (pre-9/11), scriptkiddie-me would simply send a teardrop. Luckily I have matured slightly since that time. Thanks, Sabri
On Apr 29, 2020, at 2:24 PM, Sabri Berisha <sabri@cluecentral.net> wrote:
----- On Apr 29, 2020, at 9:08 AM, Stephen Satchell list@satchell.net wrote:
Hi,
That said, I use TCPWRAPPER to limit access to SSH to specific IP addresses. I process my LogWatch messages manually. I pull the fire alarm for showshoe probes, and excessive number of probes (over 30 in a 24-hour period). No registered abuse@ address in the WHOIS? The offending netblock goes into my edge router ACL, because I have learned that ne'er-do-wells without working abuse@ usually have other bad habits.
I have a very simple method to deal with that: a server with no other purpose than to blackhole portscanning culprits. Send so much as a tcp syn to port 22 and your entire /24 goes to null0 for a month. I have a few exceptions for entities that I know are responsive to abuse@, but that's it.
Highly effective.
Thanks,
Sabri
On Wed, Apr 29, 2020 at 03:41:06PM +0000, Mel Beckman wrote:
Joe,
Is there any reason to have a root-enabled (or any) ssh server exposed to the bare Internet? Any at all? Can you name one? I can’t. That’s basically pilot error.
The last time (a couple of weeks ago) when I installed a Linux on a machine hosted at hetzner.com over the internet, when I logged in for the first time over SSH, it reported about 60 login failures by then. This is time between install and first login over SSH within a matter of minutes. If there's a lock on my door, and someone tries to pick it, you can call me at fault for having a lock on my door facing outside all you want. But the thief picking it has no business doing so, and will be guilty of a crime if caught. Mukund
On 4/29/20 9:24 AM, Mukund Sivaraman wrote:
If there's a lock on my door, and someone tries to pick it, you can call me at fault for having a lock on my door facing outside all you want. But the thief picking it has no business doing so, and will be guilty of a crime if caught.
This is a good start to an analogy. Let's build on it, courtesy to YouTube's "Lock Picking Lawyer". In a video, the host shows how to improve the security of a common easily-picked home lock: drill holes in the lock body, such that if someone picks the lock and tries to turn the keyway, the pins will fall into those carefully-placed holes and foil The Bad Guy(tm). In the networking world, we use an Access Control List to limit access to the service. Unlike the simple modification shown in LPL's video, the "lock" is still usable by users from authorized IP addresses. Or, we require the use of certificates to validate access within the SSHD server itself. Here's the deal: just blocking access or requiring certificate-based access is intrusion prevention. Having a log event when there are unsuccessful probes is intrusion [attempt] detection. Sure, the ne'er-do-well is kept out in the prevention cycle, but a persistent cracker lives by the axiom "if at first you don't succeed, try something else." You really want to stop an attacker from making a large number of attempts, such as with a Joe script. I turn off root SSH access, pinhole 22/tcp to a limited number of IP addresses, and monitor failed SUDO attempts. As I build up my new firewall, I'll turn off public SSH access completely, and instead use a robust VPN implementation. (Which has its own issues.)
That's not always feasible. My routers have ACLs, but my servers for the most part do not. It's kind of counter productive to put ACLs on SMTP, POP3, IMAP, and HTTP\S ports, now isn't it? SIP, FTP, and SSH may or may not make sense, depending on the type and volume of users. Since there are at least some services that are subject to attack that must remain available to the whole Internet, the "just ACL everything" argument goes into the trash. We're back to how to handle abuse report generation and processing in a way that it is taken seriously by those within such a desire to do so. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Stephen Satchell" <list@satchell.net> To: nanog@nanog.org Sent: Wednesday, April 29, 2020 11:50:42 AM Subject: Re: Abuse Desks On 4/29/20 9:24 AM, Mukund Sivaraman wrote:
If there's a lock on my door, and someone tries to pick it, you can call me at fault for having a lock on my door facing outside all you want. But the thief picking it has no business doing so, and will be guilty of a crime if caught.
This is a good start to an analogy. Let's build on it, courtesy to YouTube's "Lock Picking Lawyer". In a video, the host shows how to improve the security of a common easily-picked home lock: drill holes in the lock body, such that if someone picks the lock and tries to turn the keyway, the pins will fall into those carefully-placed holes and foil The Bad Guy(tm). In the networking world, we use an Access Control List to limit access to the service. Unlike the simple modification shown in LPL's video, the "lock" is still usable by users from authorized IP addresses. Or, we require the use of certificates to validate access within the SSHD server itself. Here's the deal: just blocking access or requiring certificate-based access is intrusion prevention. Having a log event when there are unsuccessful probes is intrusion [attempt] detection. Sure, the ne'er-do-well is kept out in the prevention cycle, but a persistent cracker lives by the axiom "if at first you don't succeed, try something else." You really want to stop an attacker from making a large number of attempts, such as with a Joe script. I turn off root SSH access, pinhole 22/tcp to a limited number of IP addresses, and monitor failed SUDO attempts. As I build up my new firewall, I'll turn off public SSH access completely, and instead use a robust VPN implementation. (Which has its own issues.)
On 4/29/20 9:57 AM, Mike Hammett wrote:
My routers have ACLs, but my servers for the most part do not.
I'm not trying to argue, but...what servers do you have that don't have sysadmin-definable firewalls and tun-able knobs? My edge routers are Linux boxes (CentOS 8 for the one I'm now building). Moreover, I can have NetworkManager fire off a script that modifies the firewall settings as interfaces go up and down.
It's kind of counter productive to put ACLs on SMTP, POP3, IMAP, and HTTP\S ports, now isn't it? SIP, FTP, and SSH may or may not make sense, depending on the type and volume of users. I was taught by my networking betters that you need to block certain types of public inbound packets, always, that match any of:
1. WAN packets with local/LAN source address 2. WAN packets with local/LAN broadcast/net src-dst address 3. WAN packets with known broadcast/net src-dst address 4. WAN packets with local/LAN small services 5. WAN packets with local/LAN unimplemented services 6. WAN packets with blackholed source address On EVERY device with a public IP address. WITHOUT FAIL. I have these blocks on every single public-facing mail server I build. I have these blocks on every single public-facing Web server I build. Indeed, I can't fathom why I would *not* have these in place for every single public-facing device. I don't necessarily log every occurance, but I do drop matching packets on the floor, unceremoniously. This is the foundation upon which I build custom additions, such as allowing 22/tcp only from specific IP addresses. I don't depend on the edge router to catch all the cases, because each server has specific services it provides. So, for example, my DNS servers not only implement all six basics, but also incorporates request rate limiting, to avoid participating in DDOS events. Ditto NTP servers. 80/tcp and 443/tcp? Dropped on the floor. Sorry to preach, but I'm in the process of building a NFTABLE-based firewall and this happens to be part of the specs for it.
That you have some ACLs that whack low-hanging fruit doesn't negate the fact that you can't block the untrusted Internet accessing an intentionally publicly accessible port. It's all just a distraction from the fact that *SOME* services *MUST* remain available to the general public and those services are subject to abuse. As long as there are things that must be available to the general public (likely forever), there needs to be an abuse reporting process that works. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Stephen Satchell" <list@satchell.net> To: nanog@nanog.org Sent: Wednesday, April 29, 2020 12:35:20 PM Subject: Re: Abuse Desks On 4/29/20 9:57 AM, Mike Hammett wrote:
My routers have ACLs, but my servers for the most part do not.
I'm not trying to argue, but...what servers do you have that don't have sysadmin-definable firewalls and tun-able knobs? My edge routers are Linux boxes (CentOS 8 for the one I'm now building). Moreover, I can have NetworkManager fire off a script that modifies the firewall settings as interfaces go up and down.
It's kind of counter productive to put ACLs on SMTP, POP3, IMAP, and HTTP\S ports, now isn't it? SIP, FTP, and SSH may or may not make sense, depending on the type and volume of users. I was taught by my networking betters that you need to block certain types of public inbound packets, always, that match any of:
1. WAN packets with local/LAN source address 2. WAN packets with local/LAN broadcast/net src-dst address 3. WAN packets with known broadcast/net src-dst address 4. WAN packets with local/LAN small services 5. WAN packets with local/LAN unimplemented services 6. WAN packets with blackholed source address On EVERY device with a public IP address. WITHOUT FAIL. I have these blocks on every single public-facing mail server I build. I have these blocks on every single public-facing Web server I build. Indeed, I can't fathom why I would *not* have these in place for every single public-facing device. I don't necessarily log every occurance, but I do drop matching packets on the floor, unceremoniously. This is the foundation upon which I build custom additions, such as allowing 22/tcp only from specific IP addresses. I don't depend on the edge router to catch all the cases, because each server has specific services it provides. So, for example, my DNS servers not only implement all six basics, but also incorporates request rate limiting, to avoid participating in DDOS events. Ditto NTP servers. 80/tcp and 443/tcp? Dropped on the floor. Sorry to preach, but I'm in the process of building a NFTABLE-based firewall and this happens to be part of the specs for it.
I think we all agree with this. The requl question is...how do we build such a thing? The abuse process we have clearly doesn't work. Maybe its the fault of the Big Providers (AWS/GCP/OVH/etc) who don't invest enough to have a robust abuse-processing system to actually deal with reports, maybe its because people are too liberal with automated reports, maybe its because we don't have a common abuse report format. Either way, one thing I don't think we can deny is that the abuse system today is very, very broken. Abuse reports to the biggest offenders (ie Big Providers, mostly) get ignored and abuse is not dealt with. Matt On 4/29/20 1:42 PM, Mike Hammett wrote: -snip-
As long as there are things that must be available to the general public (likely forever), there needs to be an abuse reporting process that works.
On Wed, Apr 29, 2020 at 09:50:42AM -0700, Stephen Satchell wrote:
On 4/29/20 9:24 AM, Mukund Sivaraman wrote:
If there's a lock on my door, and someone tries to pick it, you can call me at fault for having a lock on my door facing outside all you want. But the thief picking it has no business doing so, and will be guilty of a crime if caught.
This is a good start to an analogy. Let's build on it, courtesy to YouTube's "Lock Picking Lawyer". In a video, the host shows how to improve the security of a common easily-picked home lock: drill holes in the lock body, such that if someone picks the lock and tries to turn the keyway, the pins will fall into those carefully-placed holes and foil The Bad Guy(tm).
In the networking world, we use an Access Control List to limit access to the service. Unlike the simple modification shown in LPL's video, the "lock" is still usable by users from authorized IP addresses. Or, we require the use of certificates to validate access within the SSHD server itself.
Here's the deal: just blocking access or requiring certificate-based access is intrusion prevention. Having a log event when there are unsuccessful probes is intrusion [attempt] detection. Sure, the ne'er-do-well is kept out in the prevention cycle, but a persistent cracker lives by the axiom "if at first you don't succeed, try something else." You really want to stop an attacker from making a large number of attempts, such as with a Joe script.
I turn off root SSH access, pinhole 22/tcp to a limited number of IP addresses, and monitor failed SUDO attempts. As I build up my new firewall, I'll turn off public SSH access completely, and instead use a robust VPN implementation. (Which has its own issues.)
The previous criticism of running a public sshd reminds me of a person who claimed women should not wear sexy dresses and complain that they were molested. Though that may appear to be logical, it's blaming the victim rather than addressing the problem. It is no excuse. Our services are secured best as we can for our use-case. That's not what is being discussed. There are 2 things that are happening: (1) Service providers are allowing nefarious traffic to exit their network. They probably don't have tools to detect/prevent this because they probably have not budgeted it, or don't care because there are no consequences. (2) They don't want to handle a proportional level of abuse@ email that is directed back at them for the amount of crap that they inflict on the rest of the internet. They probably have not budgeted for handling it. All the explanation that has been offered, including whether one wants to pay $1000 for an internet connection are just excuses to deflect from the above two things, because they are not being held responsible to take action. Tt is a technical problem to detect scanning of TCP port 22, 587 of large swathes of IP address space from a host. If service providers were inclined to fix it by spending money on it, they could automatically detect them and stop them, even without abuse@ emails asking for manual investigation. When there is no problem, there is no email to abuse@ about it. This thread talks about the incentive to the service provider.. what is the incentive to us to let abuse@ know of a problem on their network? Trust that we don't make any money off it either. If I email an abuse@ contact, what I expect is "Thank you." "Thank you for helping us detect nefarious activity from our network, that hurts the internet." instead of all the defence that this thread has posted. That's how abuse reports should be handled. It doesn't matter if the report was emailed manually or by pigeons riding unicorns, as long as it provides some information that the abuse@ contact can act upon. It's not as if the host which had the address is suddenly going to become a good citizen that it is no longer detectable. Mukund
What if I am at home, and while working on a project, fire off a wide ranging nmap against say a /19 work network to validate something externally? Should my ISP detect that and make a decision that I shouldn't be doing that, even though it is completely legitimate and authorized activity? What if I fat fingered a digit and accidentally ran that same scan against someone else's /19? Should that accidental destination of non-malicious scans be able to file an abuse report against me and get my service disconnected because they didn't like it? Abuse departments should be properly handling LEGITIMATE abuse complaints. Not crufty background noise traffic that is never going away. On Wed, Apr 29, 2020 at 1:35 PM Mukund Sivaraman <muks@mukund.org> wrote:
On 4/29/20 9:24 AM, Mukund Sivaraman wrote:
If there's a lock on my door, and someone tries to pick it, you can call me at fault for having a lock on my door facing outside all you want. But the thief picking it has no business doing so, and will be guilty of a crime if caught.
This is a good start to an analogy. Let's build on it, courtesy to YouTube's "Lock Picking Lawyer". In a video, the host shows how to improve the security of a common easily-picked home lock: drill holes in the lock body, such that if someone picks the lock and tries to turn the keyway,
pins will fall into those carefully-placed holes and foil The Bad Guy(tm).
In the networking world, we use an Access Control List to limit access to the service. Unlike the simple modification shown in LPL's video, the "lock" is still usable by users from authorized IP addresses. Or, we require the use of certificates to validate access within the SSHD server itself.
Here's the deal: just blocking access or requiring certificate-based access is intrusion prevention. Having a log event when there are unsuccessful probes is intrusion [attempt] detection. Sure, the ne'er-do-well is kept out in the prevention cycle, but a persistent cracker lives by the axiom "if at first you don't succeed, try something else." You really want to stop an attacker from making a large number of attempts, such as with a Joe
On Wed, Apr 29, 2020 at 09:50:42AM -0700, Stephen Satchell wrote: the script.
I turn off root SSH access, pinhole 22/tcp to a limited number of IP addresses, and monitor failed SUDO attempts. As I build up my new
firewall,
I'll turn off public SSH access completely, and instead use a robust VPN implementation. (Which has its own issues.)
The previous criticism of running a public sshd reminds me of a person who claimed women should not wear sexy dresses and complain that they were molested. Though that may appear to be logical, it's blaming the victim rather than addressing the problem. It is no excuse.
Our services are secured best as we can for our use-case. That's not what is being discussed.
There are 2 things that are happening:
(1) Service providers are allowing nefarious traffic to exit their network. They probably don't have tools to detect/prevent this because they probably have not budgeted it, or don't care because there are no consequences.
(2) They don't want to handle a proportional level of abuse@ email that is directed back at them for the amount of crap that they inflict on the rest of the internet. They probably have not budgeted for handling it.
All the explanation that has been offered, including whether one wants to pay $1000 for an internet connection are just excuses to deflect from the above two things, because they are not being held responsible to take action.
Tt is a technical problem to detect scanning of TCP port 22, 587 of large swathes of IP address space from a host. If service providers were inclined to fix it by spending money on it, they could automatically detect them and stop them, even without abuse@ emails asking for manual investigation. When there is no problem, there is no email to abuse@ about it.
This thread talks about the incentive to the service provider.. what is the incentive to us to let abuse@ know of a problem on their network? Trust that we don't make any money off it either. If I email an abuse@ contact, what I expect is "Thank you." "Thank you for helping us detect nefarious activity from our network, that hurts the internet." instead of all the defence that this thread has posted. That's how abuse reports should be handled. It doesn't matter if the report was emailed manually or by pigeons riding unicorns, as long as it provides some information that the abuse@ contact can act upon. It's not as if the host which had the address is suddenly going to become a good citizen that it is no longer detectable.
Mukund
On Wed, Apr 29, 2020 at 01:49:14PM -0400, Tom Beecher wrote:
What if I am at home, and while working on a project, fire off a wide ranging nmap against say a /19 work network to validate something externally? Should my ISP detect that and make a decision that I shouldn't be doing that, even though it is completely legitimate and authorized activity? What if I fat fingered a digit and accidentally ran that same scan against someone else's /19? Should that accidental destination of non-malicious scans be able to file an abuse report against me and get my service disconnected because they didn't like it?
Abuse departments should be properly handling LEGITIMATE abuse complaints. Not crufty background noise traffic that is never going away.
Sure. Handling legitimate abuse complaints would be quite sufficient. :) Mukund
Well, I think our disagreement is on what we constitute 'legitimate abuse' to be. On Wed, Apr 29, 2020 at 1:51 PM Mukund Sivaraman <muks@mukund.org> wrote:
On Wed, Apr 29, 2020 at 01:49:14PM -0400, Tom Beecher wrote:
What if I am at home, and while working on a project, fire off a wide ranging nmap against say a /19 work network to validate something externally? Should my ISP detect that and make a decision that I shouldn't be doing that, even though it is completely legitimate and authorized activity? What if I fat fingered a digit and accidentally ran that same scan against someone else's /19? Should that accidental destination of non-malicious scans be able to file an abuse report against me and get my service disconnected because they didn't like it?
Abuse departments should be properly handling LEGITIMATE abuse complaints. Not crufty background noise traffic that is never going away.
Sure. Handling legitimate abuse complaints would be quite sufficient. :)
Mukund
On 2020-04-29 17:51, Mukund Sivaraman wrote:
On Wed, Apr 29, 2020 at 01:49:14PM -0400, Tom Beecher wrote:
What if I am at home, and while working on a project, fire off a wide ranging nmap against say a /19 work network to validate something externally? Should my ISP detect that and make a decision that I shouldn't be doing that, even though it is completely legitimate and authorized activity? What if I fat fingered a digit and accidentally ran that same scan against someone else's /19? Should that accidental destination of non-malicious scans be able to file an abuse report against me and get my service disconnected because they didn't like it?
Abuse departments should be properly handling LEGITIMATE abuse complaints. Not crufty background noise traffic that is never going away. Sure. Handling legitimate abuse complaints would be quite sufficient. :)
Mukund
Since this is a distributed network and there's not a central authority to rule on each incident being legitimate, the only way to stay out of the politics is to ignore people's abuse complaints. Someone's SSH server is being spammed with probes? That's pretty low bandwidth, not much threat to the network from a cracking script. Maybe you don't like it, maybe it's criminal or whatever else, but ostensibly it's some paying customer's traffic and it should be delivered unmolested. When someone's infrastructure is getting packeted or having their routers crashed repeatedly, they respond to that, usually without having to be emailed, because it's actual abuse of their network. A lot of this other stuff is just people abusing the abuse contacts to get someone else taken offline. Phishing websites fall into this category - it's not network abuse, it's just content someone doesn't like, and one way to get it taken down is to threaten the network that carries the traffic for it. -Laszlo
On Wed, 2020-04-29 at 09:50 -0700, Stephen Satchell wrote:
As I build up my new firewall, I'll turn off public SSH access completely, and instead use a robust VPN implementation. (Which has its own issues.)
How does that solve the problem at hand in any way? The abuse/probing just moves from ssh to your VPN service and you are back to all of the same problems/arguments. Cheers, b.
On Wed, Apr 29, 2020 at 10:12:29AM -0500, Chris Adams wrote:
Once upon a time, Mukund Sivaraman <muks@mukund.org> said:
If an abuse report is incorrect, then it is fair to complain.
The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?
It is configurable. Anyway, I don't know how else one would interpret a pattern like this other than the obvious: Apr 28 22:28:05 jupiter sshd[24509]: Invalid user java from 209.141.55.11 port 36334 Apr 28 22:28:05 jupiter sshd[24504]: Invalid user openvpn from 209.141.55.11 port 36768 Apr 28 22:28:05 jupiter sshd[24506]: Invalid user devops from 209.141.55.11 port 36756 Apr 28 22:28:05 jupiter sshd[24510]: Invalid user vagrant from 209.141.55.11 port 36784 Apr 28 22:28:05 jupiter sshd[24507]: Invalid user user from 209.141.55.11 port 36796 Apr 28 22:28:05 jupiter sshd[24508]: Invalid user oracle from 209.141.55.11 port 36776 Apr 28 22:28:05 jupiter sshd[24505]: Invalid user ubuntu from 209.141.55.11 port 36798 Apr 28 22:28:05 jupiter sshd[24514]: Invalid user test from 209.141.55.11 port 36780 Apr 28 22:28:05 jupiter sshd[24513]: Invalid user ec2-user from 209.141.55.11 port 36752 It *can* be legitimate traffic, but then I hope the owner of this machine has applied for special permission stating their reason for doing this kind of probing before they are allowed to keep doing this over time and sending such traffic to multiple IP addresses (similar to how, at some service providers, one has to apply for TCP port 25 to be allowed after claiming they're not spammers). Mukund
On Tue, Apr 28, 2020 at 12:40:12PM -0400, Matt Corallo via NANOG wrote:
Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore.
[ "you" = rhetorical "you", throughout ] No, the reason that folks don't have responsive abuse contacts is that they're some combination of: - lazy - cheap [1] - incompetent - unprofessional - actively supporting the abusers A "we received 3 login attempts on our SSH box" complaint should be read, investigated, and acted on. It means that something is going on that shouldn't, and so for your own sake, as well as for the well-being of your Internet neighbors, you should find out what that is. That "for your own sake" clause is often overlooked. An incoming abuse complaint is sometimes the canary in the coal mine. Ignoring it because it appears to be trivial at first glance is extremely foolish. The lesson of the 75-cent accounting error is now 34 years old. This would be a really good time to learn from it. You should also consider that -- thanks to the negligence and incompetence of many abuse desks -- a lot of people simply don't bother reporting incidents any more. Thus what presents to you, on the surface, as "we received 3 login attempts on our SSH box" may in fact be one isolated report of a much larger incident. It thus requires your immediate attention, if you want to even pretend to be a responsible, competent professional. Incidentally, an excellent way to reduce the number of "we received 3 login attempts on our SSH box" complaints is to deal with all of them, thus decreasing incident occurence, which will of course result in a corresponding decrease in complaints. An even better way is to run your operation in such a way that you detect and deal with as many such things as possible before anybody needs to file a complaint. After all, if they can see the traffic arriving on their side, you can see it leaving on yours. ---rsk [1] I note, for example, that Charter -- which is mentioned in the original message in this thread -- currently has a market capitalization of 116.63 billion dollars. Surely they could spare a tiny fraction of that to appropriately staff a 24x7 multi-lingual abuse desk with senior engineers and empower/equip them to do what needs to done. That's what a professional operation would do.
Rich, It’s interesting that you mention “the lesson of the 75-cent accounting error” from Cliff Stoll’s The Cuckoos Egg. Because the lesson from that account is precisely that exerting a massive human-labor-intensive effort to trace every tiny abuse signal is not worth the heavy cost — in this case, the derailing of an astronomer’s career and the infliction upon humanity of irrelevant chocolate chip cookie recipes. An even better lesson is the comparison equation of ubiquitous low-level Internet scanning activity with astronomical Cosmic Background Radiation: a fact of life and an untraceable phenomenon of the Internet universe. Imagine if astronomers emailed the IAU every time they got a tick on their QUBIC sensors. We live in an inflationary Internet. Exhaustively policing its CBR is a waste of time. Time better spent hardening interfaces — or eliminating them using established technologies such as VPN and TLS everywhere. Any network operator getting fail2ban reports from public IPs needs to overhaul his network, not spam the rest of us with automated abuse reports. I mean, what lazy, cheap, incompetent, unprofessional sysadmin leaves SSH ports open to the pubic Internet? :) -mel beckman On Apr 29, 2020, at 4:56 AM, Rich Kulawiec <rsk@gsp.org> wrote: On Tue, Apr 28, 2020 at 12:40:12PM -0400, Matt Corallo via NANOG wrote: Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore. [ "you" = rhetorical "you", throughout ] No, the reason that folks don't have responsive abuse contacts is that they're some combination of: - lazy - cheap [1] - incompetent - unprofessional - actively supporting the abusers A "we received 3 login attempts on our SSH box" complaint should be read, investigated, and acted on. It means that something is going on that shouldn't, and so for your own sake, as well as for the well-being of your Internet neighbors, you should find out what that is. That "for your own sake" clause is often overlooked. An incoming abuse complaint is sometimes the canary in the coal mine. Ignoring it because it appears to be trivial at first glance is extremely foolish. The lesson of the 75-cent accounting error is now 34 years old. This would be a really good time to learn from it. You should also consider that -- thanks to the negligence and incompetence of many abuse desks -- a lot of people simply don't bother reporting incidents any more. Thus what presents to you, on the surface, as "we received 3 login attempts on our SSH box" may in fact be one isolated report of a much larger incident. It thus requires your immediate attention, if you want to even pretend to be a responsible, competent professional. Incidentally, an excellent way to reduce the number of "we received 3 login attempts on our SSH box" complaints is to deal with all of them, thus decreasing incident occurence, which will of course result in a corresponding decrease in complaints. An even better way is to run your operation in such a way that you detect and deal with as many such things as possible before anybody needs to file a complaint. After all, if they can see the traffic arriving on their side, you can see it leaving on yours. ---rsk [1] I note, for example, that Charter -- which is mentioned in the original message in this thread -- currently has a market capitalization of 116.63 billion dollars. Surely they could spare a tiny fraction of that to appropriately staff a 24x7 multi-lingual abuse desk with senior engineers and empower/equip them to do what needs to done. That's what a professional operation would do.
I obviously agree it *can* be an indication of a bigger issue, but it isn't always. Lets take an example from one of my (isolated netblocks): ~$ whois 208.68.4.129 Comment: --------------- Comment: 208.68.4.128/28 and 208.68.7.128/28 provide privacy services Comment: (incl running tor exit node(s)!) Comment: Abuse reports will be handled but there is likely not much that can be done. Comment: Send abuse to abuse at privacysvcs net. Comment: --------------- ... RAbuseEmail: see-comments-no-bots@example.com Now you can decide to pass judgement on the idea that someone may want to run a Tor exit node (my data says a good chunk of users are regular internet users in Iran, so I'm happy with it), but that's beside the point. Only a few outbound ports are allowed, and SSH is appropriately rate-limited. And yet there's a reason the registered abuse email is a dummy one - if its not, not only do I get a flood of automated crap, but I get angry idiots complaining about lack of response. If I dont put a dummy email there, I don't get legitimate reports hidden under the giant pile of, literally one failed SSH login, or wouldn't be in a position to respond to them quickly. If I do put a dummy email there, I miss legitimate reports for other hosts. I'm open to ideas on what to do here, but the abuse system as it exists today is clearly broken for me, and its clearly broken for AWS/GCP/Azure/OVH/etc - have you ever tried emailing their registered abuse contacts? I have, the problem doesn't go away and there are no responses. The answer is balance, of course, but my concept of balance is your concept of abuse. Either way the situation we've ended up in is that the whole thing is nigh useless, especially given most of the real crap out there comes from hosting providers like the above who don't have the bandwidth to respond. Matt On 4/29/20 7:55 AM, Rich Kulawiec wrote:
On Tue, Apr 28, 2020 at 12:40:12PM -0400, Matt Corallo via NANOG wrote:
Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore.
[ "you" = rhetorical "you", throughout ]
No, the reason that folks don't have responsive abuse contacts is that they're some combination of:
- lazy - cheap [1] - incompetent - unprofessional - actively supporting the abusers
A "we received 3 login attempts on our SSH box" complaint should be read, investigated, and acted on. It means that something is going on that shouldn't, and so for your own sake, as well as for the well-being of your Internet neighbors, you should find out what that is.
That "for your own sake" clause is often overlooked. An incoming abuse complaint is sometimes the canary in the coal mine. Ignoring it because it appears to be trivial at first glance is extremely foolish.
The lesson of the 75-cent accounting error is now 34 years old. This would be a really good time to learn from it.
You should also consider that -- thanks to the negligence and incompetence of many abuse desks -- a lot of people simply don't bother reporting incidents any more. Thus what presents to you, on the surface, as "we received 3 login attempts on our SSH box" may in fact be one isolated report of a much larger incident. It thus requires your immediate attention, if you want to even pretend to be a responsible, competent professional.
Incidentally, an excellent way to reduce the number of "we received 3 login attempts on our SSH box" complaints is to deal with all of them, thus decreasing incident occurence, which will of course result in a corresponding decrease in complaints. An even better way is to run your operation in such a way that you detect and deal with as many such things as possible before anybody needs to file a complaint. After all, if they can see the traffic arriving on their side, you can see it leaving on yours.
---rsk
[1] I note, for example, that Charter -- which is mentioned in the original message in this thread -- currently has a market capitalization of 116.63 billion dollars. Surely they could spare a tiny fraction of that to appropriately staff a 24x7 multi-lingual abuse desk with senior engineers and empower/equip them to do what needs to done. That's what a professional operation would do.
On 4/28/20 11:57 AM, Mike Hammett wrote:
I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it.
On the one hand, if you have programmed your computer to originate email to lots of people without any review to consider the email's accuracy or whether the recipients would welcome it... then you are being inconsiderate and likely spamming. You should stop doing that. You're just contributing to the noise. On Tue, Apr 28, 2020 at 9:40 AM Matt Corallo via NANOG <nanog@nanog.org> wrote:
Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore.
On the other hand, if your network is the source of bad behavior that such automated messages complain of, you should be far more concerned with the criminal in your midst than any rudeness on the part of whoever made the report. Consider carefully why you didn't already know that one of your users' computers was scanning ssh ports and hadn't already mitigated it. Are you being proactive or just responding to complaints? I last worked for an ISP in 2004 and even then it was a cinch to map a default route to a capture device and see who was spraying unrouted space with connection attempts. If you want to wait until someone complains, do you have the right to be annoyed by the form that complaint take? Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
I do, in this case, have such a right, because I know exactly what is going on in my network, and any non-automated system (ie, a human who reads the one sentence in the whois comments) does as well. Of course, I'm not going to get up in arms about it because this isn't about me (I just put the abuse contact in comments and the abuse field set to @example.com and the noise goes away, though I admit I'd prefer to actually see the noise, in case there is something interesting), its about the fact that the abuse system is now nigh on useless for the big players, who are sadly often the source of things that really should be shut down. Matt On 4/29/20 5:05 PM, William Herrin wrote:
On 4/28/20 11:57 AM, Mike Hammett wrote:
I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it.
On the one hand, if you have programmed your computer to originate email to lots of people without any review to consider the email's accuracy or whether the recipients would welcome it... then you are being inconsiderate and likely spamming. You should stop doing that. You're just contributing to the noise.
On Tue, Apr 28, 2020 at 9:40 AM Matt Corallo via NANOG <nanog@nanog.org> wrote:
Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails. This is why folks don't have abuse contacts that are responsive to real issues anymore.
On the other hand, if your network is the source of bad behavior that such automated messages complain of, you should be far more concerned with the criminal in your midst than any rudeness on the part of whoever made the report. Consider carefully why you didn't already know that one of your users' computers was scanning ssh ports and hadn't already mitigated it. Are you being proactive or just responding to complaints?
I last worked for an ISP in 2004 and even then it was a cinch to map a default route to a capture device and see who was spraying unrouted space with connection attempts. If you want to wait until someone complains, do you have the right to be annoyed by the form that complaint take?
On Wed, Apr 29, 2020 at 3:36 PM Matt Corallo <nanog@as397444.net> wrote:
I do, in this case, have such a right, because I know exactly what is going on in my network,
Hi Matt, If someone in your address space is knock-knocking on a stranger's ssh ports (your example, not mine), you have some work to do convincing me of your supreme security. Don't get me wrong: I've felt the pain of the auto-generated spam complaint that scrubbed exactly the headers I need to figure out what happened. But if you're round-filing complaints solely because they were generated by a program, you're doing it really wrong. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Ah, I'd pasted the following in a response to the mail you responded to: ~$ whois 208.68.4.129 Comment: --------------- Comment: 208.68.4.128/28 and 208.68.7.128/28 provide privacy services Comment: (incl running tor exit node(s)!) Comment: Abuse reports will be handled but there is likely not much that can be done. Comment: Send abuse to abuse at privacysvcs net. Comment: --------------- ... RAbuseEmail: see-comments-no-bots@example.com Now you can decide to pass judgement on the idea that someone may want to run a Tor exit node (my data says a good chunk of users are regular internet users in Iran, and I'm not routing hidden service traffic, where a most of the morally-repugnant crap happens), but that's beside the point - outbound ports are super limited, and outbound SSH is rate-limited appropriately. Matt On 4/29/20 7:00 PM, William Herrin wrote:
On Wed, Apr 29, 2020 at 3:36 PM Matt Corallo <nanog@as397444.net> wrote:
I do, in this case, have such a right, because I know exactly what is going on in my network,
Hi Matt,
If someone in your address space is knock-knocking on a stranger's ssh ports (your example, not mine), you have some work to do convincing me of your supreme security.
Don't get me wrong: I've felt the pain of the auto-generated spam complaint that scrubbed exactly the headers I need to figure out what happened. But if you're round-filing complaints solely because they were generated by a program, you're doing it really wrong.
Regards, Bill Herrin
On Wed, Apr 29, 2020 at 4:19 PM Matt Corallo <nanog@as397444.net> wrote:
Now you can decide to pass judgement on the idea that someone may want to run a Tor exit node
Wait... You run a TOR exit node and you find it unreasonable that folks would send you automated abuse complaints? In my dictionary under "chutzpah" I'm penciling in your name. "Not much that can be done"? Try this on for size: when people tell you to stop sending them packets, stop sending them packets. Plenty of folks don't want to receive packets from people trying to hide behind your mask. If you're going to run a TOR service, you absolutely owe it to them to respect those wishes. To even THINK about running a TOR node you should be more than going the extra mile on this. Like fully automated web site to let me block traffic exiting your network towards mine extra mile Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
IMO, the answer is balance. - Handful of SSH connection attempts against a server. Nobody got in, security hardening did it's job. I don't think that is worth reporting. - Constant brute force SSH attempts from a given source over an extended period of time, or a clear pattern of probing, yes, report that. As much as some pound on the table and say there shouldn't be, there is always going to be a level of background 'cruft' traffic between networks. Forever. An argument was made somewhere in here that "scanning" is , by itself, a problem. I disagree. There are many legitimate use cases for certain types of scans, maps, etc. It's true that it sometimes can be difficult to distinguish between a malicious scan and an innocent one. Proposing a solution of "stop all scanning" is absolutely a baby/bathwater angle. I would also challenge those that say "Oh well all these companies should have perfect flow logs and pay an army of engineers to analyze them for these 5 specific TCP SYNs from 2 weeks ago." I would bet you probably couldn't do that either. On Tue, Apr 28, 2020 at 11:59 AM Mike Hammett <nanog@ics-il.net> wrote:
I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things:
1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports.
Do these mirror others' observations with security issues and how abuse desks respond?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
It is rather easy to block SSH cracking attempts from your own side. Rarely do they put any significant load on your network or computer. I would sympathize with this except for the fact that abuse desks won't even respond to DDoS attacks, something that can't be fixed on your own end without spending a lot of money. That needs to be fixed first before worrying about password cracking. On Tue, Apr 28, 2020 at 8:58 AM Mike Hammett <nanog@ics-il.net> wrote:
I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things:
1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports.
Do these mirror others' observations with security issues and how abuse desks respond?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
The machines that are ssh probing are probably doing other stuff. Take the win that you have been informed about a compromised machine and get it cleaned / quarantined. -- Mark Andrews
On 30 Apr 2020, at 06:20, Bottiger <bottiger10@gmail.com> wrote:
It is rather easy to block SSH cracking attempts from your own side. Rarely do they put any significant load on your network or computer.
I would sympathize with this except for the fact that abuse desks won't even respond to DDoS attacks, something that can't be fixed on your own end without spending a lot of money.
That needs to be fixed first before worrying about password cracking.
On Tue, Apr 28, 2020 at 8:58 AM Mike Hammett <nanog@ics-il.net> wrote: I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things:
1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports.
Do these mirror others' observations with security issues and how abuse desks respond?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
On 2020-04-28 18:57, Mike Hammett wrote:
I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things:
1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports.
Do these mirror others' observations with security issues and how abuse desks respond?
Although many people write here - no need to worry about such minor things, i strongly disagree. If someone littering server ssh logs for an hour, most likely on the other side: 1) A botnet-infected computer that needs to be fixed. Today ssh bruteforce, tomorrow spam and hosting scam and very real financial losses for some people. 2) A hacker who is looking for an easy target. If he succeed, law enforcement will come to you tomorrow and might waste lot of your time. And sometimes it’s some kid who, possibly will get an early warning, will not break his life by getting a criminal term. And how to fight with lazy operators who start differentiate on abuse, which is worth their majestic attention. I send proper abuse reports if there is no reaction to them - I make a null route of incoming SYN requests on all my servers, and sometimes i share an IP list with other operators who want to live in a "clean" internet, and not in a garbage dump. I have several resources hosted, so at the end techies of those "majestic ISPs" come with tears, when their customers start to torture their support and sales, and beg to be unlocked and most start to read abuse mailbox. Or they just lose customers.
I don't think anyone in this thread meant to suggest that there is no reason to be concerned about such scans, as you point out they are occasionally compromised hosts and the like. The real question here is what is the cost of sending all that mail? The abuse system as it exists today is largely useless - why do you think we so regularly see posts on NANOG asking for introductions to a given network? If you've tried to actually get a hold of someone to address abuse at AWS/GCP/OVH/Azure/etc/etc I'm sure you're aware that, more often than not, you just can't. For at least a few of those, this isn't because they don't care, its because 99% of the mails to their abuse contacts are things that don't make sense to take action on (see other comments in this thread for a million reasons why not). Nothing wrong with just using fail2ban for its intended purpose - banning IPs that fail logins, but using it to send mail to abuse leaves us in a world without an abuse system. Matt On 4/29/20 4:51 PM, Denys Fedoryshchenko wrote:
On 2020-04-28 18:57, Mike Hammett wrote:
I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things:
1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports.
Do these mirror others' observations with security issues and how abuse desks respond?
Although many people write here - no need to worry about such minor things, i strongly disagree.
If someone littering server ssh logs for an hour, most likely on the other side: 1) A botnet-infected computer that needs to be fixed. Today ssh bruteforce, tomorrow spam and hosting scam and very real financial losses for some people. 2) A hacker who is looking for an easy target. If he succeed, law enforcement will come to you tomorrow and might waste lot of your time. And sometimes it’s some kid who, possibly will get an early warning, will not break his life by getting a criminal term.
And how to fight with lazy operators who start differentiate on abuse, which is worth their majestic attention. I send proper abuse reports if there is no reaction to them - I make a null route of incoming SYN requests on all my servers, and sometimes i share an IP list with other operators who want to live in a "clean" internet, and not in a garbage dump. I have several resources hosted, so at the end techies of those "majestic ISPs" come with tears, when their customers start to torture their support and sales, and beg to be unlocked and most start to read abuse mailbox. Or they just lose customers.
There have been a lot of philosophical tangents to the original request. Are others seeing similar things? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mike Hammett" <nanog@ics-il.net> To: "North American Network Operators' Group" <nanog@nanog.org> Sent: Tuesday, April 28, 2020 10:57:10 AM Subject: Abuse Desks I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things: 1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports. Do these mirror others' observations with security issues and how abuse desks respond? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
I did not want to target anyone in particular, so I have responded to my original e-mail. I have seen comments about the big guys just ignoring everything. I have had a non-zero number of e-mails from each of Azure, GCP, AWS, and Hetzner claiming that they have acted on my report. It isn't a significant percentage, but they're doing something about some of the reports. I don't think I've seen anything back from the biggest offender, Digital Ocean, other than auto-responders acknowledging the report. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mike Hammett" <nanog@ics-il.net> To: "North American Network Operators' Group" <nanog@nanog.org> Sent: Tuesday, April 28, 2020 10:57:10 AM Subject: Abuse Desks I noticed over the weekend that a Fail2Ban instance's complain function wasn't working. I fixed it. I've noticed a few things: 1) Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR's whois, it just shows the ISP abuse information. Weird, but so rare it's probably just an anomaly. However, almost everything I see in LACNIC's region is returned with only the LACNIC abuse information when the ones I've checked on LACNIC's whois list valid abuse information for that prefix. Can anyone confirm they've seen similar behavior out of Abusix? I reached out to them, but haven't heard back. 2) Digital Ocean hits my radar far more than any other entity. 3) Azure shows up a lot less than GCP or AWS, which are about similar to each other. 4) Around 5% respond saying it's been addressed (or why it's not in the event of security researchers) within a couple hours. The rest I don't know. I've had a mix of small and large entities in that response. 5) HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries. 6) Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don't have enough information to act on, requesting relevant log file entries... which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered. 7) The LACNIC region has about 1/3 of my reports. Do these mirror others' observations with security issues and how abuse desks respond? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Maybe there is a market opportunity there? Develop reporting standard (or use one that was posted here), then develop reporting, processing and analytic tools, and then provide it as a service? Looks like a nice use case how to utilise clouds ;) Kind regards, Andrey Mike Hammett писал 2020-04-30 08:10:
I did not want to target anyone in particular, so I have responded to my original e-mail. I have seen comments about the big guys just ignoring everything. I have had a non-zero number of e-mails from each of Azure, GCP, AWS, and Hetzner claiming that they have acted on my report. It isn't a significant percentage, but they're doing something about some of the reports.
I don't think I've seen anything back from the biggest offender, Digital Ocean, other than auto-responders acknowledging the report.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
Is there really not already an RFC or BCP for a standard abuse reporting format? Even if what constitutes abuse is up for debate, a formatting standard would make processing and automation much easier. -Matt On Thu, Apr 30, 2020 at 9:18 AM Andrey Kostin <ankost@podolsk.ru> wrote:
Maybe there is a market opportunity there? Develop reporting standard (or use one that was posted here), then develop reporting, processing and analytic tools, and then provide it as a service? Looks like a nice use case how to utilise clouds ;)
Kind regards, Andrey
Mike Hammett писал 2020-04-30 08:10:
I did not want to target anyone in particular, so I have responded to my original e-mail. I have seen comments about the big guys just ignoring everything. I have had a non-zero number of e-mails from each of Azure, GCP, AWS, and Hetzner claiming that they have acted on my report. It isn't a significant percentage, but they're doing something about some of the reports.
I don't think I've seen anything back from the biggest offender, Digital Ocean, other than auto-responders acknowledging the report.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
-- Matt Erculiani ERCUL-ARIN
Yo Mike! On Thu, 30 Apr 2020 07:10:19 -0500 (CDT) Mike Hammett <nanog@ics-il.net> wrote:
I don't think I've seen anything back from the biggest offender, Digital Ocean, other than auto-responders acknowledging the report.
I gave up on Digital Ocean. I blackhole all their nets. Eventually I had to white list a couple of their IPs. Life got better. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can't measure it, you can't improve it." - Lord Kelvin
participants (26)
-
Andrey Kostin -
Bottiger -
Brian J. Murrell -
bzs＠theworld.com -
Chris Adams -
Dan Hollis -
Denys Fedoryshchenko -
Gary E. Miller -
J. Hellenthal -
Joe Greco -
Laszlo Hanyecz -
Mark Andrews -
Matt Corallo -
Matt Erculiani -
Matt Palmer -
Mel Beckman -
Mike Hammett -
Mukund Sivaraman -
Rich Kulawiec -
Sabri Berisha -
Shane Ronan -
sronan＠ronan-online.com -
Stephen Satchell -
Tom Beecher -
Valdis Klētnieks -
William Herrin