All, Just wanted to apologize for the attack over the weekend. The posts came from a email address that was subscribed to the list, so it was not subjected to moderation. While a filter was added to block further posts (which were made in a short time window), there were existing message queues that were not cleared in a timely basis. As Job Snijders (a fellow Communications Committee member) noted in an earlier post, we will be implementing some additional protection mechanisms to prevent this style of incident from happening again. We will be more aggressively moderating posts from addresses who have not posted recently, in addition to other filtering mechanisms. Regards, Larry Blunk NANOG Communications Committee Admins@nanog.org
On Mon, Oct 26, 2015 at 03:17:37PM -0400, Larry Blunk wrote:
Just wanted to apologize for the attack over the weekend. The posts came from a email address that was subscribed to the list, so it was not subjected to moderation. While a filter was added to block further posts (which were made in a short time window), there were existing message queues that were not cleared in a timely basis.
As Job Snijders (a fellow Communications Committee member) noted in an earlier post, we will be implementing some additional protection mechanisms to prevent this style of incident from happening again. We will be more aggressively moderating posts from addresses who have not posted recently, in addition to other filtering mechanisms.
To add to that: several people reached out off-list, offering help and recommendations. We'll be following those up in the next few days. Thank you for your support! Some people found the admins@nanog.org readership unresponsive, but I assure you this is not the case under normal circumstances. The admins mail distribution was clogged up for the same reasons as the main list. We'll work on improving our reachability. Kind regards, Job
On 10/26/2015 3:00 PM, Job Snijders wrote:
On Mon, Oct 26, 2015 at 03:17:37PM -0400, Larry Blunk wrote:
Just wanted to apologize for the attack over the weekend. The posts came from a email address that was subscribed to the list, so it was not subjected to moderation. While a filter was added to block further posts (which were made in a short time window), there were existing message queues that were not cleared in a timely basis.
To add to that: several people reached out off-list, offering help and recommendations. We'll be following those up in the next few days. Thank you for your support!
I'd made a post to the members list, in the vain hope that it was on a different server, and perhaps might go through (and it certainly did, bright and early this morning). There's a couple of things I'd said that are worth noting here. For those who didn't visit the archives, where it was at least possible to see that the deluge was noticed by folks, I'd suggest a quick look. http://mailman.nanog.org/pipermail/nanog/2015-October/date.html In my very unscientific method of knowing approximately how many lines were visible in my browser, I guesstimate that there were about 1750 messages, and they were issued in the span of perhaps twenty minutes (perhaps less), before the alarm bells went off, and the problem was addressed. From start: http://mailman.nanog.org/pipermail/nanog/2015-October/080150.html to finish: http://mailman.nanog.org/pipermail/nanog/2015-October/081902.html For those who quickly looked at the archives, it was clear that others had noticed that there was a problem (I even had off list emails with a couple of them). I might have been more draconian in the clean up (i.e. purge the queues, including valid emails), but honestly, that was a pretty tough assault, and it's a good object lesson on what might happen. You *are* all updating your security approaches and data recovery plans, right? Thanks to both Job Snijders and Larry Blunk. The check is in the mail. :-} -- Coffee, coffee, everywhere, And all the cups did clink; Coffee, coffee, everywhere, Nor any drop to drink. (Apologies to Coleridge)
Thank you team On Monday, October 26, 2015, Job Snijders <job@instituut.net> wrote:
On Mon, Oct 26, 2015 at 03:17:37PM -0400, Larry Blunk wrote:
Just wanted to apologize for the attack over the weekend. The posts came from a email address that was subscribed to the list, so it was not subjected to moderation. While a filter was added to block further posts (which were made in a short time window), there were existing message queues that were not cleared in a timely basis.
As Job Snijders (a fellow Communications Committee member) noted in an earlier post, we will be implementing some additional protection mechanisms to prevent this style of incident from happening again. We will be more aggressively moderating posts from addresses who have not posted recently, in addition to other filtering mechanisms.
To add to that: several people reached out off-list, offering help and recommendations. We'll be following those up in the next few days. Thank you for your support!
Some people found the admins@nanog.org <javascript:;> readership unresponsive, but I assure you this is not the case under normal circumstances. The admins mail distribution was clogged up for the same reasons as the main list. We'll work on improving our reachability.
Kind regards,
Job
Larry, Thank you for the work you and others do behind the scenes to make the nanog list available and functional. Mike- On 10/26/2015 12:17 PM, Larry Blunk wrote:
All, Just wanted to apologize for the attack over the weekend. The posts came from a email address that was subscribed to the list, so it was not subjected to moderation. While a filter was added to block further posts (which were made in a short time window), there were existing message queues that were not cleared in a timely basis.
As Job Snijders (a fellow Communications Committee member) noted in an earlier post, we will be implementing some additional protection mechanisms to prevent this style of incident from happening again. We will be more aggressively moderating posts from addresses who have not posted recently, in addition to other filtering mechanisms.
Regards, Larry Blunk NANOG Communications Committee Admins@nanog.org
-- Mike Ireton WillitsOnline LLC
Thank you Larry and Job for the responses, mitigation steps taken, and work to further resolve these kind of events. Food for thought for the rest of us out there. Had there been a network attack on Sunday (for example) and several of these lists (multiple received this spam "attack") were switched to require a moderator to filter all emails manually. How quickly would information have gotten out through the networking community? No NANOG and Outages are not the only places I check or subscribe to but I DO check them to see if anyone else is reporting anything. And they are some of the places I would report real network problems to. For me this didn't kill my weekend or destroy my ability to check my emails. I know for many others it didn't either. I use my android mail client to group emails with the same subject and after checking multiple of them I didn't worry about those threads anymore. Yes I received several hundred emails about it but I was still able to function and watch for anything that came in that would note a threat to the network as a whole. Maybe if this event has caused such a stir and inconvenience we should look at what we are doing and how we are doing it. These lists are tools that can be valuable to get information out to a large group of people. Anything that would block that I would consider a threat to the purpose of the list as well. This event caused blockage as well and the NANOG staff are looking into mitigation for that. Thank you Brian
To: nanog@nanog.org From: ljb@merit.edu Subject: NANOG list attack Date: Mon, 26 Oct 2015 15:17:37 -0400
All, Just wanted to apologize for the attack over the weekend. The posts came from a email address that was subscribed to the list, so it was not subjected to moderation. While a filter was added to block further posts (which were made in a short time window), there were existing message queues that were not cleared in a timely basis.
As Job Snijders (a fellow Communications Committee member) noted in an earlier post, we will be implementing some additional protection mechanisms to prevent this style of incident from happening again. We will be more aggressively moderating posts from addresses who have not posted recently, in addition to other filtering mechanisms.
Regards, Larry Blunk NANOG Communications Committee Admins@nanog.org
On 10/26/2015 03:17 PM, Larry Blunk wrote:
As Job Snijders (a fellow Communications Committee member) noted in an earlier post, we will be implementing some additional protection mechanisms to prevent this style of incident from happening again. We will be more aggressively moderating posts from addresses who have not posted recently, in addition to other filtering mechanisms.
For what it's worth, while I did see all of these that made it through the list itself, the larger portion that I saw did not come through the list but were sent directly to me, and the Received header trail shows that those did not come through the nanog mailman. So I applaud what you do with the list itself, but it wouldn't have made (and won't make, in the future) much difference, since e-mails were sent out bypassing the list server. And thanks for this note.
participants (7)
-
Brian R -
Job Snijders -
Lamar Owen -
Larry Blunk -
Mehmet Akcin -
Mike -
Shrdlu