outages list badly broken, flooding messages
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts. Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy. FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it.
Not seeing this flooding in my inbox… -Mike
On Apr 27, 2025, at 09:27, Glenn McGurrin via NANOG <nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
Same. I sent an email to OUTAGES Mailing List <outages@outages.org> at noon today and haven't seen any post or replies. On Sun, Apr 27, 2025 at 12:33 PM Mike Lyon via NANOG <nanog@lists.nanog.org> wrote:
Not seeing this flooding in my inbox…
-Mike
On Apr 27, 2025, at 09:27, Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/Y37JT7LU...
I'm looking into this right now. Apologies. On Sun, Apr 27, 2025 at 12:43:26PM -0400, Josh Luthman via NANOG wrote:
Same.
I sent an email to OUTAGES Mailing List <outages@outages.org> at noon today and haven't seen any post or replies.
On Sun, Apr 27, 2025 at 12:33 PM Mike Lyon via NANOG <nanog@lists.nanog.org> wrote:
Not seeing this flooding in my inbox…
-Mike
On Apr 27, 2025, at 09:27, Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list
Reach out if you need help. On Sun, Apr 27, 2025, 1:07 PM Jared Mauch <jared@puck.nether.net> wrote:
I'm looking into this right now. Apologies.
Same.
I sent an email to OUTAGES Mailing List <outages@outages.org> at noon today and haven't seen any post or replies.
On Sun, Apr 27, 2025 at 12:33 PM Mike Lyon via NANOG < nanog@lists.nanog.org> wrote:
Not seeing this flooding in my inbox…
-Mike
On Apr 27, 2025, at 09:27, Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run
but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the
On Sun, Apr 27, 2025 at 12:43:26PM -0400, Josh Luthman via NANOG wrote: list, list
Hi, On Sun, Apr 27, 2025 at 09:33:01AM -0700, Mike Lyon via NANOG wrote:
Not seeing this flooding in my inbox…
I am, and so is the archive on marc.info: https://marc.info/?t=174404380000002&r=1&w=2 Thanks, Andy
I want to sincerely apologize for the current state, this situation should not have unfolded in this manner. Our original intention was to conduct a gradual migration to minimize any disruption to the list. However, given the persistent challenges we’ve encountered with PUCK (Mailman), we are now opting for a decisive big bang cutover to ensure we move forward effectively and efficiently. Thank you for your understanding as we navigate this transition. I will provide additional updates soon. /vrode On Sun, 2025-04-27 at 12:26 -0400, Glenn McGurrin via NANOG wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
Meaning fixing Puck or the divorce of Outages from Puck? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "virendra rode via NANOG" <nanog@lists.nanog.org> To: nanog@lists.nanog.org Cc: "virendra rode" <virendra.rode@gmail.com> Sent: Sunday, April 27, 2025 1:19:28 PM Subject: Re: outages list badly broken, flooding messages I want to sincerely apologize for the current state, this situation should not have unfolded in this manner. Our original intention was to conduct a gradual migration to minimize any disruption to the list. However, given the persistent challenges we’ve encountered with PUCK (Mailman), we are now opting for a decisive big bang cutover to ensure we move forward effectively and efficiently. Thank you for your understanding as we navigate this transition. I will provide additional updates soon. /vrode On Sun, 2025-04-27 at 12:26 -0400, Glenn McGurrin via NANOG wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/A72XBAUV...
Making the leap from PUCK to PCH. Woody has graciously volunteered to assist with this transition /vrode On Sun, 2025-04-27 at 14:03 -0500, Mike Hammett wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
----- Original Message ----- From: "virendra rode via NANOG" <nanog@lists.nanog.org> To: nanog@lists.nanog.org Cc: "virendra rode" <virendra.rode@gmail.com> Sent: Sunday, April 27, 2025 1:19:28 PM Subject: Re: outages list badly broken, flooding messages
I want to sincerely apologize for the current state, this situation should not have unfolded in this manner. Our original intention was to conduct a gradual migration to minimize any disruption to the list. However, given the persistent challenges we’ve encountered with PUCK (Mailman), we are now opting for a decisive big bang cutover to ensure we move forward effectively and efficiently.
Thank you for your understanding as we navigate this transition.
I will provide additional updates soon.
/vrode
On Sun, 2025-04-27 at 12:26 -0400, Glenn McGurrin via NANOG wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/A72XBAUV...
let me know if there's any help I can offer On 4/27/2025 4:35 PM, virendra rode via NANOG wrote:
Making the leap from PUCK to PCH. Woody has graciously volunteered to assist with this transition
/vrode
On Sun, 2025-04-27 at 14:03 -0500, Mike Hammett wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
----- Original Message ----- From: "virendra rode via NANOG" <nanog@lists.nanog.org> To: nanog@lists.nanog.org Cc: "virendra rode" <virendra.rode@gmail.com> Sent: Sunday, April 27, 2025 1:19:28 PM Subject: Re: outages list badly broken, flooding messages
I want to sincerely apologize for the current state, this situation should not have unfolded in this manner. Our original intention was to conduct a gradual migration to minimize any disruption to the list. However, given the persistent challenges we’ve encountered with PUCK (Mailman), we are now opting for a decisive big bang cutover to ensure we move forward effectively and efficiently.
Thank you for your understanding as we navigate this transition.
I will provide additional updates soon.
/vrode
On Sun, 2025-04-27 at 12:26 -0400, Glenn McGurrin via NANOG wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/A72XBAUV...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/C5I6KCVX...
On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation. Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3. Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item. It's extremely hard when you are carrying around nearly 30 years of legacy as well. Either way I expect it all to be resolved in the next ~24 hours or less one way or another. - Jared
Can you just do a new instance and bring over the membership? The archive can just be left alone at a different URL. On Sun, Apr 27, 2025, 6:49 PM Jared Mauch via NANOG <nanog@lists.nanog.org> wrote:
On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation.
Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3.
Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item.
It's extremely hard when you are carrying around nearly 30 years of legacy as well.
Either way I expect it all to be resolved in the next ~24 hours or less one way or another.
- Jared _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/IE26PRTF...
Thanks for your work. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Jared Mauch" <jared@puck.nether.net> To: "North American Network Operators Group" <nanog@lists.nanog.org> Cc: "Mike Hammett" <nanog@ics-il.net> Sent: Sunday, April 27, 2025 5:48:36 PM Subject: Re: outages list badly broken, flooding messages On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation. Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3. Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item. It's extremely hard when you are carrying around nearly 30 years of legacy as well. Either way I expect it all to be resolved in the next ~24 hours or less one way or another. - Jared
Just checking but is this also impacting cisco-nsp? I noticed I haven't seen a message in that group since March or so. Good luck with the issues. -Drew -----Original Message----- From: Jared Mauch via NANOG <nanog@lists.nanog.org> Sent: Sunday, April 27, 2025 6:49 PM To: North American Network Operators Group <nanog@lists.nanog.org> Cc: Jared Mauch <jared@puck.nether.net> Subject: Re: outages list badly broken, flooding messages On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation. Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3. Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item. It's extremely hard when you are carrying around nearly 30 years of legacy as well. Either way I expect it all to be resolved in the next ~24 hours or less one way or another. - Jared _______________________________________________ NANOG mailing list https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_archives_list_nanog-40lists.nanog.org_message_IE26PRTF2SLZ64Y55V3W5NGCEPHSFJGX_&d=DwICAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=PcplBR5uejKfQoNHWRNX_DJDYeuYO0xmU57etr2AeRFgjtyd6FcHhONFPkKpDpfU&s=_jZEjP0Xm8nxSRPdSeyqvZf1nFlKMc5HsQEvBHVH9Ek&e=
On Apr 27, 2025, at 5:48 PM, Jared Mauch via NANOG <nanog@lists.nanog.org> wrote:
On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation.
Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3.
Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item.
It's extremely hard when you are carrying around nearly 30 years of legacy as well.
Either way I expect it all to be resolved in the next ~24 hours or less one way or another.
- Jared _______________________________________________
How goes the transition? Any help/resources/etc. needed? Or are things all sorted out and working? -Andy
Testing is currently in progress, and as the newly implemented mailman3, I'm adapting to the system and retrieving data from the archives in preparation for go live on PCH infra. It's important to note that we may require the community to re-subscribe, we are still trying to determine that. More updates will be provided soon. /vrode On Sat, 2025-05-03 at 23:18 -0500, Andy Ringsmuth via NANOG wrote:
On Apr 27, 2025, at 5:48 PM, Jared Mauch via NANOG <nanog@lists.nanog.org> wrote:
On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation.
Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3.
Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item.
It's extremely hard when you are carrying around nearly 30 years of legacy as well.
Either way I expect it all to be resolved in the next ~24 hours or less one way or another.
- Jared _______________________________________________
How goes the transition? Any help/resources/etc. needed? Or are things all sorted out and working?
-Andy _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/6I7B4KU7...
If it requires everyone to resubscribe why don't we just start up a whole new instance? On Sun, May 4, 2025, 8:01 PM virendra rode via NANOG <nanog@lists.nanog.org> wrote:
Testing is currently in progress, and as the newly implemented mailman3, I'm adapting to the system and retrieving data from the archives in preparation for go live on PCH infra. It's important to note that we may require the community to re-subscribe, we are still trying to determine that.
More updates will be provided soon.
/vrode
On Sat, 2025-05-03 at 23:18 -0500, Andy Ringsmuth via NANOG wrote:
On Apr 27, 2025, at 5:48 PM, Jared Mauch via NANOG <nanog@lists.nanog.org> wrote:
On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation.
Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3.
Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item.
It's extremely hard when you are carrying around nearly 30 years of legacy as well.
Either way I expect it all to be resolved in the next ~24 hours or less one way or another.
- Jared _______________________________________________
How goes the transition? Any help/resources/etc. needed? Or are things all sorted out and working?
-Andy _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/6I7B4KU7...
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/F2COSMLQ...
Virendra, How’s the transition going? May any of us help with testing or data migration or any of the associated challenges? ---- Andy Ringsmuth andy@andyring.com
On May 4, 2025, at 7:01 PM, virendra rode <virendra.rode@gmail.com> wrote:
Testing is currently in progress, and as the newly implemented mailman3, I'm adapting to the system and retrieving data from the archives in preparation for go live on PCH infra. It's important to note that we may require the community to re-subscribe, we are still trying to determine that.
More updates will be provided soon.
/vrode
On Sat, 2025-05-03 at 23:18 -0500, Andy Ringsmuth via NANOG wrote:
On Apr 27, 2025, at 5:48 PM, Jared Mauch via NANOG <nanog@lists.nanog.org> wrote:
On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation.
Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3.
Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item.
It's extremely hard when you are carrying around nearly 30 years of legacy as well.
Either way I expect it all to be resolved in the next ~24 hours or less one way or another.
- Jared _______________________________________________
How goes the transition? Any help/resources/etc. needed? Or are things all sorted out and working?
-Andy _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/6I7B4KU7...
Do others feel like it's about time to just start a new list? On Mon, May 12, 2025, 10:34 AM Andy Ringsmuth via NANOG < nanog@lists.nanog.org> wrote:
Virendra,
How’s the transition going? May any of us help with testing or data migration or any of the associated challenges?
---- Andy Ringsmuth andy@andyring.com
On May 4, 2025, at 7:01 PM, virendra rode <virendra.rode@gmail.com> wrote:
Testing is currently in progress, and as the newly implemented mailman3, I'm adapting to the system and retrieving data from the archives in preparation for go live on PCH infra. It's important to note that we may require the community to re-subscribe, we are still trying to determine that.
More updates will be provided soon.
/vrode
On Sat, 2025-05-03 at 23:18 -0500, Andy Ringsmuth via NANOG wrote:
On Apr 27, 2025, at 5:48 PM, Jared Mauch via NANOG <nanog@lists.nanog.org> wrote:
On Sun, Apr 27, 2025 at 02:03:23PM -0500, Mike Hammett via NANOG wrote:
Meaning fixing Puck or the divorce of Outages from Puck?
I'm working on fixing puck, there's a long backstory of what transpired here which resulted in having to revert to a backup from 2023 due to some issues related to a series of events that made it far easier to just revert to a backup than attempt to restore the stuation.
Most things should be back now but the state of mailman in the modern universe is an ongoing problem. It depended on python2 and was never updated to support python3.
Migrating to mailman3 is also a very large task, so in the midst of all the restoration efforts that had to be undertaken this was a lower priority item.
It's extremely hard when you are carrying around nearly 30 years of legacy as well.
Either way I expect it all to be resolved in the next ~24 hours or less one way or another.
- Jared _______________________________________________
How goes the transition? Any help/resources/etc. needed? Or are things all sorted out and working?
-Andy _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/6I7B4KU7...
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/BX7ISJKQ...
I do understand this is a volunteer run list
FWIW up to 96 copies now as I write this, it's been going on for about
an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it.
So let me get this straight. 1. You saw a problem with something. 2. You didn't report that problem where it was occuring. 3. You went somewhere else to complain about it. 4. You don't expect anyone to fix it on a Sunday, but you do expect someone to turn it off on a Sunday. Yeesh. On Sun, Apr 27, 2025 at 12:28 PM Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
I am not getting the flood of emails from the outages list, but I am assuming that the OP didn't want to add to the loop and that's why he posted here instead. On 4/27/25 4:44 PM, Tom Beecher via NANOG wrote:
I do understand this is a volunteer run list
FWIW up to 96 copies now as I write this, it's been going on for about
an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it.
So let me get this straight.
1. You saw a problem with something. 2. You didn't report that problem where it was occuring. 3. You went somewhere else to complain about it. 4. You don't expect anyone to fix it on a Sunday, but you do expect someone to turn it off on a Sunday.
Yeesh.
On Sun, Apr 27, 2025 at 12:28 PM Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/76B3ECDZ...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/X3PK72KX...
Not quite, I sent two emails initially as I use different addresses for outages and nanog, and so needed to send with the appropriate from lines to avoid it being rejected as a non member posting, the messages were identical other than the to/from lines, but the other went to what I could find easily as contact for Jared and the normal list admin address plus outages and discussion, I hesitated on including the list because I feared adding to the issue, but ultimately decided to include the list, but I also didn't have any confidence it would go through on the list, and with the specifics of the issue unknown and the address I had for Jared, who I understood to be the sole admin of puck from limited things I'd seen in the discussions of the issue, and his email being @puck (the server with the list that was misbehaving), I wasn't sure any of the email addresses I had for him would have gone through either, so hedged bets, also to see a bit if this was at least in part some interaction between his server and mine or a list wide issue, it sounds like it's something in the middle, affecting more than me, but not everyone. As for the turning off, as there's been some somewhat aggressive messages about the issues with the list recently, I wanted to be very clear that I understood the minimal expectations that were reasonable, and that unlike some other discussions were taken to imply I wasn't expecting large immediate efforts for free, and given the issue was ongoing and even if I've never spoken to him I know Jared by reputation I expected that there was a combination of a rather more difficult than might have first been assumed issue in play and limited time and resources available to fix it, but that I hoped that someone could find the likely very small amount of time to turn off the mailman service and either issue a hold or otherwise stop any mail queued in postfix or similar, which I would guess to take around 5 minutes or less, including logging in and such, which I had hope was something the admins of that list would have time for and would not overly intrude into their schedule, which it was clear actually fully fixing it would likely do. Perhaps I should have indicated this was a cross post in my initial email, I'd thought about it but at the time thought it unnecessary, I was also trying to make sure I didn't write aggressively or give an appearance of doing so while rather frustrated as my devices kept going off indicating new email distracting me. TLDR; In short Tom, while I appreciate it may not have been obvious or known to you given it was two different emails, I did in fact report the issue where it was occurring, though had reason to question if that email would get through, so in part because of that and in part for other reasons cross posted the message here at nanog, and I was trying to ask for a limited response while being clear that I understood I didn't have any standing to go pushing for much and that I did NOT expect anyone to go solve what was apparently quite a thorny issue immediately on a Sunday. On 4/27/2025 4:49 PM, Jason Canady via NANOG wrote:
I am not getting the flood of emails from the outages list, but I am assuming that the OP didn't want to add to the loop and that's why he posted here instead.
On 4/27/25 4:44 PM, Tom Beecher via NANOG wrote:
I do understand this is a volunteer run list
FWIW up to 96 copies now as I write this, it's been going on for about
an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it.
So let me get this straight.
1. You saw a problem with something. 2. You didn't report that problem where it was occuring. 3. You went somewhere else to complain about it. 4. You don't expect anyone to fix it on a Sunday, but you do expect someone to turn it off on a Sunday.
Yeesh.
On Sun, Apr 27, 2025 at 12:28 PM Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/ message/76B3ECDZIUCYS5ETGZ6CNMUTXTBYFGVR/
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/ X3PK72KX4RJ4ZC3I6IROAH53VT4R2XAB/
NANOG mailing list https://lists.nanog.org/archives/list/ nanog@lists.nanog.org/message/CJ52UFBY7MRAASZJVTTEBUSE2ZT4FM67/
A very simple message would have sufficed. "Hey all, looks like the outages list is on the fritz, seems to be spamming the same message out over and over. I tried reaching out there, but does anyone else have a contact by chance or could reach out?" No need for all the extra pontificating. And if alert spam is annoying you, squelch them on your device. You have the tools. Use them. I apologize if it seems like I'm lashing out at you personally. I'm not trying to. On Sun, Apr 27, 2025 at 6:10 PM Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
Not quite, I sent two emails initially as I use different addresses for outages and nanog, and so needed to send with the appropriate from lines to avoid it being rejected as a non member posting, the messages were identical other than the to/from lines, but the other went to what I could find easily as contact for Jared and the normal list admin address plus outages and discussion, I hesitated on including the list because I feared adding to the issue, but ultimately decided to include the list, but I also didn't have any confidence it would go through on the list, and with the specifics of the issue unknown and the address I had for Jared, who I understood to be the sole admin of puck from limited things I'd seen in the discussions of the issue, and his email being @puck (the server with the list that was misbehaving), I wasn't sure any of the email addresses I had for him would have gone through either, so hedged bets, also to see a bit if this was at least in part some interaction between his server and mine or a list wide issue, it sounds like it's something in the middle, affecting more than me, but not everyone.
As for the turning off, as there's been some somewhat aggressive messages about the issues with the list recently, I wanted to be very clear that I understood the minimal expectations that were reasonable, and that unlike some other discussions were taken to imply I wasn't expecting large immediate efforts for free, and given the issue was ongoing and even if I've never spoken to him I know Jared by reputation I expected that there was a combination of a rather more difficult than might have first been assumed issue in play and limited time and resources available to fix it, but that I hoped that someone could find the likely very small amount of time to turn off the mailman service and either issue a hold or otherwise stop any mail queued in postfix or similar, which I would guess to take around 5 minutes or less, including logging in and such, which I had hope was something the admins of that list would have time for and would not overly intrude into their schedule, which it was clear actually fully fixing it would likely do.
Perhaps I should have indicated this was a cross post in my initial email, I'd thought about it but at the time thought it unnecessary, I was also trying to make sure I didn't write aggressively or give an appearance of doing so while rather frustrated as my devices kept going off indicating new email distracting me.
TLDR; In short Tom, while I appreciate it may not have been obvious or known to you given it was two different emails, I did in fact report the issue where it was occurring, though had reason to question if that email would get through, so in part because of that and in part for other reasons cross posted the message here at nanog, and I was trying to ask for a limited response while being clear that I understood I didn't have any standing to go pushing for much and that I did NOT expect anyone to go solve what was apparently quite a thorny issue immediately on a Sunday.
On 4/27/2025 4:49 PM, Jason Canady via NANOG wrote:
I am not getting the flood of emails from the outages list, but I am assuming that the OP didn't want to add to the loop and that's why he posted here instead.
On 4/27/25 4:44 PM, Tom Beecher via NANOG wrote:
I do understand this is a volunteer run list
FWIW up to 96 copies now as I write this, it's been going on for about
an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it.
So let me get this straight.
1. You saw a problem with something. 2. You didn't report that problem where it was occuring. 3. You went somewhere else to complain about it. 4. You don't expect anyone to fix it on a Sunday, but you do expect someone to turn it off on a Sunday.
Yeesh.
On Sun, Apr 27, 2025 at 12:28 PM Glenn McGurrin via NANOG < nanog@lists.nanog.org> wrote:
I've already received 76 copies of "Re: [Outages-discussion] Lists dead - after action report?" and continue receiving them every few minutes, apologies, 77 copies now, this is clearly badly broken, I do understand this is a volunteer run list, but normally I do want to receive notifications as it's generally important if it's on this list, but currently I'm looking at having to block the list as it's sending my notifications nuts.
Also as an aside, I'd hesitated offering this before, not feeling like I was known enough or had enough standing in the community for it to be accepted and used, but if help is needed figuring out the issues, or if it would help the community to have a backup list, running mailman or potentially for added diversity running a different mailing list software, it's been a bit but I've used sympa before as well as mailman to run lists, I'm happy to host a secondary backup list(s) to offer some extra redundancy to the community (or if people really wanted to have someone take over the main I'm willing to, but I'm not asking for that or expecting anyone wants that), I know some mention has been made in a thread somewhere about adding redundancy to the list but nothing about the form of that redundancy.
FWIW up to 96 copies now as I write this, it's been going on for about an hour and a half at this point, I held of sending anything initially hoping it would be jumped on and at least the sending shut down even if that meant taking any remaining online part of the list down for now until there was time to fix it, as I get it, it's Sunday, people have plans, etc, at this point all I'd ask to be expedited is shutting it down until such time as people have time to fix it, not for people to abandon plans on a Sunday to root cause this and fully fix it. _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/ message/76B3ECDZIUCYS5ETGZ6CNMUTXTBYFGVR/
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/ X3PK72KX4RJ4ZC3I6IROAH53VT4R2XAB/
NANOG mailing list https://lists.nanog.org/archives/list/ nanog@lists.nanog.org/message/CJ52UFBY7MRAASZJVTTEBUSE2ZT4FM67/
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/BTNLDZUY...
participants (12)
-
Andy Ringsmuth -
Andy Smith -
Drew Weaver -
Glenn McGurrin -
Jared Mauch -
Jason Canady -
Jay Hennigan -
Josh Luthman -
Mike Hammett -
Mike Lyon -
Tom Beecher -
virendra rode