Hi folks, I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me. The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides. I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that. As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails. us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721 Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer? Thanks for your time! Mike Tindor
There are DISA folks lurking here. I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help. I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem. Scott On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote: Hi folks, I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me. The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides. I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that. As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails. us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721 Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer? Thanks for your time! Mike Tindor
Scott, Thanks for responding. Unfortunately, I think my situation is a little more dire, or at least involved. I probably should have said this before, but I had done TCP 25 outbound testing from our /23 to various .MIL MX's that I know were responding and could not establish a connection / get an SMTP banner. I could then go to Azure, or Digital Ocean, or somewhere else that I have a box and am able to make the outbound connection to the same MIL MXs that wouldn't respond to me from our /23. So it isn't a simple case of DNS not resolving, although we certainly did notice that issue. Fortunately, we do have nameservers in place that are external to our /23 and which are able to actually do the resolving. But your comment does remind that this definitely is not just a TCP 25 issue, as the MIL DNS servers are not responding to queries from our /23 hosts. The situation is difficult for multiple reasons: 1. inabiity to engage somebody from the other end - DISA 2. Unwillingness on my part to stab at a hornets nest and poke around trying to verify connections (other than TCP 25 to known MIL MXs) in DOD-land. 3. Not knowing exactly where to go from here The latest/last thing DISA told me was that I would have to get one of the people with MIL email addresses who can't email our customers to actually open a ticket with DISA. And that is fraught with problems since even if a MIL email user did open a ticket, they would not have any information about our network to convey to the Helpdesk -- and would have no way of answering questions that the Helpdesk asked, and also wouldn't be able to do any troubleshooting. I did realize a few days ago we had no ROA for the specific /23, and so I created one at ARIN. And we had no specific route object published for our /23, and I got one added. Been trying to clean up some old (and invalid) stuff that is in RADB from our larger /19, since we don't even own all the space in the /19 anymore and are only actively using a /23 from what we have left. Hoping to get that taken care of Monday. Everything has worked fine for 26 years, until Jun 1. But things change, and I'm obviously behind the times given that I didn't have proper ROA and route object in place. Mike Tindor On Sat, Jun 29, 2024 at 11:26 AM Scott Q. <qmail@top-consulting.net> wrote:
There are DISA folks lurking here.
I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help.
I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem.
Scott
On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote:
Hi folks,
I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me.
The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides.
I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that.
As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails.
us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721
Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer?
Thanks for your time!
Mike Tindor
All that sounds very familiar, I'm 100% sure it's the same issue. As I said, there are DISA folks here, they might reach out and give you further steps. They did in my case, you just have to be more patient / on the ball than I was... Good luck! On Saturday, 29/06/2024 at 11:44 Mike Tindor wrote: Scott, Thanks for responding. Unfortunately, I think my situation is a little more dire, or at least involved. I probably should have said this before, but I had done TCP 25 outbound testing from our /23 to various .MIL MX's that I know were responding and could not establish a connection / get an SMTP banner. I could then go to Azure, or Digital Ocean, or somewhere else that I have a box and am able to make the outbound connection to the same MIL MXs that wouldn't respond to me from our /23. So it isn't a simple case of DNS not resolving, although we certainly did notice that issue. Fortunately, we do have nameservers in place that are external to our /23 and which are able to actually do the resolving. But your comment does remind that this definitely is not just a TCP 25 issue, as the MIL DNS servers are not responding to queries from our /23 hosts. The situation is difficult for multiple reasons: 1. inabiity to engage somebody from the other end - DISA 2. Unwillingness on my part to stab at a hornets nest and poke around trying to verify connections (other than TCP 25 to known MIL MXs) in DOD-land. 3. Not knowing exactly where to go from here The latest/last thing DISA told me was that I would have to get one of the people with MIL email addresses who can't email our customers to actually open a ticket with DISA. And that is fraught with problems since even if a MIL email user did open a ticket, they would not have any information about our network to convey to the Helpdesk -- and would have no way of answering questions that the Helpdesk asked, and also wouldn't be able to do any troubleshooting. I did realize a few days ago we had no ROA for the specific /23, and so I created one at ARIN. And we had no specific route object published for our /23, and I got one added. Been trying to clean up some old (and invalid) stuff that is in RADB from our larger /19, since we don't even own all the space in the /19 anymore and are only actively using a /23 from what we have left. Hoping to get that taken care of Monday. Everything has worked fine for 26 years, until Jun 1. But things change, and I'm obviously behind the times given that I didn't have proper ROA and route object in place. Mike Tindor On Sat, Jun 29, 2024 at 11:26 AM Scott Q. wrote: There are DISA folks lurking here. I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help. I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem. Scott On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote: Hi folks, I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me. The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides. I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that. As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails. us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721 Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer? Thanks for your time! Mike Tindor
Thanks again,Scott. I'll be patient! Mike Tindor On Sat, Jun 29, 2024 at 12:18 PM Scott Q. <qmail@top-consulting.net> wrote:
All that sounds very familiar, I'm 100% sure it's the same issue.
As I said, there are DISA folks here, they might reach out and give you further steps. They did in my case, you just have to be more patient / on the ball than I was...
Good luck!
On Saturday, 29/06/2024 at 11:44 Mike Tindor wrote:
Scott,
Thanks for responding. Unfortunately, I think my situation is a little more dire, or at least involved. I probably should have said this before, but I had done TCP 25 outbound testing from our /23 to various .MIL MX's that I know were responding and could not establish a connection / get an SMTP banner. I could then go to Azure, or Digital Ocean, or somewhere else that I have a box and am able to make the outbound connection to the same MIL MXs that wouldn't respond to me from our /23.
So it isn't a simple case of DNS not resolving, although we certainly did notice that issue. Fortunately, we do have nameservers in place that are external to our /23 and which are able to actually do the resolving. But your comment does remind that this definitely is not just a TCP 25 issue, as the MIL DNS servers are not responding to queries from our /23 hosts.
The situation is difficult for multiple reasons:
1. inabiity to engage somebody from the other end - DISA 2. Unwillingness on my part to stab at a hornets nest and poke around trying to verify connections (other than TCP 25 to known MIL MXs) in DOD-land. 3. Not knowing exactly where to go from here
The latest/last thing DISA told me was that I would have to get one of the people with MIL email addresses who can't email our customers to actually open a ticket with DISA. And that is fraught with problems since even if a MIL email user did open a ticket, they would not have any information about our network to convey to the Helpdesk -- and would have no way of answering questions that the Helpdesk asked, and also wouldn't be able to do any troubleshooting.
I did realize a few days ago we had no ROA for the specific /23, and so I created one at ARIN. And we had no specific route object published for our /23, and I got one added. Been trying to clean up some old (and invalid) stuff that is in RADB from our larger /19, since we don't even own all the space in the /19 anymore and are only actively using a /23 from what we have left. Hoping to get that taken care of Monday.
Everything has worked fine for 26 years, until Jun 1. But things change, and I'm obviously behind the times given that I didn't have proper ROA and route object in place.
Mike Tindor
On Sat, Jun 29, 2024 at 11:26 AM Scott Q. <qmail@top-consulting.net> wrote:
There are DISA folks lurking here.
I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help.
I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem.
Scott
On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote:
Hi folks,
I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me.
The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides.
I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that.
As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails.
us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721
Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer?
Thanks for your time!
Mike Tindor
Thanks. That makes a little more sense to me. I know the questions DISA asked me when I called them, and I couldn't imagine just having the MIL-side email correspondent open a ticket directly with DISA. They would likely be more overwhelmed than I was. I'll talk to a couple of my customers who do biz with DOD on Monday and will ask them to reach out to their MIL contacts and request that the MIL contacts open a ticket with their IT. Since this has been going on now, some of my customers have switched temporarily to using Gmail/Yahoo just to stay in touch with their MIL contacts. So I know they can get the message through. Mike On Sat, Jun 29, 2024 at 12:55 PM Mike Tindor <mtindor@gmail.com> wrote:
Thanks again,Scott. I'll be patient!
Mike Tindor
On Sat, Jun 29, 2024 at 12:18 PM Scott Q. <qmail@top-consulting.net> wrote:
All that sounds very familiar, I'm 100% sure it's the same issue.
As I said, there are DISA folks here, they might reach out and give you further steps. They did in my case, you just have to be more patient / on the ball than I was...
Good luck!
On Saturday, 29/06/2024 at 11:44 Mike Tindor wrote:
Scott,
Thanks for responding. Unfortunately, I think my situation is a little more dire, or at least involved. I probably should have said this before, but I had done TCP 25 outbound testing from our /23 to various .MIL MX's that I know were responding and could not establish a connection / get an SMTP banner. I could then go to Azure, or Digital Ocean, or somewhere else that I have a box and am able to make the outbound connection to the same MIL MXs that wouldn't respond to me from our /23.
So it isn't a simple case of DNS not resolving, although we certainly did notice that issue. Fortunately, we do have nameservers in place that are external to our /23 and which are able to actually do the resolving. But your comment does remind that this definitely is not just a TCP 25 issue, as the MIL DNS servers are not responding to queries from our /23 hosts.
The situation is difficult for multiple reasons:
1. inabiity to engage somebody from the other end - DISA 2. Unwillingness on my part to stab at a hornets nest and poke around trying to verify connections (other than TCP 25 to known MIL MXs) in DOD-land. 3. Not knowing exactly where to go from here
The latest/last thing DISA told me was that I would have to get one of the people with MIL email addresses who can't email our customers to actually open a ticket with DISA. And that is fraught with problems since even if a MIL email user did open a ticket, they would not have any information about our network to convey to the Helpdesk -- and would have no way of answering questions that the Helpdesk asked, and also wouldn't be able to do any troubleshooting.
I did realize a few days ago we had no ROA for the specific /23, and so I created one at ARIN. And we had no specific route object published for our /23, and I got one added. Been trying to clean up some old (and invalid) stuff that is in RADB from our larger /19, since we don't even own all the space in the /19 anymore and are only actively using a /23 from what we have left. Hoping to get that taken care of Monday.
Everything has worked fine for 26 years, until Jun 1. But things change, and I'm obviously behind the times given that I didn't have proper ROA and route object in place.
Mike Tindor
On Sat, Jun 29, 2024 at 11:26 AM Scott Q. <qmail@top-consulting.net> wrote:
There are DISA folks lurking here.
I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help.
I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem.
Scott
On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote:
Hi folks,
I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me.
The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides.
I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that.
As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails.
us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721
Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer?
Thanks for your time!
Mike Tindor
The people at DISA you were dealing w/ aren't a Tier I service desk, they're the service desk that lower service desks open tickets w/. Think of DISA as a Tier I ISP and the normal .mil user as a residential user. See if one of your customers can put you in contact w/ their IT people (Usually a S6/G6/N6/A6/J6). 6 means IT or communications, the letter prefix is determined by the Military branch and level of the unit (A is Air Force, N is Navy, J is Joint, G is Army General Staff, and S is Army Staff). On Sat, Jun 29, 2024 at 6:25 PM Mike Tindor - mtindor at gmail.com < mtindor_at_gmail_com_rgphbc@simplelogin.co> wrote:
Thanks. That makes a little more sense to me. I know the questions DISA asked me when I called them, and I couldn't imagine just having the MIL-side email correspondent open a ticket directly with DISA. They would likely be more overwhelmed than I was. I'll talk to a couple of my customers who do biz with DOD on Monday and will ask them to reach out to their MIL contacts and request that the MIL contacts open a ticket with their IT.
Since this has been going on now, some of my customers have switched temporarily to using Gmail/Yahoo just to stay in touch with their MIL contacts. So I know they can get the message through.
Mike
On Sat, Jun 29, 2024 at 12:55 PM Mike Tindor <mtindor@gmail.com> wrote:
Thanks again,Scott. I'll be patient!
Mike Tindor
On Sat, Jun 29, 2024 at 12:18 PM Scott Q. <qmail@top-consulting.net> wrote:
All that sounds very familiar, I'm 100% sure it's the same issue.
As I said, there are DISA folks here, they might reach out and give you further steps. They did in my case, you just have to be more patient / on the ball than I was...
Good luck!
On Saturday, 29/06/2024 at 11:44 Mike Tindor wrote:
Scott,
Thanks for responding. Unfortunately, I think my situation is a little more dire, or at least involved. I probably should have said this before, but I had done TCP 25 outbound testing from our /23 to various .MIL MX's that I know were responding and could not establish a connection / get an SMTP banner. I could then go to Azure, or Digital Ocean, or somewhere else that I have a box and am able to make the outbound connection to the same MIL MXs that wouldn't respond to me from our /23.
So it isn't a simple case of DNS not resolving, although we certainly did notice that issue. Fortunately, we do have nameservers in place that are external to our /23 and which are able to actually do the resolving. But your comment does remind that this definitely is not just a TCP 25 issue, as the MIL DNS servers are not responding to queries from our /23 hosts.
The situation is difficult for multiple reasons:
1. inabiity to engage somebody from the other end - DISA 2. Unwillingness on my part to stab at a hornets nest and poke around trying to verify connections (other than TCP 25 to known MIL MXs) in DOD-land. 3. Not knowing exactly where to go from here
The latest/last thing DISA told me was that I would have to get one of the people with MIL email addresses who can't email our customers to actually open a ticket with DISA. And that is fraught with problems since even if a MIL email user did open a ticket, they would not have any information about our network to convey to the Helpdesk -- and would have no way of answering questions that the Helpdesk asked, and also wouldn't be able to do any troubleshooting.
I did realize a few days ago we had no ROA for the specific /23, and so I created one at ARIN. And we had no specific route object published for our /23, and I got one added. Been trying to clean up some old (and invalid) stuff that is in RADB from our larger /19, since we don't even own all the space in the /19 anymore and are only actively using a /23 from what we have left. Hoping to get that taken care of Monday.
Everything has worked fine for 26 years, until Jun 1. But things change, and I'm obviously behind the times given that I didn't have proper ROA and route object in place.
Mike Tindor
On Sat, Jun 29, 2024 at 11:26 AM Scott Q. <qmail@top-consulting.net> wrote:
There are DISA folks lurking here.
I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help.
I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem.
Scott
On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote:
Hi folks,
I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me.
The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides.
I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that.
As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails.
us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721
Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer?
Thanks for your time!
Mike Tindor
Thank you. That is helpful. Mike On Sun, Jun 30, 2024 at 12:31 AM <nanog.org@junk-mail.us> wrote:
The people at DISA you were dealing w/ aren't a Tier I service desk, they're the service desk that lower service desks open tickets w/.
Think of DISA as a Tier I ISP and the normal .mil user as a residential user.
See if one of your customers can put you in contact w/ their IT people (Usually a S6/G6/N6/A6/J6). 6 means IT or communications, the letter prefix is determined by the Military branch and level of the unit (A is Air Force, N is Navy, J is Joint, G is Army General Staff, and S is Army Staff).
On Sat, Jun 29, 2024 at 6:25 PM Mike Tindor - mtindor at gmail.com < mtindor_at_gmail_com_rgphbc@simplelogin.co> wrote:
Thanks. That makes a little more sense to me. I know the questions DISA asked me when I called them, and I couldn't imagine just having the MIL-side email correspondent open a ticket directly with DISA. They would likely be more overwhelmed than I was. I'll talk to a couple of my customers who do biz with DOD on Monday and will ask them to reach out to their MIL contacts and request that the MIL contacts open a ticket with their IT.
Since this has been going on now, some of my customers have switched temporarily to using Gmail/Yahoo just to stay in touch with their MIL contacts. So I know they can get the message through.
Mike
On Sat, Jun 29, 2024 at 12:55 PM Mike Tindor <mtindor@gmail.com> wrote:
Thanks again,Scott. I'll be patient!
Mike Tindor
On Sat, Jun 29, 2024 at 12:18 PM Scott Q. <qmail@top-consulting.net> wrote:
All that sounds very familiar, I'm 100% sure it's the same issue.
As I said, there are DISA folks here, they might reach out and give you further steps. They did in my case, you just have to be more patient / on the ball than I was...
Good luck!
On Saturday, 29/06/2024 at 11:44 Mike Tindor wrote:
Scott,
Thanks for responding. Unfortunately, I think my situation is a little more dire, or at least involved. I probably should have said this before, but I had done TCP 25 outbound testing from our /23 to various .MIL MX's that I know were responding and could not establish a connection / get an SMTP banner. I could then go to Azure, or Digital Ocean, or somewhere else that I have a box and am able to make the outbound connection to the same MIL MXs that wouldn't respond to me from our /23.
So it isn't a simple case of DNS not resolving, although we certainly did notice that issue. Fortunately, we do have nameservers in place that are external to our /23 and which are able to actually do the resolving. But your comment does remind that this definitely is not just a TCP 25 issue, as the MIL DNS servers are not responding to queries from our /23 hosts.
The situation is difficult for multiple reasons:
1. inabiity to engage somebody from the other end - DISA 2. Unwillingness on my part to stab at a hornets nest and poke around trying to verify connections (other than TCP 25 to known MIL MXs) in DOD-land. 3. Not knowing exactly where to go from here
The latest/last thing DISA told me was that I would have to get one of the people with MIL email addresses who can't email our customers to actually open a ticket with DISA. And that is fraught with problems since even if a MIL email user did open a ticket, they would not have any information about our network to convey to the Helpdesk -- and would have no way of answering questions that the Helpdesk asked, and also wouldn't be able to do any troubleshooting.
I did realize a few days ago we had no ROA for the specific /23, and so I created one at ARIN. And we had no specific route object published for our /23, and I got one added. Been trying to clean up some old (and invalid) stuff that is in RADB from our larger /19, since we don't even own all the space in the /19 anymore and are only actively using a /23 from what we have left. Hoping to get that taken care of Monday.
Everything has worked fine for 26 years, until Jun 1. But things change, and I'm obviously behind the times given that I didn't have proper ROA and route object in place.
Mike Tindor
On Sat, Jun 29, 2024 at 11:26 AM Scott Q. <qmail@top-consulting.net> wrote:
There are DISA folks lurking here.
I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help.
I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem.
Scott
On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote:
Hi folks,
I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me.
The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides.
I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that.
As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails.
us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721
Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer?
Thanks for your time!
Mike Tindor
One option is you said you have companies that do business with the DOD. Have one of them reach out to a contracting person about opening a ticket. That also allows there to be a claim that unblocking is mission essential vs someone trying to communicate with family. Normally the contracting people would reach out to their 6-shop (IT support) and have them open the ticket with DISA (or they might open a ticket with their base or military branch service desk who will reach out to DISA). On Sat, Jun 29, 2024, 12:19 Scott Q. - qmail at top-consulting.net < qmail_at_top-consulting_net_yioenfs@simplelogin.co> wrote:
All that sounds very familiar, I'm 100% sure it's the same issue.
As I said, there are DISA folks here, they might reach out and give you further steps. They did in my case, you just have to be more patient / on the ball than I was...
Good luck!
On Saturday, 29/06/2024 at 11:44 Mike Tindor wrote:
Scott,
Thanks for responding. Unfortunately, I think my situation is a little more dire, or at least involved. I probably should have said this before, but I had done TCP 25 outbound testing from our /23 to various .MIL MX's that I know were responding and could not establish a connection / get an SMTP banner. I could then go to Azure, or Digital Ocean, or somewhere else that I have a box and am able to make the outbound connection to the same MIL MXs that wouldn't respond to me from our /23.
So it isn't a simple case of DNS not resolving, although we certainly did notice that issue. Fortunately, we do have nameservers in place that are external to our /23 and which are able to actually do the resolving. But your comment does remind that this definitely is not just a TCP 25 issue, as the MIL DNS servers are not responding to queries from our /23 hosts.
The situation is difficult for multiple reasons:
1. inabiity to engage somebody from the other end - DISA 2. Unwillingness on my part to stab at a hornets nest and poke around trying to verify connections (other than TCP 25 to known MIL MXs) in DOD-land. 3. Not knowing exactly where to go from here
The latest/last thing DISA told me was that I would have to get one of the people with MIL email addresses who can't email our customers to actually open a ticket with DISA. And that is fraught with problems since even if a MIL email user did open a ticket, they would not have any information about our network to convey to the Helpdesk -- and would have no way of answering questions that the Helpdesk asked, and also wouldn't be able to do any troubleshooting.
I did realize a few days ago we had no ROA for the specific /23, and so I created one at ARIN. And we had no specific route object published for our /23, and I got one added. Been trying to clean up some old (and invalid) stuff that is in RADB from our larger /19, since we don't even own all the space in the /19 anymore and are only actively using a /23 from what we have left. Hoping to get that taken care of Monday.
Everything has worked fine for 26 years, until Jun 1. But things change, and I'm obviously behind the times given that I didn't have proper ROA and route object in place.
Mike Tindor
On Sat, Jun 29, 2024 at 11:26 AM Scott Q. <qmail@top-consulting.net> wrote:
There are DISA folks lurking here.
I had a similar issue where our block was labeled as residential by their new firewall, and DISA front-desk isn't yet trained on this mechanism so they can't help.
I escalated the issue to a lot of groups but in the end I gave up, too much bureaucracy. The issue is simply DNS - their DNS servers don't let you resolve. So I simply set 8.8.8.8 as the resolver for *.mil and it temp (permanently) fixed the problem.
Scott
On Saturday, 29/06/2024 at 09:16 Mike Tindor wrote:
Hi folks,
I'm looking for a DISA/DOD contact who feels that my issue has merit. I've tried the DISA Helpdesk and have been told since I'm a commercial entity with no affiliation with the DOD, they can't help me.
The issue at hand is that our /23 netblock has lost communication (at least email TCP 25) with AS345 / AS721 as of May 31, 2024 and I cannot figure out why. We are in a Flexential datacenter in Richmond VA and use Flexential for transport. We cannot send emails to .MIL or receive emails from .MIL. It is not that they are being rejected on either end. The deliveries are timing out and being returned to sender, from both sides.
I don't know if DISA/DOD has a block on our ASN and-or /23, or if there is a routing issue somewhere between us and AS345 / AS721. I had asked the Flexential folks to look into it from their side, and they indicated that historic data does indeed show that there TCP 25 communications to and fro between us and AS345 prior to June 1, but nothing from June 1 onward. And all they could say was that they (Flex) were not in any way blocking. And I'd agree with that.
As you can imagine, my customers are not happy with not being able to communicate with their family / friends via email in the MIL domains, and our customers who are vendors / contractors cannot do business with the military effectively if they cannot send/receive emails.
us --> Flexential --> GTT --> Level3 --> Qwest --> ? --> AS345 / AS721
Any idea where one would go next? Is it likely that any of those entities further upstream like GTT / Level3 / Qwest would even assist since we are not their customer?
Thanks for your time!
Mike Tindor
participants (3)
-
Mike Tindor -
nanog.org＠junk-mail.us -
Scott Q.