TL;DR I suspect there are middle boxes that don't like IPs ending in .255.
Anyone seen that?
Folks,
We are troubleshooting a strange issue where some of our customers cannot
establish a successful connection with our HTTP front end. In addition to
checking the usual things like routing and interface errors and security
policy configurations, hopening support tickets with the load balancer
vendor so far all to no avail, we did packet captures.
Based on the packet captures we receive a SYN, we reply with SYN-ACK, but
the client never actually receives that SYN-ACK. In a different instance
the 3-way completes, followed by TLS client hello to us, we reply with TLS
Server Hello and that server hello never makes it to the client.
And again, this is only affecting a small subset of customers thus
suggesting it's not the load balancer or the edge routing configuration (in
fact we can traceroute fine to the customer's IP).
So far the only remaining theory that remains is that there are middle
boxes out there that do not like IPs ending in .255. The service that the
clients can't get to is hosted on two IPs ending in .255
Let's just say they are x.x.121.255 and x.x.125.255. We even stood up a
basic "hello world" web server on x.x.124.255 with the same result.
Standing up the very same basic webserver on x.x.124.250 allows the client
to succeed.
So far we have a friendly customer who has been working with us on
troubleshooting the issue and we have some pcaps from the client's side
somewhat confirming that it's not the customer's system either.
This friendly customer is in a small 5 people office with Spectrum business
internet (that's the SYN-ACK case). The same customer tried hopping on his
LTE hotspot which came up as Cellco Partnership DBA Verizon Wireless with
the same result (that's the TLS server hello case). That same customer with
the same workstation drives a town over and he can get to the application
fine (we are still waiting for the customer to let us know what that source
IP is when it does work).
Before you suggest that those .255 addresses are broadcasts on some VLAN,
they are not. They are injected as /32s using a routing protocol, while the
VLAN addressing is all RFC1918 addressing.
--Andrey
