Blah...forgot to send this to nanog as well...it seems there is a lack of understanding of why this works in some situations and not in others, so attempted to spread some knowledge here. Also Sprach Leo Bicknell

but I find it slightly (emphasis on the slightly) that someone would turn on PMTU discovery, and then filter it out right in front of the boxes where they turned it on.

As someone else mentioned...it happens all the time...disconnect between server and network admins probably, or something along those lines (or just general cluelessness)

Also, it seems to me most DSL users are behind PPPoE links with lower MTU, and should get hit by the same problem.

No. The trick, here, is that the PPPoE (typically) terminates on the same system that's terminating the TCP connection, so the PPPoE end system can see that the PMTU is going to be, at most, 1492, so it can use a lower Maximum Segment Size in TCP to start the whole scenario off at the 1492 MTU size and try to go down from there. You're seeing the problem because the tunnel is not terminated on the system that's also terminating the TCP connection, so the TCP processing can't know about the 14xx MTU somewhere out there except through PMTU (which is broken in this case), so it can't set the corresponding MSS to compensate for it initially.

The temporary hack is to have tunnelbox1 clear the DF bit on all incoming packets, which just causes the packets to get fragmented going down the tunnel. A minor performance hit, but it works.

An only slightly better hack would be to have the tunnel and/or firewall twiddle the MSS on outgoing TCP connections to compensate for the lower tunnel MTU. Still pretty gross, but won't have as much of an effect on the TCP performance.

Are the servers really that broken (PMTU enabled, ICMP Can't Fragement filtered)?

Yes. The last time I ran into this, my test site was www.harvard.edu (!)...though that's been a year or more ago, so they may have resolved their issues since then. We ran into plenty more sites that had the problem, but that's the one that sticks out in my mind because, like I said, it was the one that I used as a site to try to connect to as a test.

Does the head end box of DSL services generally do something to work around this (ie, clear the DF bit)? Am I just being an idiot and missing something obvious?

I wouldn't say idiot, or missing anything obvious...but you were missing the whole MSS issue. I've never thought the behavior was intuitive or obvious...but once you think about it becomes a "Why didn't I think of that?" sorta thing. -- Jeff McAdams "He who laughs last, thinks slowest." -- anonymous