It's not like IPsec protocols (it's a suite of protocols and concepts, not one) are proprietary or something. There are pretty ASCII pictures in RFCs with all about how the packets are put together. See section 3 of RFC 4303 to see how ESP transport and tunnel mode datagrams are put together.

For the tl;dr, in transport mode everything above IP header is the payload. In tunnel mode, the whole IP datagram is the payload. The contents of the payload are specified by the "Next Header" field of the ESP header. For an encapsulated IPv4 packet, it would be protocol 4 (IP-in-IP). For an IPv6 packet, it would be 41. For TCP in transport mode, it would be 6. UDP is 17. Etc.

If you want to see it in action yourself, you can set the encryption algo to NULL and do a capture.

On Tue, Feb 15, 2022 at 10:16 AM Grant Taylor via NANOG <nanog@nanog.org> wrote:

Hi Bill,

On 2/12/22 8:55 PM, William Herrin wrote:
> It's tunnel mode plus a tunneling protocol plus some implicit routing
> and firewalling which gets in the way of dynamic routing.

I assume you meant to say that it's /transport/ mode plus a tunneling
protocol.

I wonder if you are thinking more of an IPSec VPN management suite of
sorts, e.g. wizard / helper that is included in some devices. I'm
thinking at a very low (manual) level. The "implicit routing" and
"firewalling" are the strongest indicators of this to me. The manual
IPSec that I've done on Linux (via the `ip xfrm` command) doesn't touch
firewalling and I believe that addresses inside the tunnel would be
completely separate operations / commands.

> Try it if you don't believe me. Set up tunnel mode ipsec manually on
> two nodes (no IKE) and get them talking to each other. Then change
> one to transport mode and add I think it's an IPIP tunnel but I don't
> remember for certain. And add the appropriate routes into the tunnel
> virtual device. You'll find they talk.

Unfortunately I don't have the leisure time to do this experimentation
currently. As such I'm going to put this on my to-do pile for future
investigation ~> follow up.

I do not recall reading about IPSec /Tunnel/ mode re-using an existing
tunneling protocol; IPIP, etc. Perhaps I'm misremembering. Perhaps it
inherently does so without declaring as such.

> What did you think IPSec was doing? Transport mode encrypts the layer
> 4 and up of the packet between two machines; it doesn't encapsulate
> it. When they added tunnel mode, the inner layer 3 had to go somewhere.

My understanding is that /Transport/ mode applies AH (no encryption) and
/ or ESP (encryption) to L4 datagrams and that /Tunnel/ mode does the
same to L3 packets.

P.S. I'm sending this reply to NANOG in case anyone else has any
contribution / comments. I suspect any future reply will be directly to
Bill as this is getting further off topic, both for NANOG in general and
for this VPN recommendations thread.

--
Grant. . . .
unix || die