If DOS is such a large concern, IPSEC to an extent can be used to mitigate against it. And IKEv1/v2 with IPSEC is not the horribly inefficient mechanism it is made out to be. In practice, it is quite easy to use.
IPSEC does nothing to protect a network device from a DOS attack. You know that. DOS prevention on a network device needs to happen before the TCP/Packet termination - not the Key/MD5/IPSEC stage. The signing or encrypting of the BGP message protects against Man in the Middle and replay attacks - not DOS attacks. Once a bad packet gets terminated, your DOS stress on the router kicks in (especially on ASIC/NP routers). The few extra CPU cycles it takes for walking through keys or IPSEC decrypt are irrelevant to the router's POV. You SOL if a miscreant can get a packet through your classification & queuing protections on the router and have it terminated. The key to DOS mitigation on a network device is to have many fields in the packet to classify as possible before the TCP/Packet termination. The more you have to classify on, the more granular you can construct your policy. This is one of the reasons for GTSM - which adds one more field (the IP packet's TTL) to the classification options. Yes Jared - our software does the TTL after the MD5, but the hardware implementations does the check in hardware before the packet gets punted to the receive path. That is exactly where you need to do the classification to minimize DOS on a router - as close to the point where the optical-electrical-airwaves convert to a IP packet as possible.