ipmi access

so how to folk protect yet access ipmi? it is pretty vulnerable, so 99% of the time i want it blocked off. but that other 1%, i want kvm console, remote media, and dim sum. currently, i just block the ip address chunk into which i put ipmi at the border of the rack. when i want access, i reconfig the acl. bit of a pita. anyone care to share better idea(s)? thanks. randy