I already had this idea, I even implemented it in the desperate time of the 512K "bug".
And with that I can tell you:
Do not do it! You will be bothered!

But if you want to go this way, what I can recommend is to try not to put routes in the FIB that match your Default.

Talking about having a default route other than /dev/null is already a problem at first...
Because a Transit Provider is not expected to use the Default route.
But I'm not even going to get into that (many flames will arise).

If you really decide to use a default route and choose what will not and what will not apply to the FIB, you must be prepared for a certain complexity in these choices.
And the more Peers and DFZ views you have, the more complex it will be.

In a very simplified hypothetical example of dual-homed DFZ, take the best routes from link B, and leave the default by link A.

There are even tools that, comparing flow analysis and routes exported from the RIB, "choose" the routes with more matching packages, and apply that for you.
But thinking in this way, the transition to the dark side of the force is already beginning to be made. Walking through the valley of death until arriving in the land of the Route Optimizers.

My memory is not helping me...
But I think the name of one of the projects that did this magic was rt-flow or flow-rt. Something like.