It depends upon how low a probability failure you're willing to consider and how paranoid you are. For one thing, the U.S. National Command Authority could decide that GPS represents a threat to national security and disable or derate GPS temporarily or indefinitely over a limited or unlimited area.
Derating GPS wouldn't affect the time reference functionality. Turning off GPS entirely would seriously affect military aviation operations.
It is well known that GPS is vulnerable to deliberate attacks in limited areas, perhaps even over large areas (see Presidential Decision Directive 63). Backup systems are officially recommended for "safety-critical applications" and the US government is actively intersted in developing low-cost backup systems (presumably because they're concerned about GPS as a SPOF too).
The US government, and other entities, do perform "GPS interference testing". This basically means they interfere with GPS. The government is also actively investigating "phase-over to private operation", which could mean changes to operation, fee system, or reliability of the GPS system.
One could also imagine conditions that would result in concurrent failures of large numbers of satellites. Remember what happened to Anik E-1 and E-2 (space weather caused them to spin out of control).
If you do develop a system with GPS as a SPOF, you should certainly be aware of these risks and monitor any changes to the political and technical climate surrounding GPS. I do believe that it is currently reasonable to have GPS as a SPOF for a timing application that is not life critical (that is, where people won't die if it fails).
Aviators try very, very hard not to trust their lives to GPS.
As opposed to LORAN ?