My dad flew on the Concorde several times for business trips. He was actually on one of the flights that had to turn back to paris after an engine blew out (or something). Considering what happened to other flights, could have been a lot worse for my family...
Stuff breaks. Engine failures and "turning back" are not amazingly rare events. Like all complex machines, sometimes something fails, and it's best to fix it sooner rather than later. The engineers that design the airframes understand that things fail, and they design the system to function in a degraded state. But you don't really want to keep the thing in a degraded state for very long, so sometimes you try to land the plane as quickly as possible (even though there was almost no actual danger).
It's like when a drive fails in your RAID array. The hot spare takes care of the failure, and you aren't even in a degraded state. But you still order the replacement drive as soon as possible, because it feels bad to not have as many backup systems as you had before. Same with planes.
The engineers that design the airframes understand that things fail, and they design the system to function in a degraded state.
The linked site has some interesting info on the technical design of some of the flight systems. There are multiple redundancies on the hydraulics, for instance. Very interesting stuff.