Well, you could look at the fact that it's not turned on by default by GCC itself with any -O level, that it is inclusive of the "-funsafe-math-optimizations" setting (which name the GCC team picked deliberately), or that it is listed in the documentation of GCC itself to cause accuracy problems.
Then again there are various indications around that it can cause failures in both operation and compilation.
It's not like fast-math is "broken", but it causes a completely different math lib than default to be used, and one that is known (by design) to not be as accurate as the default float path. On Intel architectures where the floating point units are so hyper-optimized it would seem of questionable value to sacrifice potential compatibility for the feature - on ARM or something like that, sure, since float is going to be dog slow by comparison.
If you really want to know the subset of flags that can make a set of code be all that it can be under gcc, I'd run it through Acovea, although that's pretty exhaustive and you have to provide repeatable invariant examples for it to crunch on.
-Frob