gcc: fix inf * NaN in m68k's soft-float double multiplication
__muldf3 in libgcc/config/m68k/lb1sf68.S returns infinity for inf * NaN, where
IEEE 754 asks for NaN. When a's high word equals 0x7ff00000 the code branches to
Lmuldf$a$nf, decides there whether a is a NaN, and if a turns out to be infinity
signals overflow -- without ever examining b. The comment two lines above says
"we still have to check d1 and b", but b never gets checked. The float path next
to it does it right: Lmulsf$inf exists precisely to look at b first.
Measured on the ColdFire toolchain, over all 64 combinations of {+0, -0, 1, -2.5,
1e300, +inf, -inf, NaN}: two are wrong without this patch, +inf * NaN and
-inf * NaN, and none with it. Only double multiplication is affected; float
multiplication, both divisions and subtraction already return NaN as they should.
Verified by assembling the patched file and linking it ahead of libgcc, so no
toolchain rebuild was needed to see it.
It surfaced through uClibc-ng's scalb, which propagates a NaN argument with
x * fn and therefore returned inf for scalb(inf, NaN). musl carries the same
line from the same fdlibm ancestor, so it is affected too.
Signed-off-by: Ramin Moussavi <ramin.moussavi@yacoub.de>
Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
ramin