[DXIL] Null pointer exception due to double floating-point literals in fmod intrinsic

[LukasBanana]

Description
There is a null pointer exception during the translation of FMod (TranslateFMod) when the arguments are floating-point literals, because the latest DXC version does not reduce them to float during compilation. The null pointer occurs inside TrivialDxilUnaryOperation(OP::OpCode::Frc, absDiv, hlslOP, Builder) because Frc won't find the respective function for double-precision floats. Previous versions had this assertion in OP::GetOpFunc:

DXASSERT(IsOverloadLegal(opCode, pOverloadType), "otherwise the caller requested illegal operation overload (eg HLSL function with unsupported types for mapped intrinsic function)");

But the latest version only returns a null pointer that is not handled correctly:

if (!IsOverloadLegal(opCode, pOverloadType))
  return nullptr;

Steps to Reproduce
This issue can be reproduced with the latest version of the main branch and compiling with dxc -E PSMain -T ps_6_0 (or by following https://godbolt.org/z/beK7oPsx6):

float4 PSMain() : SV_Target {
    return (float4)fmod(1.0, 2.0);
}

Making the floating-point literals explicitly single-precision with the f suffix compiles fine:

float4 PSMain() : SV_Target {
    return (float4)fmod(1.0f, 2.0f);
}

This does not reproduce with other intrinsics such as sin.

[damyanp]

Possibly introduced in #6302

[LukasBanana]

@damyanp this definitely looks like it introduced the null pointer exception, since all those assertions were removed and the null pointer isn't handled. The fix @tex3d linked to this issue works for us, although it's not a long term fix as he mentioned it can still crash with explicit double variables.

[damyanp]

See #6419 - plan is to revert #6302 from the release branch and to continue to use this issue to track a more permament fix.