__functor causes infinite loop when it returns itself

[andir]

Describe the bug
As of 81e7c40 a functor that doesn't take two arguments but one leads to an inifite callFunction loop. Previously this lead to a stack exhaustion and did terminate eventually.

Example expression:

let a = { __functor = x: x; }; in a {}

This will cause the functor that resides within the attribute set a to return itself. Since __functor requires 2 (or more?) arguments, the language then tries to call the attribute set again and that results in the __functor function to be called again... This now loops until you manually terminate it.

The recent code changes seem to have encouraged the compiler to apply tail call optimization, and thus the loop never consumes the entire stack(?).

Steps To Reproduce

1. nix-instantiate --eval -E '({ __functor = args@{ ... }: args; }) {}'
2. Observe that it never terminates.
3. Attach gdb to the process and see that it loops in callFunction but never actually adds another stack frame.

Expected behavior

I'd expect the evaluation to realize that this is an infinite loop and terminate with an error.

Stack overflow on 2.3.16
Infinite loop on 81e7c40 (and newer).

Additional context

I encountered this while writing some additional libexpr unit tests for #5377 .