Minimal reproducer for clash when binding types defined in the unnamed namespace.

[rwgk]

Description

This PR

• works as expected only with gcc, but
• fails with any clang in otherwise identical environments, and also
• fails with any MSVC.

The root cause for the failures is:

• here for clang except macOS

  pybind11/include/pybind11/detail/internals.h

  Line 120 in ee2b522

  using type_equal_to = std::equal_to<std::type_index>;

• here for MSVC and clang macOS

  pybind11/include/pybind11/detail/internals.h

  Line 139 in ee2b522

  return lhs.name() == rhs.name() || std::strcmp(lhs.name(), rhs.name()) == 0;

  )

Full details of which platform is or is not using the __GLIBCXX__ branch: #4313 (comment)

Unknown: How could this be fixed?

@wjakob Are you aware of this situation already? — Reason for bringing this up now: If the internals version is bumped, this would be a good fix to include (assuming the fix might be an ABI break, which isn't clear until we have a fix).

@charlesbeattie FYI It turns out the root cause is not what we thought it is when we looked at this back in February.

Suggested changelog entry:

[rwgk]

Note re CI failures: 1 flake: CI / 🐍 3.10 • ubuntu-latest • x64

[rwgk]

@henryiii @Skylion007 @lalaland for awareness / comment.

[rwgk]

For easy reference, the CI results for the current "Debug printf()s" commit are here (everything failed, as expected):

https://github.com/pybind/pybind11/actions/runs/3393602034/jobs/5641129754

The name of the "Download log archive" file: logs_24742.zip

Results of inspecting the logs:

$ grep 'LOOOK __GLIBCXX__' *.txt | sed 's/^[1-9][0-9]*//' | cutniq | sort
______3.10_____CUDA_11.7_____Ubuntu_22.04.txt
______3.10_____ubuntu-latest_____x64.txt
______3.11__deadsnakes______x64.txt
______3.11_____ubuntu-latest_____x64.txt
______3.6_____ubuntu-latest_____x64_-DPYBIND11_FINDPYTHON=ON_-DCM.txt
______3.7_____Debian_____x86______Install.txt
______3.9-dbg__deadsnakes______Valgrind_____x64.txt
______3.9_____ubuntu-latest_____x64.txt
______3_____almalinux8_____x64.txt
______3_____almalinux9_____x64.txt
______3_____CentOS7__PGI_22.9_____x64.txt
______3_____centos7_____x64.txt
______3_____Clang_10_____C++17_____x64.txt
______3_____Clang_10_____C++20_____x64.txt
______3_____Clang_11_____C++20_____x64.txt
______3_____Clang_12_____C++20_____x64.txt
______3_____Clang_13_____C++20_____x64.txt
______3_____Clang_14_____C++20_____x64.txt
______3_____Clang_3.6_____C++11_____x64.txt
______3_____Clang_3.7_____C++11_____x64.txt
______3_____Clang_3.9_____C++11_____x64.txt
______3_____Clang_5_____C++14_____x64.txt
______3_____Clang_7_____C++11_____x64.txt
______3_____Clang_9_____C++11_____x64.txt
______3_____Clang_dev_____C++11_____x64.txt
______3_____GCC_10_____C++17____x64.txt
______3_____GCC_11_____C++20____x64.txt
______3_____GCC_12_____C++20____x64.txt
______3_____GCC_7_____C++11____x64.txt
______3_____GCC_7_____C++17____x64.txt
______3_____GCC_8_____C++14____x64.txt
______3_____GCC_8_____C++17____x64.txt
______3_____ICC_latest_____x64.txt
______3_____windows-latest_____mingw32.txt
______3_____windows-latest_____mingw64.txt
______pypy-3.7_____ubuntu-latest_____x64.txt
______pypy-3.8_____ubuntu-latest_____x64_-DPYBIND11_FINDPYTHON=ON.txt
______pypy-3.9_____ubuntu-latest_____x64.txt

$ grep 'LOOOK ___OTHER___' *.txt | sed 's/^[1-9][0-9]*//' | cutniq | sort
______3.10_____macos-latest_____x64.txt
______3.10_____windows-2022_____x64.txt
______3.11_____macos-latest_____x64.txt
______3.11_____windows-2022_____x64.txt
______3.6_____macos-latest_____x64.txt
______3.6_____MSVC_2019_____x86.txt
______3.6_____windows-2019_____x64_-DPYBIND11_FINDPYTHON=ON.txt
______3.6_____windows-2022_____x64.txt
______3.7_____MSVC_2019_____x86_-DCMAKE_CXX_STANDARD=14.txt
______3.8_____MSVC_2019__Debug______x86_-DCMAKE_CXX_STANDARD=17.txt
______3.8_____MSVC_2019_____x86_-DCMAKE_CXX_STANDARD=17.txt
______3.9_____macos-latest_____x64.txt
______3.9_____MSVC_2019__Debug______x86_-DCMAKE_CXX_STANDARD=20.txt
______3.9_____MSVC_2019_____x86_-DCMAKE_CXX_STANDARD=20.txt
______3.9_____MSVC_2022_C++20_____x64.txt
______3.9_____windows-2019_____x64.txt
______3.9_____windows-2022_____x64.txt
______pypy-3.7_____macos-latest_____x64.txt
______pypy-3.7_____windows-2022_____x64.txt
______pypy-3.8_____macos-latest_____x64.txt
______pypy-3.8_____windows-2022_____x64.txt
______pypy-3.9_____macos-latest_____x64.txt
______pypy-3.9_____windows-2022_____x64.txt

Minor detail, the expected failures are (e.g.):

2022-11-04T12:05:02.8744572Z FAILED ../../tests/test_iostream.py::test_flush - AssertionError: assert '\nL...
2022-11-04T12:05:02.8744990Z FAILED ../../tests/test_iostream.py::test_redirect - AssertionError: assert '...
2022-11-04T12:05:02.8745293Z FAILED ../../tests/test_iostream.py::test_redirect_err - AssertionError: asse...
2022-11-04T12:05:02.8745615Z FAILED ../../tests/test_iostream.py::test_redirect_both - AssertionError: ass...

[EthanSteinberg]

Just to make sure I am understanding this correctly, the problem is that std::type_info is not unique in Clang?

[rwgk]

Just to make sure I am understanding this correctly, the problem is that std::type_info is not unique in Clang?

Yes, I believe so. Initially I wasn't sure at all, but I asked internally and it looks like this is an extremely long-lived clang bug. I'm still not completely sure, but the evidence is pretty obvious.

I found this with a Google search and I'm wondering if that's a fix: https://reviews.llvm.org/D122775

I tried clang-15 in a container that was apparently built in July, but that still had the problem. Again, I'm not sure yet what the situation is. Maybe I'll ask around more next week.

We also have a problem with the ! __GLIBCXX__ branch that's being used under Windows and macOS. I don't know if that can be fixed: how can we get unique names/ids/identifiers of some sort?

The easiest way out maybe: clearly document the issue. It's easy to work around when you know.

[EthanSteinberg]

The easiest way out maybe: clearly document the issue. It's easy to work around when you know.

Can we detect classes / structs that belong to unnamed scopes and throw an error message when users try to register one in Clang?

If we can just throw an appropriate error message, we don't even need to document this as any user could just see the error message.

[rwgk]

Can we detect classes / structs that belong to unnamed scopes and throw an error message when users try to register one in Clang?

If we can just throw an appropriate error message, we don't even need to document this as any user could just see the error message.

Interesting question.

Note that we're generating an error message already, it's just not immediately obvious why, e.g.:

ImportError: generic_type: type "unnamed_namespace_b_any_struct" is already registered!

We could look for these substrings in the mangled names:

clang (and gcc):

N12_GLOBAL__N_1

Windows:

`anonymous namespace'

Then add a hint to that error message, something like "Note that this type is in an unnamed namespace, which may not be handled correctly on some platforms. See for more information."

But inspecting the mangled names is a pretty ugly heuristic. I'm thinking, if we cannot find workarounds/fixes for all platforms, it's better to add a small section to the docs explaining the likely error case (class_ in two extensions) and the limitations for unnamed namespaces, and point to that unconditionally from the error message.

FWIW, while looking around just now I found this:

• https://www.boost.org/doc/libs/1_66_0/doc/html/boost_typeindex/examples.html

boost seems to have very comprehensive tools for working with mangled names. But again, I'm not sure it's worth the effort including long-term maintenance to fold this into the code for the pybind11 type registry.

[rwgk]

For easy reference, link to comment under [Experiment] boost::typeindex::type_index & unnamed namespace:

• [Experiment] boost::typeindex::type_index & unnamed namespace #4315 (comment)

Takeaway: I don't think there is much hope that unnamed namespace clashes can be fixed.

[rwgk]

EDIT: At second thought: this only proves that types in the unnamed namespace in different translation units do not compare equal, as desired. But do types in named namespaces compare equal, as desired?

---

Original message:
Good news, I think:

• [Experiment] std::type_index & un/named namespace #4316 (comment)

I think we can make internals.h simpler & more correct, by removing the !__GLIBCXX__ code path entirely. I'll try this later in another PR.

[rwgk]

PR #4319 first observation: only macOS fails, and only test_local_bindings.py FFF.FF...F

[rwgk]

The unnamed namespace clang bug is still not fixed even in the latest nightly builds for clang-15 and clang-16, using PR #4316 for testing:

========================================================= short test summary info ==========================================================
SKIPPED [1] test_callbacks.py:203: Current PYBIND11_INTERNALS_VERSION too low
SKIPPED [1] test_exc_named_namespace_a.py:10: std::type_index-EQ-GOOD
SKIPPED [1] test_stl.py:143: no <experimental/optional>
SKIPPED [1] test_stl.py:175: no <boost/optional>
SKIPPED [1] test_unnamed_namespace_a.py:11: std::type_index-EQ-BAD
===================================================== 789 passed, 5 skipped in 11.21s ======================================================

root@53f2abca540e:/build_clang16# clang++-15 --version
Debian clang version 15.0.4-++20221102053030+5c68a1cb1231-1~exp1~20221102053120.92
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

root@53f2abca540e:/build_clang16# clang++-16 --version
Debian clang version 16.0.0-++20221106071805+b5626ae9751f-1~exp1~20221106071907.446
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

root@53f2abca540e:/build_clang16# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"