device handle hotfixes for 2.7

[mbolivar-nordic]

Fixes: #38181

When CONFIG_USERSPACE is enabled, the ELF file from linker pass 1 is
used to create a hash table that identifies kernel objects by address.
We therefore can't allow the size of any object in the pass 2 ELF to
change in a way that would change those addresses, or we would create
a garbage hash table.

Simultaneously (and regardless of CONFIG_USERSPACE's value),
gen_handles.py must transform arrays of handles from their pass 1
values to their pass 2 values; see the file's docstring for more
details on that transformation.

However, it is possible that a device has more direct dependencies in
the pass 2 handles array than its corresponding devicetree node had in
the pass 1 array. When this happens and gen_handles.py has no room to fill
the end of the array with a final DEVICE_HANDLE_ENDS,
the build fails and leaves users with no recourse to fix it. That's a
formerly theoretical showstopper, which became a real life showstopper when
the "supported" devices patchset landed in #37836.

To work around this for now:

• revert the 'supports' patches; which seem to have other problems anyway (see item 4. in device: insert padding to ordinal array #38487 (review))
• although the reverts make the problem go away for current tests, add a new config option, CONFIG_DEVICE_HANDLE_PADDING, that users can use to hack around this limitation in case they run into the problem elsewhere

We'll chase a more invasive but better fix after 2.7 is released and try to backport it if we can.

[tejlmand]

The fact that users are informed during build (linking stage) on what to do / workaround the issue when this error occurs is very good.

Accepted as a short term solution.

[dcpleung]

LGTM as a workaround.

[mbolivar-nordic]

Build errors seem to be related to an inability to "nest" UTIL_LISTIFY calls within each other, which happens in some wrappers that allocate devices using UTIL_LISTIFY. Still investigating.

[mbolivar-nordic]

inability to "nest" UTIL_LISTIFY calls within each other

Yep, that was it. I pushed a new version that avoids UTIL_LISTIFY.

We have code in the tree that's doing a UTIL_LISTIFY loop over DEVICE_DEFINE. We therefore can't use UTIL_LISTIFY within DEVICE_DEFINE as I did in my initial attempt. Harbison & Steele explains why:

Macros appearing in their own expansion---either immediately or through some intermediate sequence of nested macro expansions---are not reexpanded in standard C [...] Older C preprocessors do not detect this recursion, and will attempt to continue the expansion until they are stopped by some system error.

E.g. this:

#define BAR(i, j, ...) int bar_##i_##j;
#define FOO(j, ...) UTIL_LISTIFY(2, BAR, j)
UTIL_LISTIFY(3, FOO)

expands to this:

UTIL_LISTIFY(2, BAR, 0) UTIL_LISTIFY(2, BAR, 1) UTIL_LISTIFY(2, BAR, 2)

and stops there.

[ceolin]

@mbolivar-nordic Sorry for the late review, isn't e7f2b73 enough to live with the "supported" devices API ? What are the other problems in this API ? I saw problems with the fix proposed, but if adding paddings is a viable workaround why not keep that API ?

[cfriedt]

Is the undefined Kconfig a false positive?

[mbolivar-nordic]

Is the undefined Kconfig a false positive?

Yes

[mbolivar-nordic]

isn't e7f2b73 enough to live with the "supported" devices API ?

No because of point 4. in #38487 (review)

[cfriedt]

@mbolivar-nordic - please rebase and fix conflicts

[mbolivar-nordic]

Rebased. I removed the comment causing the kconfig false positive also, just to pacify the linter.