Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-91320: Add _Py_reinterpret_cast() macro #91959

Merged
merged 2 commits into from
Apr 27, 2022
Merged

gh-91320: Add _Py_reinterpret_cast() macro #91959

merged 2 commits into from
Apr 27, 2022

Conversation

vstinner
Copy link
Member

Fix C++ compiler warnings about "old-style cast"
(g++ -Wold-style-cast) in the Python C API. Use C++
reinterpret_cast<> and static_cast<> casts when the Python C API is
used in C++.

Example of fixed warning:

Include/object.h:107:43: error: use of old-style cast to
‘PyObject*’ {aka ‘struct _object*’} [-Werror=old-style-cast]
#define _PyObject_CAST(op) ((PyObject*)(op))

Add _Py_reinterpret_cast() and _Py_static_cast() macros.

Fix C++ compiler warnings about "old-style cast"
(g++ -Wold-style-cast) in the Python C API.  Use C++
reinterpret_cast<> and static_cast<> casts when the Python C API is
used in C++.

Example of fixed warning:

    Include/object.h:107:43: error: use of old-style cast to
    ‘PyObject*’ {aka ‘struct _object*’} [-Werror=old-style-cast]
    #define _PyObject_CAST(op) ((PyObject*)(op))

Add _Py_reinterpret_cast() and _Py_static_cast() macros.
@vstinner
Copy link
Member Author

cc @serge-sans-paille

@serge-sans-paille
Copy link
Contributor

conceptually looks good to me :-)

@vstinner
Copy link
Member Author

cc @erlend-aasland @corona10

@@ -50,7 +50,8 @@ PyAPI_FUNC(PyObject *) _PyObject_MakeTpCall(
PyObject *const *args, Py_ssize_t nargs,
PyObject *keywords);

#define PY_VECTORCALL_ARGUMENTS_OFFSET ((size_t)1 << (8 * sizeof(size_t) - 1))
#define PY_VECTORCALL_ARGUMENTS_OFFSET \
(_Py_static_cast(size_t, 1) << (8 * sizeof(size_t) - 1))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This macro should be updated since it's used in a static inline function: PyVectorcall_NARGS().

It seems like the warning is emitted if an old-style cast is done in a static inline function, but it seems to be fine if it's only done in macros.

Copy link
Member Author

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative is to don't add two macros, _Py_static_cast() and _Py_reinterpret_cast(), but only one _Py_CAST() macro which always uses reinterpret_cast<type>(expr) on C++.

@vstinner
Copy link
Member Author

vstinner commented Apr 26, 2022

An alternative is to don't add two macros, _Py_static_cast() and _Py_reinterpret_cast(), but only one _Py_CAST() macro which always uses reinterpret_cast(expr) on C++.

Sadly, size_t var = reinterpret_cast<size_t>(1); fails on C++, whereas size_t var = static_cast<int>(1); is fine. C++ requires to use right cast.

x.cpp:8:18: error: invalid cast from type 'int' to type 'size_t' {aka 'long unsigned int'}
    8 |     size_t var = reinterpret_cast<size_t>(1);
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~

I used static_cast<Py_ssize_t>(...) to fix C++ warnings in datatable: h2oai/datatable@753197c

Copy link
Member

@corona10 corona10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me.

@vstinner vstinner merged commit 29e2245 into python:main Apr 27, 2022
@vstinner vstinner deleted the cpp_cast branch April 27, 2022 08:41
@vstinner
Copy link
Member Author

Thanks for the reviews @serge-sans-paille and @corona10.

Copy link
Contributor

@erlend-aasland erlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense; looks good.

@tacaswell
Copy link
Contributor

This suspect that this does not appear to preserve const correctness, I am seeing errors like:

$ gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o

In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:38,
                 from src/greenlet/greenlet.cpp:14:
src/greenlet/greenlet_refs.hpp: In member function ‘void greenlet::refs::CreatedModule::PyAddObject(const char*, const PyObject*)’:
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: ‘reinterpret_cast’ from type ‘const PyObject*’ {aka ‘const _object*’} to type ‘PyObject*’ {aka ‘_object*’} casts away qualifiers
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:505:35: note: in expansion of macro ‘_PyObject_CAST’
  505 | #  define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op))
      |                                   ^~~~~~~~~~~~~~
src/greenlet/greenlet_refs.hpp:854:13: note: in expansion of macro ‘Py_INCREF’
  854 |             Py_INCREF(new_object);
      |             ^~~~~~~~~
src/greenlet/greenlet_refs.hpp: In member function ‘void greenlet::refs::PyErrPieces::normalize()’:
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: invalid cast from type ‘greenlet::refs::OwnedErrPiece’ to type ‘PyObject*’ {aka ‘_object*’}
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:780:41: note: in expansion of macro ‘_PyObject_CAST’
  780 | #  define PyType_Check(op) PyType_Check(_PyObject_CAST(op))
      |                                         ^~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyerrors.h:57:6: note: in expansion of macro ‘PyType_Check’
   57 |     (PyType_Check((x)) &&                                               \
      |      ^~~~~~~~~~~~
src/greenlet/greenlet_refs.hpp:993:17: note: in expansion of macro ‘PyExceptionClass_Check’
  993 |             if (PyExceptionClass_Check(type)) {
      |                 ^~~~~~~~~~~~~~~~~~~~~~
In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:44,
                 from src/greenlet/greenlet.cpp:14:
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: invalid cast from type ‘greenlet::refs::OwnedErrPiece’ to type ‘PyObject*’ {aka ‘_object*’}
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:774:59: note: in definition of macro ‘PyType_FastSubclass’
  774 | #define PyType_FastSubclass(type, flag) PyType_HasFeature(type, flag)
      |                                                           ^~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:136:31: note: in expansion of macro ‘_PyObject_CAST’
  136 | #  define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob))
      |                               ^~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyerrors.h:61:25: note: in expansion of macro ‘Py_TYPE’
   61 |     PyType_FastSubclass(Py_TYPE(x), Py_TPFLAGS_BASE_EXC_SUBCLASS)
      |                         ^~~~~~~
src/greenlet/greenlet_refs.hpp:1003:22: note: in expansion of macro ‘PyExceptionInstance_Check’
 1003 |             else if (PyExceptionInstance_Check(type)) {
      |                      ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:38,
                 from src/greenlet/greenlet.cpp:14:
src/greenlet/greenlet_thread_state.hpp: In destructor ‘greenlet::ThreadState::~ThreadState()’:
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: invalid cast from type ‘greenlet::refs::BorrowedObject’ {aka ‘greenlet::refs::BorrowedReference<_object, greenlet::refs::NoOpChecker>’} to type ‘PyObject*’ {aka ‘_object*’}
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:265:59: note: in expansion of macro ‘_PyObject_CAST’
  265 | #  define PyObject_TypeCheck(ob, type) PyObject_TypeCheck(_PyObject_CAST(ob), type)
      |                                                           ^~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/methodobject.h:17:31: note: in expansion of macro ‘PyObject_TypeCheck’
   17 | #define PyCFunction_Check(op) PyObject_TypeCheck(op, &PyCFunction_Type)
      |                               ^~~~~~~~~~~~~~~~~~
src/greenlet/greenlet_thread_state.hpp:400:33: note: in expansion of macro ‘PyCFunction_Check’
  400 |                              && PyCFunction_Check(refs.at(0))
      |                                 ^~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: invalid cast from type ‘greenlet::refs::BorrowedObject’ {aka ‘greenlet::refs::BorrowedReference<_object, greenlet::refs::NoOpChecker>’} to type ‘PyObject*’ {aka ‘_object*’}
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:127:35: note: in expansion of macro ‘_PyObject_CAST’
  127 | #  define Py_REFCNT(ob) Py_REFCNT(_PyObject_CAST(ob))
      |                                   ^~~~~~~~~~~~~~
src/greenlet/greenlet_thread_state.hpp:401:33: note: in expansion of macro ‘Py_REFCNT’
  401 |                              && Py_REFCNT(refs.at(0)) == 2) {
      |                                 ^~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: invalid cast from type ‘greenlet::refs::BorrowedObject’ {aka ‘greenlet::refs::BorrowedReference<_object, greenlet::refs::NoOpChecker>’} to type ‘PyObject*’ {aka ‘_object*’}
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:580:29: note: in expansion of macro ‘_PyObject_CAST’
  580 |         PyObject *_py_tmp = _PyObject_CAST(op); \
      |                             ^~~~~~~~~~~~~~
src/greenlet/greenlet_thread_state.hpp:422:33: note: in expansion of macro ‘Py_CLEAR’
  422 |                                 Py_CLEAR(function_w);
      |                                 ^~~~~~~~
src/greenlet/greenlet.cpp: In function ‘PyObject* green_repr(greenlet::refs::BorrowedGreenlet)’:
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: invalid cast from type ‘greenlet::refs::BorrowedGreenlet’ {aka ‘greenlet::refs::_BorrowedGreenlet<_greenlet, greenlet::refs::GreenletChecker>’} to type ‘PyObject*’ {aka ‘_object*’}
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:136:31: note: in expansion of macro ‘_PyObject_CAST’
  136 | #  define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob))
      |                               ^~~~~~~~~~~~~~
src/greenlet/greenlet.cpp:2395:33: note: in expansion of macro ‘Py_TYPE’
 2395 |     const char* const tp_name = Py_TYPE(self)->tp_name;
      |                                 ^~~~~~~
src/greenlet/greenlet_refs.hpp: In instantiation of ‘greenlet::refs::OwnedReference<T, <anonymous> >& greenlet::refs::OwnedReference<T, <anonymous> >::operator=(const greenlet::refs::OwnedReference<T, <anonymous> >&) [with T = _object; void (* TC)(void*) = greenlet::refs::NoOpChecker]’:
src/greenlet/greenlet_refs.hpp:908:11:   required from here
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: ‘reinterpret_cast’ from type ‘const _object*’ to type ‘PyObject*’ {aka ‘_object*’} casts away qualifiers
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:605:37: note: in expansion of macro ‘_PyObject_CAST’
  605 | #  define Py_XDECREF(op) Py_XDECREF(_PyObject_CAST(op))
      |                                     ^~~~~~~~~~~~~~
src/greenlet/greenlet_refs.hpp:342:13: note: in expansion of macro ‘Py_XDECREF’
  342 |             Py_XDECREF(tmp);
      |             ^~~~~~~~~~
src/greenlet/greenlet_refs.hpp: In instantiation of ‘greenlet::refs::OwnedReference<T, <anonymous> >& greenlet::refs::OwnedReference<T, <anonymous> >::operator=(const greenlet::refs::OwnedReference<T, <anonymous> >&) [with T = _object; void (* TC)(void*) = greenlet::refs::ContextExactChecker]’:
src/greenlet/greenlet.cpp:2337:40:   required from here
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: ‘reinterpret_cast’ from type ‘const _object*’ to type ‘PyObject*’ {aka ‘_object*’} casts away qualifiers
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:605:37: note: in expansion of macro ‘_PyObject_CAST’
  605 | #  define Py_XDECREF(op) Py_XDECREF(_PyObject_CAST(op))
      |                                     ^~~~~~~~~~~~~~
src/greenlet/greenlet_refs.hpp:342:13: note: in expansion of macro ‘Py_XDECREF’
  342 |             Py_XDECREF(tmp);
      |             ^~~~~~~~~~

when trying to build greenlets, however I have not bisected back to be sure this PR is at fault.

@vstinner
Copy link
Member Author

This suspect that this does not appear to preserve const correctness, I am seeing errors like:

The C API requires non-const PyObject*. Do you have an idea on how to fix the C++ cast to fix these errors?

@tacaswell
Copy link
Contributor

The C API requires non-const PyObject*. Do you have an idea on how to fix the C++ cast to fix these errors

My c++ is very rusty, but if the C API requires non-const there failures are correct. The casting to non-const should be done by greenlets (or what ever the calling library is as they should be aware that as escape hatch is being used). I will open an issue with them.

@vstinner
Copy link
Member Author

vstinner commented May 1, 2022

Sadly, it's a Python regression compared to Python 3.10. It should be fixed.

@serge-sans-paille
Copy link
Contributor

serge-sans-paille commented May 1, 2022 via email

@tacaswell
Copy link
Contributor

Sadly, it's a Python regression compared to Python 3.10. It should be fixed.

Take my view with a grain of 🧂 , but I think you could argue that this was a bug in py310 and below where constness was incorrectly disregarded? Looking at the changes I had to make to greenlet to get it to compile again, I think they are all more "correct" than the code was before.

The const_cast changes were in code that already had a similar cast for dealing with the c++ side of their code, it only had to be added to the python macro calls.

This is fixing the errors to looked like:

``` In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:38, from src/greenlet/greenlet.cpp:14: src/greenlet/greenlet_refs.hpp: In member function ‘void greenlet::refs::CreatedModule::PyAddObject(const char*, const PyObject*)’: /home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: ‘reinterpret_cast’ from type ‘const PyObject*’ {aka ‘const _object*’} to type ‘PyObject*’ {aka ‘_object*’} casts away qualifiers 20 | # define _Py_reinterpret_cast(type, expr) reinterpret_cast(expr) | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’ 107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op)) | ^~~~~~~~~~~~~~~~~~~~ /home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:505:35: note: in expansion of macro ‘_PyObject_CAST’ 505 | # define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op)) | ^~~~~~~~~~~~~~ src/greenlet/greenlet_refs.hpp:854:13: note: in expansion of macro ‘Py_INCREF’ 854 | Py_INCREF(new_object); | ^~~~~~~~~ ```

There is a another class of changes I had to make was to access the wrapped PyObject rather than blindly casting the wrapper object to pyobject (which I think used to work because in memory layout the PyObject was first so if you looked at that point in memory up to the length of PyObject you did indeed see a PyObject). It is not clear to me that you can move to the "new-style" casts and not have this issue. As with the const_cast, the resulting code reads as "more correct" to me than it did before.

These are the errors that look like:

/home/tcaswell/.pybuild/bleeding/include/python3.11/pyport.h:20:44: error: invalid cast from type ‘greenlet::refs::OwnedErrPiece’ to type ‘PyObject*’ {aka ‘_object*’}
   20 | #  define _Py_reinterpret_cast(type, expr) reinterpret_cast<type>(expr)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_reinterpret_cast’
  107 | #define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op))
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/object.h:780:41: note: in expansion of macro ‘_PyObject_CAST’
  780 | #  define PyType_Check(op) PyType_Check(_PyObject_CAST(op))
      |                                         ^~~~~~~~~~~~~~
/home/tcaswell/.pybuild/bleeding/include/python3.11/pyerrors.h:57:6: note: in expansion of macro ‘PyType_Check’
   57 |     (PyType_Check((x)) &&                                               \
      |      ^~~~~~~~~~~~
src/greenlet/greenlet_refs.hpp:993:17: note: in expansion of macro ‘PyExceptionClass_Check’
  993 |             if (PyExceptionClass_Check(type)) {
      |                 ^~~~~~~~~~~~~~~~~~~~~~
In file included from /home/tcaswell/.pybuild/bleeding/include/python3.11/Python.h:44,
                 from src/greenlet/greenlet.cpp:14:
```
</details>

@vstinner
Copy link
Member Author

vstinner commented May 2, 2022

PEP 670 was accepted with the condition: conversions must not introduce new compiler warnings (so no new compile errors neither). https://peps.python.org/pep-0670/

@vstinner
Copy link
Member Author

vstinner commented May 2, 2022

We are commenting a closed PR. I created a new issue to discuss the new C++ warning: #92135

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants