Enhancement request for PyType_FromSpecWIthBases add option for meta class

[Thrameos]

BPO	42617
Nosy	@encukou, @shihai1991, @Thrameos

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-10.21:40:40.021>
labels = ['expert-C-API', 'type-feature', '3.10']
title = 'Enhancement request for PyType_FromSpecWIthBases add option for meta class'
updated_at = <Date 2020-12-12.19:47:26.352>
user = 'https://github.com/Thrameos'

bugs.python.org fields:

activity = <Date 2020-12-12.19:47:26.352>
actor = 'Thrameos'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2020-12-10.21:40:40.021>
creator = 'Thrameos'
dependencies = []
files = []
hgrepos = []
issue_num = 42617
keywords = []
message_count = 3.0
messages = ['382850', '382911', '382920']
nosy_count = 3.0
nosy_names = ['petr.viktorin', 'shihai1991', 'Thrameos']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue42617'
versions = ['Python 3.10']

[Thrameos]

PyType_FromSpecWithBases is missing an argument for taking a meta class. As a result it is necessary to replicate a large portion of Python code when I need to create a new heap type with a specified meta class. This is a maintenance issue as replicating Python code is likely to get broken in future.

I have replicated to code from JPype so that it is clear how the meta class is coming into place. PyJPClass_Type is derived from a
PyHeapType with additional slots for Java to use and overrides allocation slots to that it can add extra slots (needs true multiple inheritance as it needed to add Java slots to types with mismatching memory layouts object, long, float, exception, ...).

This code could be made much safer if there were a PyType_FromSpecWithBasesMeta which used the meta class to allocate the memory (then I could safely override the slots after creation).

PyObject* PyJPClass_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
{
        JP_PY_TRY("PyJPClass_FromSpecWithBases");
        // Python lacks a FromSpecWithMeta so we are going to have to fake it here.
        PyTypeObject* type = (PyTypeObject*) PyJPClass_Type->tp_alloc(PyJPClass_Type, 0); // <== we need to use the meta class here
        PyHeapTypeObject* heap = (PyHeapTypeObject*) type;
        type->tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE | Py_TPFLAGS_HAVE_GC;
        type->tp_name = spec->name;
        const char *s = strrchr(spec->name, '.');
        if (s == NULL)
                s = spec->name;
        else
                s++;
        heap->ht_qualname = PyUnicode_FromString(s);
        heap->ht_name = heap->ht_qualname;
        Py_INCREF(heap->ht_name);
        if (bases == NULL)
                type->tp_bases = PyTuple_Pack(1, (PyObject*) & PyBaseObject_Type);
        else
        {
                type->tp_bases = bases;
                Py_INCREF(bases);
        }
        type->tp_base = (PyTypeObject*) PyTuple_GetItem(type->tp_bases, 0);
        Py_INCREF(type->tp_base);
        type->tp_as_async = &heap->as_async;
        type->tp_as_buffer = &heap->as_buffer;
        type->tp_as_mapping = &heap->as_mapping;
        type->tp_as_number = &heap->as_number;
        type->tp_as_sequence = &heap->as_sequence;
        type->tp_basicsize = spec->basicsize;
        if (spec->basicsize == 0)
                type->tp_basicsize = type->tp_base->tp_basicsize;
        type->tp_itemsize = spec->itemsize;
        if (spec->itemsize == 0)
                type->tp_itemsize = type->tp_base->tp_itemsize;

        // <=== Replicated code from the meta class 
        type->tp_alloc = PyJPValue_alloc;
        type->tp_free = PyJPValue_free;
        type->tp_finalize = (destructor) PyJPValue_finalize;

        // <= Replicated code from Python
        for (PyType_Slot* slot = spec->slots; slot->slot; slot++)
        {
               switch (slot->slot)
                {
                        case Py_tp_free:
                                type->tp_free = (freefunc) slot->pfunc;
                                break;
                        case Py_tp_new:
                                type->tp_new = (newfunc) slot->pfunc;
                                break;
                        case Py_tp_init:
                                type->tp_init = (initproc) slot->pfunc;
                                break;
                        case Py_tp_getattro:
                                type->tp_getattro = (getattrofunc) slot->pfunc;
                                break;
                        case Py_tp_setattro:
                                type->tp_setattro = (setattrofunc) slot->pfunc;
                                break;
                        case Py_tp_dealloc:
                                type->tp_dealloc = (destructor) slot->pfunc;
                                break;
                        case Py_tp_str:
                                type->tp_str = (reprfunc) slot->pfunc;
                                break;
                        case Py_tp_repr:
                                type->tp_repr = (reprfunc) slot->pfunc;
                                break;
                        case Py_tp_methods:
                                type->tp_methods = (PyMethodDef*) slot->pfunc;
                                break;
                        case Py_sq_item:
                                heap->as_sequence.sq_item = (ssizeargfunc) slot->pfunc;
                                break;
                        case Py_sq_length:
                                heap->as_sequence.sq_length = (lenfunc) slot->pfunc;
                                break;
                        case Py_mp_ass_subscript:
                                heap->as_mapping.mp_ass_subscript = (objobjargproc) slot->pfunc;
                                break;
                        case Py_tp_hash:
                                type->tp_hash = (hashfunc) slot->pfunc;
                                break;
                        case Py_nb_int:
                                heap->as_number.nb_int = (unaryfunc) slot->pfunc;
                                break;
                        case Py_nb_float:
                                heap->as_number.nb_float = (unaryfunc) slot->pfunc;
                                break;
                        case Py_tp_richcompare:
                                type->tp_richcompare = (richcmpfunc) slot->pfunc;
                                break;
                        case Py_mp_subscript:
                                heap->as_mapping.mp_subscript = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_index:
                                heap->as_number.nb_index = (unaryfunc) slot->pfunc;
                                break;
                        case Py_nb_absolute:
                                heap->as_number.nb_absolute = (unaryfunc) slot->pfunc;
                                break;
                        case Py_nb_and:
                                heap->as_number.nb_and = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_or:
                                heap->as_number.nb_or = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_xor:
                                heap->as_number.nb_xor = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_add:
                                heap->as_number.nb_add = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_subtract:
                                heap->as_number.nb_subtract = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_multiply:
                                heap->as_number.nb_multiply = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_rshift:
                                heap->as_number.nb_rshift = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_lshift:
                                heap->as_number.nb_lshift = (binaryfunc) slot->pfunc;
                               break;
                        case Py_nb_negative:
                                heap->as_number.nb_negative = (unaryfunc) slot->pfunc;
                                break;
                        case Py_nb_bool:
                                heap->as_number.nb_bool = (inquiry) slot->pfunc;
                                break;
                        case Py_nb_invert:
                                heap->as_number.nb_invert = (unaryfunc) slot->pfunc;
                                break;
                        case Py_nb_positive:
                                heap->as_number.nb_positive = (unaryfunc) slot->pfunc;
                                break;
                        case Py_nb_floor_divide:
                                heap->as_number.nb_floor_divide = (binaryfunc) slot->pfunc;
                                break;
                        case Py_nb_divmod:
                                heap->as_number.nb_divmod = (binaryfunc) slot->pfunc;
                                break;
                        case Py_tp_getset:
                                type->tp_getset = (PyGetSetDef*) slot->pfunc;
                                break;
                                // GCOVR_EXCL_START
                        default:
                                PyErr_Format(PyExc_TypeError, "slot %d not implemented", slot->slot);
                                JP_RAISE_PYTHON();
                                // GCOVR_EXCL_STOP
                }
        }
        PyType_Ready(type);
        PyDict_SetItemString(type->tp_dict, "__module__", PyUnicode_FromString("_jpype"));
        return (PyObject*) type;
        JP_PY_CATCH(NULL); // GCOVR_EXCL_LINE
}

[shihai1991]

This code could be made much safer if there were a PyType_FromSpecWithBasesMeta which used the meta class to allocate the memory

IMHO, PyType_FromSpecWithBases is a atomic API of TypeObject.
Adding a class as a param will result in PyType_FromSpecWithBases will be complicated(the logic of this function depends on the children's tp_alloc sometimes).

I add pter in this bpo, maybe he have some better ideas :)

[Thrameos]

Perhaps just having PyType_InitHeapFromSpec which have just heap type linkage and slot copy section would help with the issue. The major problem I have is maintaining the slot code which may change at some point in the future. There would still be some risk of missing a portion of the init procedure but it would be a bit safer some of the process was in Python.

[kumaraditya303]

Fixed by #93012