Initial PythonCallable implementation

[Thirumalai-Shaktivel]

Towards: #703

[Thirumalai-Shaktivel]

We need to handle the array as a return variable.

[Thirumalai-Shaktivel]

@certik Am I in the right direction?

[certik]

Yes, exactly!

[certik]

@Thirumalai-Shaktivel yes, I think that is exactly what I was imagining. Once this works, we remove lpython and rename temp_lpython to lpython.

[certik]

I would take this:

@pythoncallable
def fast_sum(n: i32, x: f64[:]) -> f64:
    s: f64 = 0.0
    i: i32
    for i in range(n):
        s += x[i]
    return s

and produce just one function in C:

static PyObject* fast_sum(PyObject* n, PyObject* x) {
// convert n from Python to LPython
// convert x from Python/NumPy array to LPython descriptor
// Now we emit code for
/*
    s: f64 = 0.0
    i: i32
    for i in range(n):
        s += x[i]
*/
// Here we convert s: f64 to CPython PyObject and return it correctly
//    return s
}

[Thirumalai-Shaktivel]

// Implementations
double fast_sum(int32_t n, struct r64* x, struct r64* y)
{
    int32_t __1_k;
    double _lpython_return_variable;
    int32_t i;
    double s;
    s =   0.00000000000000000e+00;
    for (i=0; i<=n - 1; i++) {
        s = s + x->data[(i - x->dims[0].lower_bound)];
    }
    for (__1_k=((int32_t)y->dims[1-1].lower_bound); __1_k<=((int32_t) y->dims[1-1].length + y->dims[1-1].lower_bound - 1); __1_k++) {
        printf("%lf%s", y->data[(__1_k - y->dims[0].lower_bound)], " ");
    }
    printf("%s", "\b");
    printf("\n");
    _lpython_return_variable = s;
    return _lpython_return_variable;
}

// Define the Python module and method mappings
static PyObject* fast_sum_define_module(PyObject* self, PyObject* args) {
    // Initialize NumPy
    import_array();

    // Declare arguments and return variable
    int32_t n;
    PyArrayObject *x;
    PyArrayObject *y;
    double _lpython_return_variable;

    // Parse the arguments from Python
    if (!PyArg_ParseTuple(args, "iOO", &n, &x, &y)) {
        PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
            "Failed to parse or receive arguments from Python");
        return NULL;
    }

    // fill array details for x
    if (PyArray_NDIM(x) != 1) {
        PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
            "Only 1 dimension array is supported for now.");
        return NULL;
    }

    struct r64 *s_array_x = malloc(sizeof(struct r64));
    {
        double *array;
        // Create C arrays from numpy objects:
        PyArray_Descr *descr = PyArray_DescrFromType(PyArray_TYPE(x));
        npy_intp dims[1];
        if (PyArray_AsCArray((PyObject **)&x, (void *)&array, dims, 1, descr) < 0) {
            PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
                "Failed to create a C array");
            return NULL;
        }

        s_array_x->data = array;
        s_array_x->n_dims = 1;
        s_array_x->dims[0].lower_bound = 0;
        s_array_x->dims[0].length = dims[0];
        s_array_x->is_allocated = false;
    }


    // fill array details for y
    if (PyArray_NDIM(y) != 1) {
        PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
            "Only 1 dimension array is supported for now.");
        return NULL;
    }

    struct r64 *s_array_y = malloc(sizeof(struct r64));
    {
        double *array;
        // Create C arrays from numpy objects:
        PyArray_Descr *descr = PyArray_DescrFromType(PyArray_TYPE(y));
        npy_intp dims[1];
        if (PyArray_AsCArray((PyObject **)&y, (void *)&array, dims, 1, descr) < 0) {
            PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
                "Failed to create a C array");
            return NULL;
        }

        s_array_y->data = array;
        s_array_y->n_dims = 1;
        s_array_y->dims[0].lower_bound = 0;
        s_array_y->dims[0].length = dims[0];
        s_array_y->is_allocated = false;
    }

    // Call the C function
    _lpython_return_variable = fast_sum(n, s_array_x, s_array_y);

    // Build and return the result as a Python object
    return Py_BuildValue("d", _lpython_return_variable);
}

// Define the module's method table
static PyMethodDef fast_sum_module_methods[] = {
    {"fast_sum", fast_sum_define_module, METH_VARARGS,
        "Handle arguments & return variable and call the function"},
    {NULL, NULL, 0, NULL}
};

// Define the module initialization function
static struct PyModuleDef fast_sum_module_def = {
    PyModuleDef_HEAD_INIT,
    "lpython_module_fast_sum",
    "Shared library to use LPython generated functions",
    -1,
    fast_sum_module_methods
};

PyMODINIT_FUNC PyInit_lpython_module_fast_sum(void) {
    PyObject* module;

    // Create the module object
    module = PyModule_Create(&fast_sum_module_def);
    if (!module) {
        return NULL;
    }

    return module;
}

[certik]

Let's rename fast_sum to _internal_fast_sum and rename fast_sum_define_module to fast_sum.

[certik]

Later, the pythoncallable decorator can take the decorated fast_sum function and keep it as internal_fast_sum. Then create a new fast_sum function that is roughly equivalent to:

fast_sum(n, x, y) {
    return internal_fast_sum(CastPythonToLPython(n), CastPythonToLPython(x), CastPythonToLPython(y));
}

This will be in ASR, so the arguments "x" will be BindPython in fast_sum, but LPython in internal_fast_sum.

[Thirumalai-Shaktivel]

C contents for lpython_decorator_02.py:

--show-c

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <Python.h>
#include <inttypes.h>
#include <numpy/ndarrayobject.h>

#include <stdlib.h>
#include <stdbool.h>
#include <stdio.h>
#include <string.h>
#include <lfortran_intrinsics.h>

struct dimension_descriptor
{
    int32_t lower_bound, length;
};

struct r64
{
    double *data;
    struct dimension_descriptor dims[32];
    int32_t n_dims;
    bool is_allocated;
};


// Implementations
void _xx_internal_multiply_xx(int32_t n, struct r64* x, struct r64* _lpython_return_variable)
{
    int32_t __1_t;
    int32_t __1_v;
    int32_t i;
    for (i=0; i<=n - 1; i++) {
        x->data[(i - x->dims[0].lower_bound)] = x->data[(i - x->dims[0].lower_bound)]*  5.00000000000000000e+00;
    }
    __1_v = ((int32_t)x->dims[1-1].lower_bound);
    for (__1_t=0; __1_t<=n + 0 - 1; __1_t++) {
        _lpython_return_variable->data[(__1_t - _lpython_return_variable->dims[0].lower_bound)] = x->data[(__1_v - x->dims[0].lower_bound)];
        __1_v = __1_v + 1;
    }
    return;
}

// Define the Python module and method mappings
static PyObject* multiply(PyObject* self, PyObject* args) {
    // Initialize NumPy
    import_array();

    // Declare arguments and return variable
    int32_t n;
    PyArrayObject *x;
    struct r64 *_lpython_return_variable = malloc(sizeof(struct r64));

    // Parse the arguments from Python
    if (!PyArg_ParseTuple(args, "iO", &n, &x)) {
        PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
            "Failed to parse or receive arguments from Python");
        return NULL;
    }

    // Fill array details for x
    if (PyArray_NDIM(x) != 1) {
        PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
            "Only 1 dimension array is supported for now.");
        return NULL;
    }

    struct r64 *s_array_x = malloc(sizeof(struct r64));
    {
        double *array;
        // Create C arrays from numpy objects:
        PyArray_Descr *descr = PyArray_DescrFromType(PyArray_TYPE(x));
        npy_intp dims[1];
        if (PyArray_AsCArray((PyObject **)&x, (void *)&array, dims, 1, descr) < 0) {
            PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
                "Failed to create a C array");
            return NULL;
        }

        s_array_x->data = array;
        s_array_x->n_dims = 1;
        s_array_x->dims[0].lower_bound = 0;
        s_array_x->dims[0].length = dims[0];
        s_array_x->is_allocated = false;
    }

    // Fill _lpython_return_variable
    _lpython_return_variable->data = malloc(n * sizeof(double));
    _lpython_return_variable->n_dims = 1;
    _lpython_return_variable->dims[0].lower_bound = 0;
    _lpython_return_variable->dims[0].length = n;
    _lpython_return_variable->is_allocated = false;

    // Call the C function
    _xx_internal_multiply_xx(n, s_array_x, _lpython_return_variable);

    // Copy the array elements and return the result as a Python object
    {
        npy_intp dims[] = {n};
        PyObject* numpy_array = PyArray_SimpleNewFromData(1, dims, NPY_DOUBLE,
            _lpython_return_variable->data);
        if (numpy_array == NULL) {
            PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
                "Failed to create an array that was used as a return variable");
            return NULL;
        }
        return numpy_array;
    }
}

// Define the module's method table
static PyMethodDef multiply_module_methods[] = {
    {"multiply", multiply, METH_VARARGS,
        "Handle arguments & return variable and call the function"},
    {NULL, NULL, 0, NULL}
};

// Define the module initialization function
static struct PyModuleDef multiply_module_def = {
    PyModuleDef_HEAD_INIT,
    "lpython_module_multiply",
    "Shared library to use LPython generated functions",
    -1,
    multiply_module_methods
};

PyMODINIT_FUNC PyInit_lpython_module_multiply(void) {
    PyObject* module;

    // Create the module object
    module = PyModule_Create(&multiply_module_def);
    if (!module) {
        return NULL;
    }

    return module;
}

void _xx_internal_multiply_x_xx(int32_t n, struct r64* x, int32_t __1x, int32_t __2x, struct r64* _lpython_return_variable)
{
    int32_t __1_t;
    int32_t __1_v;
    int32_t i;
    for (i=0; i<=n - 1; i++) {
        x->data[(i - x->dims[0].lower_bound)] = x->data[(i - x->dims[0].lower_bound)]*  5.00000000000000000e+00;
    }
    __1_v = __1x;
    for (__1_t=0; __1_t<=n + 0 - 1; __1_t++) {
        _lpython_return_variable->data[(__1_t - _lpython_return_variable->dims[0].lower_bound)] = x->data[(__1_v - x->dims[0].lower_bound)];
        __1_v = __1_v + 1;
    }
    return;
}

// Define the Python module and method mappings
static PyObject* multiply_x(PyObject* self, PyObject* args) {
    // Initialize NumPy
    import_array();

    // Declare arguments and return variable
    int32_t n;
    PyArrayObject *x;
    int32_t __1x;
    int32_t __2x;
    struct r64 *_lpython_return_variable = malloc(sizeof(struct r64));

    // Parse the arguments from Python
    if (!PyArg_ParseTuple(args, "iOii", &n, &x, &__1x, &__2x)) {
        PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
            "Failed to parse or receive arguments from Python");
        return NULL;
    }

    // Fill array details for x
    if (PyArray_NDIM(x) != 1) {
        PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
            "Only 1 dimension array is supported for now.");
        return NULL;
    }

    struct r64 *s_array_x = malloc(sizeof(struct r64));
    {
        double *array;
        // Create C arrays from numpy objects:
        PyArray_Descr *descr = PyArray_DescrFromType(PyArray_TYPE(x));
        npy_intp dims[1];
        if (PyArray_AsCArray((PyObject **)&x, (void *)&array, dims, 1, descr) < 0) {
            PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
                "Failed to create a C array");
            return NULL;
        }

        s_array_x->data = array;
        s_array_x->n_dims = 1;
        s_array_x->dims[0].lower_bound = 0;
        s_array_x->dims[0].length = dims[0];
        s_array_x->is_allocated = false;
    }

    // Fill _lpython_return_variable
    _lpython_return_variable->data = malloc(n * sizeof(double));
    _lpython_return_variable->n_dims = 1;
    _lpython_return_variable->dims[0].lower_bound = 0;
    _lpython_return_variable->dims[0].length = n;
    _lpython_return_variable->is_allocated = false;

    // Call the C function
    _xx_internal_multiply_x_xx(n, s_array_x, __1x, __2x, _lpython_return_variable);

    // Copy the array elements and return the result as a Python object
    {
        npy_intp dims[] = {n};
        PyObject* numpy_array = PyArray_SimpleNewFromData(1, dims, NPY_DOUBLE,
            _lpython_return_variable->data);
        if (numpy_array == NULL) {
            PyErr_SetString(PyExc_TypeError, "An error occurred in the `lpython` decorator: "
                "Failed to create an array that was used as a return variable");
            return NULL;
        }
        return numpy_array;
    }
}

// Define the module's method table
static PyMethodDef multiply_x_module_methods[] = {
    {"multiply_x", multiply_x, METH_VARARGS,
        "Handle arguments & return variable and call the function"},
    {NULL, NULL, 0, NULL}
};

// Define the module initialization function
static struct PyModuleDef multiply_x_module_def = {
    PyModuleDef_HEAD_INIT,
    "lpython_module_multiply_x",
    "Shared library to use LPython generated functions",
    -1,
    multiply_x_module_methods
};

PyMODINIT_FUNC PyInit_lpython_module_multiply_x(void) {
    PyObject* module;

    // Create the module object
    module = PyModule_Create(&multiply_x_module_def);
    if (!module) {
        return NULL;
    }

    return module;
}

[Thirumalai-Shaktivel]

This PR is ready for review!

[certik]

I think that it looks good. I don't see any major issues.

Is this ready for a final review & merge?

[Thirumalai-Shaktivel]

Yup, this is ready for final review and merge!

[Thirumalai-Shaktivel]

I will merge this and add more tests in a separate PR!
Thanks for the review!

[Thirumalai-Shaktivel]

The changes in the PR: #2012, alter the return type handling. So the seg fault. I will look into it and report back.

[Shaikh-Ubaid]

lpython/src/libasr/pass/pass_utils.h

Lines 670 to 672 in c459bb5

	if (ASRUtils::get_FunctionType(x)->m_abi == ASR::abiType::BindPython) {
	return;
	}

Maybe you could change the above as follows:

if (ASRUtils::get_FunctionType(x)->m_abi == ASR::abiType::BindPython
    && ASRUtils::get_FunctionType(x)->m_deftype == ASR::deftypeType::Interface) { 
    return; 
}

I guess it might fix the issue you are facing.

[Thirumalai-Shaktivel]

Ready

[Thirumalai-Shaktivel]

Thank you @Shaikh-Ubaid, but I modified the code to use your changes!
The changes were very useful, it reduces unwanted operations to handle array return type.

[certik]

I think this looks great.

Right now we use the C backend anyway, so this PR is an improvement.

To later use the LLVM backend, I think we need to refactor this into an ASR->ASR pass, see #1996 for details. This PR is the first step towards this (it moves the Python C/API logic from lpython.py into the C backend). The next step will be to lift it from the C backend into an ASR->ASR pass. Then LLVM should work out of the box.