[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Python-Dev] General concerns about C API changes


I made many different kinds of changes to the C API last weeks. None
of them should impact the backward compatibility. If it's the case,
the changes causing the backward compatibility should be reverted. The
most controversial changes are the conversion of macros to static
inline functions, but these changes have been discussed in length and
approved by multiple core developers.

(*) Move private internal API outside Include/: continue the work
started in Python 3.7 to move it to Include/internal/

This work is almost complete. The remaining issue is the
_PyObject_GC_TRACK() function.

The main change is that an explicit #include "pycore_<header>.h" is
now required in C code.

(*) Move "unstable" API outside Include/: move "#ifndef
Py_LIMITED_API" code to a new subdirectory

This work didn't start. It's still being discussed to see how we
should do it. I chose to try to finish Include/internal/ first.

(*) Add a *new* C API which doesn't leak implementation details

This work didn't start. It's still under discussion, I'm not sure that
it's going to happen in the master branch in the short term.

(*) Convert macros to static inline functions

I guess that Raymond is worried by the impact on performance of these changes.

So far, the following macros have been converted to static inline functions:

* PyObject_INIT(), PyObject_INIT_VAR()
* _Py_NewReference(), _Py_ForgetReference()
* _Py_Dealloc()

First, I attempted for "force inlining" (ex:
__attribute__((always_inline)) for GCC/Clang), but it has been decided
to not do that. Please read the discussion on the issue for the

I modified the Visual Studio project to ask the compiler to respect
"inline" *hint* when CPython is compiled in debug mode: "Set
InlineFunctionExpansion to OnlyExplicitInline ("/Ob1" option) on all
projects (in pyproject.props) in Debug mode on Win32 and x64 platforms
to expand functions marked as inline." This change spotted a bug in
_decimal ("Add missing EXTINLINE in mpdecimal.h").

Macros have many pitfalls, and the intent here is to prevent these pitfalls.:

_Py_Dealloc() example:

#define _Py_INC_TPFREES(OP)     dec_count(Py_TYPE(OP))
#define _Py_INC_TPFREES(OP)
#endif /* COUNT_ALLOCS */

#define _Py_Dealloc(op) (                               \
    _Py_INC_TPFREES(op) _Py_COUNT_ALLOCS_COMMA          \
    (*Py_TYPE(op)->tp_dealloc)((PyObject *)(op)))

_Py_Dealloc() produced something like "dealloc()" or "dec_count(),
dealloc()" depending on compiler options on Python 3.7. On Python 3.8,
it's now a well defined function call which returns "void" (no
result): "[static inline] void _Py_Dealloc(PyObject *);"

Another example:

#define Py_DECREF(op)                                   \
    do {                                                \
        PyObject *_py_decref_tmp = (PyObject *)(op);    \
        if (_Py_DEC_REFTOTAL  _Py_REF_DEBUG_COMMA       \
        --(_py_decref_tmp)->ob_refcnt != 0)             \
            _Py_CHECK_REFCNT(_py_decref_tmp)            \
        else                                            \
            _Py_Dealloc(_py_decref_tmp);                \
    } while (0)

This macro required a temporary "_py_decref_tmp" variable in Python
3.7, to cast the argument to PyObject*. It's gone in Python 3.8:

static inline void _Py_DECREF(const char *filename, int lineno,
                              PyObject *op)
    if (--op->ob_refcnt != 0) {
#ifdef Py_REF_DEBUG
        if (op->ob_refcnt < 0) {
            _Py_NegativeRefcount(filename, lineno, op);
    else {

#define Py_DECREF(op) _Py_DECREF(__FILE__, __LINE__, (PyObject *)(op))

The cast is now done in Py_DECREF() macro.

The Py_DECREF() contract is that it accepts basically any pointer. I
chose to not change that (always require the exact PyObject* type and
nothing else), since it would create a compiler warning on basically
every usage of Py_DECREF() for no real benefit.