[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

On Fri, Nov 9, 2018 at 5:50 PM Nathaniel Smith <njs at> wrote:

> On Fri, Nov 9, 2018 at 4:30 PM, Victor Stinner <vstinner at>
> wrote:
> > Ah, important points. I don't want to touch the current C API nor make
> > it less efficient. And compatibility in both directions (current C API
> > <=> new C API) is very important for me. There is no such plan as
> > "Python 4" which would break the world and *force* everybody to
> > upgrade to the new C API, or stay to Python 3 forever. No. The new C
> > API must be an opt-in option, and current C API remains the default
> > and not be changed.
> Doesn't this mean that you're just making the C API larger and more
> complicated, rather than simplifying it? You cite some benefits
> (tagged pointers, changing the layout of PyObject, making PyPy's life
> easier), but I don't see how you can do any of those things so long as
> the current C API remains supported.
> -n

I believe the implied missing thing from Victor's description is this:

Experimentation with new internal implementations can begin once we have a
new C API by explicitly breaking the old C API with-in such experiments (as
is required for most anything interesting).  All code that is written to
the new C API still works during this process, thus making the job of
practical testing of such new VM internals easier.

>From there, you can make decisions on how heavily to push the world towards
adoption of the new C API and by when so that a runtime not supporting the
old API can be realized with a list of enticing carrot tasting benefits.
(devising any necessary pypy cpyext-like compatibility solutions for super
long lived code or things that sadly want to use the 3.7 ABI for 10 years
in the process - and similarly an extension to provide some or all of this
API/ABI on top of older existing stable Python releases!)

I'd *love* to get to a situation where the only valid ABI we support knows
nothing about internal structs at all. Today, PyObject memory layout is
exposed to the world and unchangable. :(

This is a long process release wise (assume multiple stable releases go by
before we could declare that). But *we've got to start* by defining what we
want to provide as a seriously limited but functional API and ABI even if
it doesn't perform as well as things compiled against our existing
exposed-internals C API.  For *most* extension modules, performance of this
sort is not important. For the numpys of the world life is more
complicated, we should work with them to figure out their C API needs.

If it wasn't already obvious, you've got my support on this. :)

PS If the conversation devolves to arguing about "new" being a bad name,
that's a good sign.  I suggest calling it the Vorpal API after the bunny.
Or be boring and just use a year number for the name.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>