[Python-Dev] PEP 590 discussion
On 02/04/2019 1:49 pm, Petr Viktorin wrote:
> On 3/30/19 11:36 PM, Jeroen Demeyer wrote:
>> On 2019-03-30 17:30, Mark Shannon wrote:
>>> 2. The claim that PEP 580 allows "certain optimizations because other
>>> code can make assumptions" is flawed. In general, the caller cannot make
>>> assumptions about the callee or vice-versa. Python is a dynamic
>> PEP 580 is meant for extension classes, not Python classes. Extension
>> classes are not dynamic. When you implement tp_call in a given way,
>> the user cannot change it. So if a class implements the C call
>> protocol or the vectorcall protocol, callers can make assumptions
>> about what that means.
>>> PEP 579 is mainly a list of supposed flaws with the
>>> 'builtin_function_or_method' class.
>>> The general thrust of PEP 579 seems to be that builtin-functions and
>>> builtin-methods should be more flexible and extensible than they are. I
>>> don't agree. If you want different behaviour, then use a different
>>> object. Don't try an cram all this extra behaviour into a pre-existing
>> I think that there is a misunderstanding here. I fully agree with the
>> "use a different object" solution. This isn't a new solution: it's
>> already possible to implement those different objects (Cython does
>> it). It's just that this solution comes at a performance cost and
>> that's what we want to avoid.
> It does seem like there is some misunderstanding.
> PEP 580 defines a CCall structure, which includes the function pointer,
> flags, "self" and "parent". Like the current implementation, it has
> various METH_ flags for various C signatures. When called, the info from
> CCall is matched up (in relatively complex ways) to what the C function
> PEP 590 only adds the "vectorcall". It does away with flags and only has
> one C signatures, which is designed to fit all the existing ones, and is
> well optimized. Storing the "self"/"parent", and making sure they're
> passed to the C function is the responsibility of the callable object.
> There's an optimization for "self" (offsetting using
> PY_VECTORCALL_ARGUMENTS_OFFSET), and any supporting info can be provided
> as part of "self". >
>>> I'll reiterate that PEP 590 is more general than PEP 580 and that once
>>> the callable's code has access to the callable object (as both PEPs
>>> allow) then anything is possible. You can't can get more extensible than
> Anything is possible, but if one of the possibilities becomes common and
> useful, PEP 590 would make it hard to optimize for it.
> Python has grown many "METH_*" signatures over the years as we found
> more things that need to be passed to callables. Why would
> "METH_VECTORCALL" be the last? If it won't (if you think about it as one
> more way to call functions), then dedicating a tp_* slot to it sounds
> quite expensive.
I doubt METH_VECTORCALL will be the last.
Let me give you an example: It is quite common for a function to take
two arguments, so we might want add a METH_OO flag for builtin-functions
with 2 parameters.
To support this in PEP 590, you would make exactly the same change as
you would now; which is to add another case to the switch statement in
For PEP 580, you would add another case to the switch in PyCCall_FastCall.
No difference really.
PEP 580 uses a slot as well. It's only 8 bytes per class.
> In one of the ways to call C functions in PEP 580, the function gets
> access to:
> - the arguments,
> - "self", the object
> - the class that the method was found in (which is not necessarily
> I still have to read the details, but when combined with
> LOAD_METHOD/CALL_METHOD optimization (avoiding creation of a "bound
> method" object), it seems impossible to do this efficiently with just
> the callable's code and callable's object.
It is possible, and relatively straightforward.
Why do you think it is impossible?
>> I would argue the opposite: PEP 590 defines a fixed protocol that is
>> not easy to extend. PEP 580 on the other hand uses a new data
>> structure PyCCallDef which could easily be extended in the future
>> (this will intentionally never be part of the stable ABI, so we can do
>> I have also argued before that the generality of PEP 590 is a bad
>> thing rather than a good thing: by defining a more rigid protocol as
>> in PEP 580, more optimizations are possible.
>>> PEP 580 has the same limitation for the same reasons. The limitation is
>>> necessary for correctness if an object supports calls via `__call__` and
>>> through another calling convention.
>> I don't think that this limitation is needed in either PEP. As I
>> explained at the top of this email, it can easily be solved by not
>> using the protocol for Python classes. What is wrong with my proposal
>> in PEP 580: https://www.python.org/dev/peps/pep-0580/#inheritance
> I'll add Jeroen's notes from the review of the proposed PEP 590
> The statement "PEP 580 is specifically targetted at function-like
> objects, and doesn't support other callables like classes, partial
> functions, or proxies" is factually false. The motivation for PEP 580 is
> certainly function/method-like objects but it's a general protocol that
> every class can implement. For certain classes, it may not be easy or
> desirable to do that but it's always possible. >
> Given that `PY_METHOD_DESCRIPTOR` is a flag for tp_flags, shouldn't it
> be called `Py_TPFLAGS_METHOD_DESCRIPTOR` or something?
> Py_TPFLAGS_HAVE_VECTOR_CALL should be Py_TPFLAGS_HAVE_VECTORCALL, to be
> consistent with tp_vectorcall_offset and other uses of "vectorcall" (not
> "vector call")
Thanks for the comments, I'll update the PEP when I get the chance.
> And mine, so far:
> I'm not clear on the constness of the "args" array.
> If it is mutable (PyObject **), you can't, for example, directly pass a
> tuple's storage (or any other array that could be used in the call).
> If it is not (PyObject * const *), you can't insert the "self" argument in.
> The reference implementations seems to be inconsistent here. What's the
I'll make it clearer in the PEP.
My thinking was that if `PY_VECTORCALL_ARGUMENTS_OFFSET` is set then the
caller is allowing the callee to mutate element -1.
It would make sense to generalise that to any element of the vector
When passing the contents of a tuple, `PY_VECTORCALL_ARGUMENTS_OFFSET`
should not be set, and thus the vector could not be mutated.