git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Python-Dev] PEP 590 discussion


Hi,

On 02/04/2019 1:49 pm, Petr Viktorin wrote:
> On 3/30/19 11:36 PM, Jeroen Demeyer wrote:
>> On 2019-03-30 17:30, Mark Shannon wrote:
>>> 2. The claim that PEP 580 allows "certain optimizations because other
>>> code can make assumptions" is flawed. In general, the caller cannot make
>>> assumptions about the callee or vice-versa. Python is a dynamic 
>>> language.
>>
>> PEP 580 is meant for extension classes, not Python classes. Extension 
>> classes are not dynamic. When you implement tp_call in a given way, 
>> the user cannot change it. So if a class implements the C call 
>> protocol or the vectorcall protocol, callers can make assumptions 
>> about what that means.
>>
>>> PEP 579 is mainly a list of supposed flaws with the
>>> 'builtin_function_or_method' class.
>>> The general thrust of PEP 579 seems to be that builtin-functions and
>>> builtin-methods should be more flexible and extensible than they are. I
>>> don't agree. If you want different behaviour, then use a different
>>> object. Don't try an cram all this extra behaviour into a pre-existing
>>> object.
>>
>> I think that there is a misunderstanding here. I fully agree with the 
>> "use a different object" solution. This isn't a new solution: it's 
>> already possible to implement those different objects (Cython does 
>> it). It's just that this solution comes at a performance cost and 
>> that's what we want to avoid.
> 
> It does seem like there is some misunderstanding.
> 
> PEP 580 defines a CCall structure, which includes the function pointer, 
> flags, "self" and "parent". Like the current implementation, it has 
> various METH_ flags for various C signatures. When called, the info from 
> CCall is matched up (in relatively complex ways) to what the C function 
> expects.
> 
> PEP 590 only adds the "vectorcall". It does away with flags and only has 
> one C signatures, which is designed to fit all the existing ones, and is 
> well optimized. Storing the "self"/"parent", and making sure they're 
> passed to the C function is the responsibility of the callable object.
> There's an optimization for "self" (offsetting using 
> PY_VECTORCALL_ARGUMENTS_OFFSET), and any supporting info can be provided 
> as part of "self". >
>>> I'll reiterate that PEP 590 is more general than PEP 580 and that once
>>> the callable's code has access to the callable object (as both PEPs
>>> allow) then anything is possible. You can't can get more extensible than
>>> that.
> 
> Anything is possible, but if one of the possibilities becomes common and 
> useful, PEP 590 would make it hard to optimize for it.
> Python has grown many "METH_*" signatures over the years as we found 
> more things that need to be passed to callables. Why would 
> "METH_VECTORCALL" be the last? If it won't (if you think about it as one 
> more way to call functions), then dedicating a tp_* slot to it sounds 
> quite expensive.

I doubt METH_VECTORCALL will be the last.
Let me give you an example: It is quite common for a function to take 
two arguments, so we might want add a METH_OO flag for builtin-functions 
with 2 parameters.

To support this in PEP 590, you would make exactly the same change as 
you would now; which is to add another case to the switch statement in 
_PyCFunction_FastCallKeywords.
For PEP 580, you would add another case to the switch in PyCCall_FastCall.

No difference really.

PEP 580 uses a slot as well. It's only 8 bytes per class.

> 
> 
> In one of the ways to call C functions in PEP 580, the function gets 
> access to:
> - the arguments,
> - "self", the object
> - the class that the method was found in (which is not necessarily 
> type(self))
> I still have to read the details, but when combined with 
> LOAD_METHOD/CALL_METHOD optimization (avoiding creation of a "bound 
> method" object), it seems impossible to do this efficiently with just 
> the callable's code and callable's object.

It is possible, and relatively straightforward.
Why do you think it is impossible?

> 
> 
>> I would argue the opposite: PEP 590 defines a fixed protocol that is 
>> not easy to extend. PEP 580 on the other hand uses a new data 
>> structure PyCCallDef which could easily be extended in the future 
>> (this will intentionally never be part of the stable ABI, so we can do 
>> that).
>>
>> I have also argued before that the generality of PEP 590 is a bad 
>> thing rather than a good thing: by defining a more rigid protocol as 
>> in PEP 580, more optimizations are possible.
>>
>>> PEP 580 has the same limitation for the same reasons. The limitation is
>>> necessary for correctness if an object supports calls via `__call__` and
>>> through another calling convention.
>>
>> I don't think that this limitation is needed in either PEP. As I 
>> explained at the top of this email, it can easily be solved by not 
>> using the protocol for Python classes. What is wrong with my proposal 
>> in PEP 580: https://www.python.org/dev/peps/pep-0580/#inheritance
> 
> 
> I'll add Jeroen's notes from the review of the proposed PEP 590
> (https://github.com/python/peps/pull/960):
> 
> The statement "PEP 580 is specifically targetted at function-like 
> objects, and doesn't support other callables like classes, partial 
> functions, or proxies" is factually false. The motivation for PEP 580 is 
> certainly function/method-like objects but it's a general protocol that 
> every class can implement. For certain classes, it may not be easy or 
> desirable to do that but it's always possible. >
> Given that `PY_METHOD_DESCRIPTOR` is a flag for tp_flags, shouldn't it 
> be called `Py_TPFLAGS_METHOD_DESCRIPTOR` or something?
> 
> Py_TPFLAGS_HAVE_VECTOR_CALL should be Py_TPFLAGS_HAVE_VECTORCALL, to be 
> consistent with tp_vectorcall_offset and other uses of "vectorcall" (not 
> "vector call")
> 

Thanks for the comments, I'll update the PEP when I get the chance.

> 
> And mine, so far:
> 
> I'm not clear on the constness of the "args" array.
> If it is mutable (PyObject **), you can't, for example, directly pass a 
> tuple's storage (or any other array that could be used in the call).
> If it is not (PyObject * const *), you can't insert the "self" argument in.
> The reference implementations seems to be inconsistent here. What's the 
> intention?
> 

I'll make it clearer in the PEP.
My thinking was that if `PY_VECTORCALL_ARGUMENTS_OFFSET` is set then the 
caller is allowing the callee to mutate element -1.
It would make sense to generalise that to any element of the vector 
(including -1).
When passing the contents of a tuple, `PY_VECTORCALL_ARGUMENTS_OFFSET` 
should not be set, and thus the vector could not be mutated.


Cheers,
Mark.