git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build


On 10Apr2019 0401, Victor Stinner wrote:
> Le mar. 9 avr. 2019 ? 22:16, Steve Dower <steve.dower at python.org> a ?crit :
>> What are the other changes that would be required?
> 
> I don't know.
> 
>> And is there another
>> way to get the same functionality without ABI modifications?
> 
> Py_TRACE_REFS is a double linked list of *all* Python objects. To get
> this functionality, you need to store the list somewhere. I don't know
> how to maintain such list outside the PyObject structure.

There's certainly no more convenient way to do it. Maybe if we had 
detached reference counts it would be easier, but it would likely still 
result in ABI compatibility issues between debug builds of extensions 
and release builds of Python (the most common scenario, in my experience).

> One solution would be to enable Py_TRACE_REFS in release mode. Does
> anyone want to add 16 bytes to every PyObject? I don't want that :-)

Yeah, nobody suggested that anyway :)

>> I think it's worthwhile if we can really get to debug and non-debug
>> builds being ABI compatible. Getting partway there in this case doesn't
>> seem to offer any benefits.
> 
> Disabling Py_TRACE_REFS by default in debug mode reduces the Python
> memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16
> bytes on 64-bit platforms.

Right, except it's debug mode.

> I don't think that I ever used sys.getobjects(), whereas many projects
> use gc.get_objects() which is also available in release builds (not
> only in debug builds).
> 
> I'm quite sure that almost nobody uses debug builds because the ABI is
> incompatible.

There were just over 250,000 downloads of the prebuilt debug binaries 
for Windows (which are optional in the installer and turned off by 
default) in March. Whether they are being used is another question, but 
I know for sure at least a few people who use them.

When you want to use a debug build of your extension module, using a 
debug build of CPython is the only way to do it. So unless we can get 
rid of *all* the ABI incompatibilities, a debug build of CPython is 
still going to be necessary and disabling/removing reference tracking 
doesn't provide any benefit.

> The main question is if anyone ever used Py_TRACE_REFS? Does someone
> use sys.getobjects() or PYTHONDUMPREFS environment variable?
> 
> Using PYTHONDUMPREFS=1 on a debug build (with Py_TRACE_REFS) does
> simply crash Python 3.7 at exit. So I don't think that anyone use it
> :-)

How do we track reference leaks in the buildbots? Can/should we be using 
this?

It doesn't crash on Python 3.8, so I suspect fixing the bug is a better 
option than using it as an excuse to remove the feature. From a quick 
test, it seems that a tuple element is being freed but not removed from 
the tuple, so it's probably a double-decref bug somewhere in 3.7.

Cheers,
Steve