[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Python-Dev] Have a big machine and spare time? Here's a possible Python bug.

On Thu, May 23, 2019 at 5:15 PM Steve Dower <steve.dower at> wrote:

> On 23May2019 0542, Inada Naoki wrote:
> > 1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel
> space.
> > 2. This loop is cleary hot:
> >
> >
> > I can attach the process by gdb and I confirmed many arenas have
> > same nfreepools.
> It's relatively easy to test replacing our custom allocators with the
> system ones, yes? Can we try those to see whether they have the same
> characteristic?
> Given the relative amount of investment over the last 19 years [1], I
> wouldn't be surprised if most system ones are at least as good for our
> needs now. Certainly Windows HeapAlloc has had serious improvements in
> that time to help with fragmentation and small allocations.

FYI, and I've mentioned this at PyCon to a few people (might've been you,
Steve, I don't remember) -- but at Google we've experimented with disabling
obmalloc when using TCMalloc (a faster and thread-aware malloc, which makes
a huge difference within Google when dealing with multi-threaded C++
libraries), both using the usual Python benchmarks and real-world code with
real-world workloads (a core part of YouTube, for example), all on Linux.
There's still a significant benefit to using obmalloc when using glibc's
malloc, and also a noticeable benefit when using TCMalloc. There are
certainly cases where it doesn't matter much, and there may even be cases
where the overhead of obmalloc isn't worth it, but I do believe it's still
a firm net benefit.

> Cheers,
> Steve
> [1]:
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at
> Unsubscribe:

Thomas Wouters <thomas at>

Hi! I'm an email virus! Think twice before sending your email to help me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>