More CPUs doen't equal more speed
On 2019-05-24 01:22, Chris Angelico wrote:
> On Fri, May 24, 2019 at 10:07 AM Bob van der Poel <bob at mellowood.ca> wrote:
>> Thanks all! The sound you are hearing is my head smacking against my hand!
>> Or is it my hand against my head?
>> Anyway, yes the problem is that I was naively using command.getoutput()
>> which blocks until the command is finished. So, of course, only one process
>> was being run at one time! Bad me!
>> I guess I should be looking at subprocess.Popen(). Now, a more relevant
>> question ... if I do it this way I then need to poll though a list of saved
>> process IDs to see which have finished? Right? My initial thought is to
>> batch them up in small groups (say CPU_COUNT-1) and wait for that batch to
>> finish, etc. Would it be foolish to send send a large number (1200 in this
>> case since this is the number of files) and let the OS worry about
>> scheduling and have my program poll 1200 IDs?
> That might create a lot of contention, resulting in poor performance.
> But depending on what your tasks saturate on, that might not matter
> all that much, and it _would_ be a simple and straight-forward
> technique. In fact, you could basically just write your code like
> for job in jobs:
> for process in processes:
> Once they're all started, you just wait for the first one to finish.
> Then when that's finished, wait for the next, and the next, and the
> next. If the first process started is actually the slowest to run, all
> the others will be in the "done" state for a while, but that's not a
> big deal.
>> Someone mentioned the GIL. If I launch separate processes then I don't
>> encounter this issue? Right?
> The GIL is basically irrelevant here. Most of the work is being done
> in subprocesses, so your code is spending all its time waiting.
> What I'd recommend is a thread pool. Broadly speaking, it would look
> something like this:
> jobs = [...]
> def run_jobs():
> while jobs:
> try: job = jobs.pop()
> except IndexError: break # deal with race
> threads = [threading.Thread(target=run_jobs).start()
> for _ in THREAD_COUNT]
> for thread in threads:
> Note that this has the same "start them all, then wait on them in
> order" model. In this case, though, there won't be 1200 threads -
> there'll be THREAD_COUNT of them (which may not be the same as your
> CPU count, but you could use that same figure as an initial estimate).
> Within each thread, the logic is also quite simple: take a job, do the
> job, repeat till you run out of jobs. The GIL ensures that "job =
> jobs.pop()" is a safe atomic operation that can't possibly corrupt
> internal state, and will always retrieve a unique job every time. The
> run_jobs function simply runs one job at a time, waiting for its
> This kind of pattern keeps everything clean and simple, and is easy to
> tweak for performance.
Personally, I'd use a queue (from the 'queue' module), instead of a
list, for the job pool.