diff options
author | David Kalnischkies <david@kalnischkies.de> | 2019-04-27 16:13:02 +0200 |
---|---|---|
committer | David Kalnischkies <david@kalnischkies.de> | 2019-07-08 13:18:31 +0200 |
commit | 79b1a82983e737e74359bc306d9edb357c5bdd46 (patch) | |
tree | 28fe86abdf88e54e8e9407e81e59d01e149a2ef2 /ftparchive | |
parent | 8ff87e9cd37a4436eb7e56f814a099cb30845ae1 (diff) |
Distribute host-less work based on backlog of the queues
Work like applying patches via rred can be performed by many concurrent
rred processes, but we can't just spawn new ones forever: We limit us to
the number of CPUs which can drive them and reuse existing ones if they
have nothing to do at the moment.
The problem arises if we have reached the limit of queues and all of
them are busy which is more likely to happen on "slow" machines with few
CPUs. In this case we opted for random distribution, but that can result
in many big files (e.g. Contents) being added to one queue while the
others get none or only small files.
Ideally we would ask the methods how much they still have to do, but
they only know that for the current item, not for all items in the
queue, so we use the filesize of the expected result.
Diffstat (limited to 'ftparchive')
0 files changed, 0 insertions, 0 deletions