summaryrefslogtreecommitdiff
path: root/ftparchive
diff options
context:
space:
mode:
authorDavid Kalnischkies <david@kalnischkies.de>2019-04-27 16:13:02 +0200
committerDavid Kalnischkies <david@kalnischkies.de>2019-07-08 13:18:31 +0200
commit79b1a82983e737e74359bc306d9edb357c5bdd46 (patch)
tree28fe86abdf88e54e8e9407e81e59d01e149a2ef2 /ftparchive
parent8ff87e9cd37a4436eb7e56f814a099cb30845ae1 (diff)
Distribute host-less work based on backlog of the queues
Work like applying patches via rred can be performed by many concurrent rred processes, but we can't just spawn new ones forever: We limit us to the number of CPUs which can drive them and reuse existing ones if they have nothing to do at the moment. The problem arises if we have reached the limit of queues and all of them are busy which is more likely to happen on "slow" machines with few CPUs. In this case we opted for random distribution, but that can result in many big files (e.g. Contents) being added to one queue while the others get none or only small files. Ideally we would ask the methods how much they still have to do, but they only know that for the current item, not for all items in the queue, so we use the filesize of the expected result.
Diffstat (limited to 'ftparchive')
0 files changed, 0 insertions, 0 deletions