summaryrefslogtreecommitdiff
path: root/ftparchive
diff options
context:
space:
mode:
authorJulian Andres Klode <jak@debian.org>2016-06-15 23:13:43 +0200
committerJulian Andres Klode <jak@debian.org>2016-09-02 17:16:36 +0200
commit2a440328ea19e9646a93f847dd9eff21e03ad16d (patch)
treedb03a2391b2bf1ad59103a46e14c6a6c0a839c1d /ftparchive
parent544e1528b18025fad8318e6fb825ad296976cf24 (diff)
acquire: Use priority queues and a 3 stage pipeline design
Employ a priority queue instead of a normal queue to hold the items; and only add items to the running pipeline if their priority is the same or higher than the priority of items in the queue. The priorities are designed for a 3 stage pipeline system: In stage 1, all Release files and .diff/Index files are fetched. This allows us to determine what files remain to be fetched, and thus ensures a usable progress reporting. In stage 2, all Pdiff patches are fetched, so we can apply them in parallel with fetching other files in stage 3. In stage 3, all other files are fetched (complete index files such as Contents, Packages). Performance improvements, mainly from fetching the pdiff patches before complete files, so they can be applied in parallel: For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update of Debian unstable and testing with Contents and appstream for amd64 and i386, update time reduced from 37 seconds to 24-28 seconds. Previously, apt would first download new DEP11 icon tarballs and metadata files, causing the CPU to be idle. By fetching the diffs in stage 2, we can now patch our contents and Packages files while we are downloading the DEP11 stuff.
Diffstat (limited to 'ftparchive')
0 files changed, 0 insertions, 0 deletions