summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2016-01-08support '-' and no parameter for stdin in apt-helper cat-fileDavid Kalnischkies
This way it works more similar to the compressor binaries, which we can relief in this way from their job in the test framework avoiding the need of adding e.g. liblz4-tool to the test dependencies.
2016-01-08keep compressed indexes in a low-cost formatDavid Kalnischkies
Downloading and storing are two different operations were different compression types can be preferred. For downloading we provide the choice via Acquire::CompressionTypes::Order as there is a choice to be made between download size and speed – and limited by whats available in the repository. Storage on the other hand has all compressions currently supported by apt available and to reduce runtime of tools accessing these files the compression type should be a low-cost format in terms of decompression. apt traditionally stores its indexes uncompressed on disk, but has options to keep them compressed. Now that apt downloads additional files we also deal with files which simply can't be stored uncompressed as they are just too big (like Contents for apt-file). Traditionally they are downloaded in a low-cost format (gz) as repositories do not provide other formats, but there might be even lower-cost formats and for download we could introduce higher-cost in the repositories. Downloading an entire index potentially requires recompression to another format, so an update takes potentially longer – but big files are usually updated via pdiffs which has to de- and re-compress anyhow and does it on the fly anyhow, so there is no extra time needed and in general it seems to be benefitial to invest the time in update to save time later on file access.
2016-01-08tests: try to pick up compressors from config automaticallyDavid Kalnischkies
Less hardcoding should help while introducing new compressors. Git-Dch: Ignore
2016-01-08allow pdiff bootstrap from all supported compressorsDavid Kalnischkies
There is no reason to enforce that the file we start the bootstrap with is compressed with a compressor which is available online. This allows us to change the on-disk format as well as deals with repositories adding/removing support for a specific compressor.
2016-01-08ensure compression cleanup even without lists-cleanupDavid Kalnischkies
If we store files compressed in lists/ and the file switched compression formats we happened to retain the "old" format, but by default the cleanup process catched this oversight and removed the file. [The initial situation described doesn't arise as we store no files by default compressed and even with apt-file configuring Contents files, we don't really have that problem as there is just .gz files for those.] We solve this by just removing any uncompressed as well as compressed (we support) file just before we move the 'new' version of the file in.
2016-01-08use one 'store' method to rule all (de)compressorsDavid Kalnischkies
Adding a new compressor method meant adding a new method as well – even if that boilt down to just linking to our generalized decompressor with a new name. That is unneeded busywork if we can instead just call the generalized decompressor and let it figure out which compressor to use based on the filenames rather than by program name. For compatibility we ship still 'gzip', 'bzip2' and co, but they are just links to our "new" 'store' method.
2016-01-08include all compressed Packages/Sources files in Release fileDavid Kalnischkies
Having a hardcoded list of compression types here doesn't really provide us with anything beside added complexity each time someone adds a new compression type. That we don't need to be that specific is evident by Contents and Translation-* matchers which are a lot more generic and didn't generate problems anyhow.
2016-01-08fix typo in SrvRecords documentationDavid Kalnischkies
Git-Dch: Ignore
2016-01-08AvailableDescriptionLanguages: Use one string for all iterationsJulian Andres Klode
Do not create strings within the loop, that creates one string per language and does more work than needed. Instead, reserve enough space at the beginning and assign the prefix, and then resize and append inside the loop. Also call exists with the string itself instead of the c_str(), this means that the lookup uses the size information in the string now and does not have to call strlen() on it.
2016-01-08pkgCacheGenerator: CurMd5.Value() cannot be emptyJulian Andres Klode
It makes no sense to check if the value is empty, as it cannot be. It will always be a hexstring of exactly 32 bytes.
2016-01-08operator==(char*, StringView) use StringView.operator==Julian Andres Klode
Use the same path for both comparisons, as the operator== path is faster than just calling compare() - it avoids any comparison if the size differs. Gbp-Dch: ignore
2016-01-08pkgCacheGenerator::hash: Do not call tolower_ascii()Julian Andres Klode
Gbp-Dch: ignore
2016-01-08pkgCacheGenerator::StoreString: Get rid of std::stringJulian Andres Klode
Instead of storing a string -> map_stringitem_t mapping, create our own data type that can point to either a normal string or a string inside the cache. This avoids the creation of any string and improves performance slightly (about 4%).
2016-01-08Replace compare() == 0 checks with this == other checksJulian Andres Klode
This improves performance, as we now can ignore unequal strings based on their length already. Gbp-Dch: ignore
2016-01-08pkgCacheGenerator: Use StringView for toStringJulian Andres Klode
This removes some minor overhead. Gbp-Dch: ignore
2016-01-08pkgCacheGenerator::StoreString: Move the string into the mapJulian Andres Klode
Moving the string is likely faster than copying it. We could probably avoid strings alltogether in the future using some more crazy code, but I have not looked at that yet. Gbp-Dch: ignore
2016-01-07StringView: rfind: pos should be end of substr, not startJulian Andres Klode
Gbp-Dch: ignore
2016-01-07StringView: pos argument default should be nposJulian Andres Klode
Gbp-Dch: ignore
2016-01-07StringView::rfind(): Call rfind() instead of find() on substJulian Andres Klode
Thanks: Niels Thykier for reporting on IRC Gbp-Dch: ignore
2016-01-07Switch performance critical code to use APT::StringViewJulian Andres Klode
This improves performance of the cache generation on my ARM platform (4x Cortex A15) by about 10% to 20% from 2.35-2.50 to 2.1 seconds.
2016-01-07Introduce internal APT::StringView classJulian Andres Klode
The class APT::StringView implements a drop-in replacement for a subset of C++17 std::string_view() features. It will be dropped at a later point and may not be used in public interfaces.
2016-01-07rred: Run in parallelJulian Andres Klode
Remove the SingleInstance flag so we can use the new randomized queue feature to run parallel.
2016-01-07acquire: Allow parallelizing methods without hostsJulian Andres Klode
The maximum parallelization soft limit is the number of CPU cores * 2 on systems defining _SC_NPROCESSORS_ONLN. The hard limit in all cases is Acquire::QueueHost::Limit.
2016-01-07CopyFile: Use 64 * 1024 instead of 64000 as buffer sizeJulian Andres Klode
This is a multiple of the page size and thus results in less page faults, speeding up copying. Also, while we're at at, unify all uses of that size in a constant variable APT_BUFFER_SIZE.
2016-01-07apt-helper: cat-file: Add -C/--compress optionJulian Andres Klode
This allows passing compressing the output. The compressor must be a compressor name, extension, or an extension without the leading dot.
2016-01-07FileFd: (native) LZ4 supportJulian Andres Klode
Implement native support for LZ4 compression, using the official lz4 library.
2016-01-07travis: pull liblz4-dev from wilyJulian Andres Klode
2016-01-05Do not remove a not working SrvRecords server twiceMichael Vogt
The PopFromSrvRecs() already removed the entry from the active list, so the extra SrvRecords.erase() was incorrect. Git-Dch: ignore
2016-01-05Add documentation about the SrvRecords support in aptMichael Vogt
Git-Dch: ignore
2016-01-05Document new APT::Keep-Downloaded-Packages option in NEWSMichael Vogt
Git-Dch: ignore
2016-01-03test-apt-cache: Adjust for hashtable size changeJulian Andres Klode
Gbp-Dch: ignore
2016-01-03Increase APT::Cache-HashTableSize default to 50503Julian Andres Klode
This drop the hash table utilization from a high 98% to acceptable 74% on unstable, and the average bucket length from 4.6 to 1.8. This improves performance by about 5%, while increasing the size of the cache by 0.2 out of 38MB, that is 0.5%. 48481 is a nice number
2016-01-03apt-cache: stats: Show a table utilization as percentageJulian Andres Klode
Gbp-Dch: ignore
2016-01-03apt-cache: stats: Average is over used, not all, bucketsJulian Andres Klode
It does not make sense to consider empty buckets in the average, as they do not affect the lookup performance.
2016-01-03Allow building without libgtest-dev under <nocheck> build profileHelmut Grohne
I'd like to avoid pulling libgtest-dev into the bootstrap set. Fortunately, libgtest-dev is only used for testing apt and apt correctly implements DEB_BUILD_OPTIONS=nocheck now. So this bug is about getting rid of the Build-Depends. Simply removing it (by adding a build profile) is not sufficient however as configure fails hard, so an additional bit is necessary to cover for that. Closes: #809726
2016-01-03Change compressor costs to be 100 apartJulian Andres Klode
This will give us the freedom to insert more compressors at positions in between. Also change the cost of uncompressed to 0, as that really has no overhead, and the values do not really mean much.
2016-01-03simple_compressor: Provide some accessors for end and freeJulian Andres Klode
This makes code easier to read, and somewhat more correct. Gbp-Dch: ignore
2016-01-03simple_buffer: Allow buffer size to changeJulian Andres Klode
Gbp-Dch: ignore
2016-01-03prepare-release: travis: Ignore build profiles in build-dependsJulian Andres Klode
This aids our introduction of lz4, which we do not want to impose as an additional requirement for bootstrappers. Gbp-Dch: ignore
2016-01-02Add new APT::Keep-Downloaded-Packages optionMichael Vogt
This option controls if downloaded packages should be kept after a successful install or if they should be deleted. The default for "apt-get" is that they are kept (just like before). However the default for "apt" is that they get deleted. Closes: #160743
2016-01-02Add bash-completion support for the "apt" commandMichael Vogt
The apt bash-completion support was submited to the bash-completion package as a patch in May 2014. It is still not included to this date and because it is an important feature for many users it is now part of apt until the bash-completion package is mantained more actively again. Note that the "Relaces" line is only required for Ubuntu it will have no effect on Debian. Closes: #747094
2016-01-02Add support for APT::Periodic::CleanIntervalMichael Vogt
In order to allow regular cleaning of the downloaded deb archives the new option APT::Periodic::CleanInterval is added. It will run "apt-get clean" in the given time interval. Thanks: Martin-Éric Racine
2016-01-02properly parse comments in apt_preferences and deb822-style sourcesDavid Kalnischkies
apt_preferences and deb822-style sources used the specialized class pkgUserTagSection to deal with comments before/after a given stanza, but it couldn't deal with comments in the stanza at all. codesearch suggests that nobody else does and a vastely superior way of working with potentially commented files is implemented now, so we can officially discourage the use of the old incomplete hack class.
2016-01-02support comments in debian/control parsingDavid Kalnischkies
Now (55153bf94ff28a23318e79aa48242244c4d82b3c) that pkgTagFile can be told to deal with all sorts of comments we can use this mode to parse dsc (as by catch) and debian/control files properly even in the wake of multiline fields spliced with comments like Build-Depends. Closes: 806775
2016-01-02add optional support for comments in pkgTagFileDavid Kalnischkies
APT usually deals with perfectly formatted files generated automatically be other programs – and as it has to parse multiple MBs of such files it tries to be fast rather than forgiving. This was always a problem if we reused this parser for files with a deb822 syntax which are mostly written by hand however, like apt_preferences or the deb822-style sources as these can include stray newlines and more importantly comments all over the place. As a stopgap we had pkgUserTagSection which deals at least with comments before and after a given stanza, but comments in between weren't really supported and now that we support parsing debian/control for e.g. build-dep we face the full comment problem e.g. with comments inbetween multi-line fields (like Build-Depends). We can't easily deal with this on the pkgTagSection level as the interface gives access to 'raw' char-pointers for performance reasons so we would need to optionally add a buffer here on which we could remove comments to hand out pointers into this buffer instead. The interface is quite large already and supports writing stanzas as well, which does not support comments at all either. So while in future it might make sense to have a parser setup which deals with and keeps comments in this commit we opt for the simpler solution for now: We officially declare that pkgTagSection does not support comments and instead expect the caller to deal with them, which in our case is pkgTagFile: pkgTagFile is extended with an additional mode which can deal with comments by dropping them from the buffer which will later form the input of pkgTagSection. The actual implementation is slightly more complex than this sentence suggests at first on one hand to have good performance and on the other to allow jumping directly to stanzas with offsets collected in a previous run (like our cache generation does it for example).
2016-01-02fail installing build-deps if parsing them failedDavid Kalnischkies
Git-Dch: Ignore
2016-01-02Italian manpages translation updateBeatrice Torracca
Closes: 809522
2015-12-30Release 1.1.101.1.10Julian Andres Klode
2015-12-29Do not sync the cache fileJulian Andres Klode
Integrity is taken care of by the checksum now.
2015-12-29Add support for calculating hashes over the entire cacheJulian Andres Klode