summaryrefslogtreecommitdiff
path: root/apt-pkg/deb
AgeCommit message (Collapse)Author
2016-01-08keep compressed indexes in a low-cost formatDavid Kalnischkies
Downloading and storing are two different operations were different compression types can be preferred. For downloading we provide the choice via Acquire::CompressionTypes::Order as there is a choice to be made between download size and speed – and limited by whats available in the repository. Storage on the other hand has all compressions currently supported by apt available and to reduce runtime of tools accessing these files the compression type should be a low-cost format in terms of decompression. apt traditionally stores its indexes uncompressed on disk, but has options to keep them compressed. Now that apt downloads additional files we also deal with files which simply can't be stored uncompressed as they are just too big (like Contents for apt-file). Traditionally they are downloaded in a low-cost format (gz) as repositories do not provide other formats, but there might be even lower-cost formats and for download we could introduce higher-cost in the repositories. Downloading an entire index potentially requires recompression to another format, so an update takes potentially longer – but big files are usually updated via pdiffs which has to de- and re-compress anyhow and does it on the fly anyhow, so there is no extra time needed and in general it seems to be benefitial to invest the time in update to save time later on file access.
2016-01-08AvailableDescriptionLanguages: Use one string for all iterationsJulian Andres Klode
Do not create strings within the loop, that creates one string per language and does more work than needed. Instead, reserve enough space at the beginning and assign the prefix, and then resize and append inside the loop. Also call exists with the string itself instead of the c_str(), this means that the lookup uses the size information in the string now and does not have to call strlen() on it.
2016-01-08Replace compare() == 0 checks with this == other checksJulian Andres Klode
This improves performance, as we now can ignore unequal strings based on their length already. Gbp-Dch: ignore
2016-01-07Switch performance critical code to use APT::StringViewJulian Andres Klode
This improves performance of the cache generation on my ARM platform (4x Cortex A15) by about 10% to 20% from 2.35-2.50 to 2.1 seconds.
2016-01-02support comments in debian/control parsingDavid Kalnischkies
Now (55153bf94ff28a23318e79aa48242244c4d82b3c) that pkgTagFile can be told to deal with all sorts of comments we can use this mode to parse dsc (as by catch) and debian/control files properly even in the wake of multiline fields spliced with comments like Build-Depends. Closes: 806775
2015-12-27allow repositories to forbid arch:all for specific index targetsDavid Kalnischkies
Debian has a Packages file for arch:all already, but the arch:any files contain arch:all packages as well, so downloading it would be a total waste of resources. Getting this solved is on the list of things to do, but it is also the hardest part – for index targets like Contents the situation is much easier and less server/client implementations are involved so we might not want to stall them. A repository can now declare via: No-Support-for-Architecture-all: Packages that even if an arch:all Packages exists, it shouldn't be downloaded, so that support for Contents files can be added now. See also 1dd20368486820efb6ef4476ad739e967174bec4 for the implementation of downloading arch:all index targets, which this is limiting. The field uses the name of the target from the apt configuration for simplicity and is negative by design as this field is intended to be supported/needed only for a "short" time (one or two Debian releases). While this commit theoretically supports any target, its expected to only see "Packages" as a value in reality.
2015-12-27ParseDepends: Mark branches for build-dep parsing as unlikelyJulian Andres Klode
We do not see those branches at all during normal mode of operation (that is, during cache generation), so tell the compiler about it.
2015-12-27debListParser: Do not validate Description-md5 for correctness twiceJulian Andres Klode
The Set() method returns false if the input is no hex number, so simply use that.
2015-12-27debListParser: ParseDepends: Only query native arch if neededJulian Andres Klode
This makes the code parsing architecture lists slower, but on the other hand, improves the more generic case of reading dependencies from Packages files.
2015-12-27Convert most callers of isspace() to isspace_ascii()Julian Andres Klode
This converts all callers that read machine-generated data, callers that might work with user input are not converted.
2015-12-14show a more descriptive error for weak Release filesDavid Kalnischkies
If we can't work with the hashes we parsed from the Release file we display now an error message if the Release file includes only weak hashes instead of downloading the indexes and failing to verify them with "Hash Sum mismatch" even through the hashes didn't mismatch (they were just weak). If for some (unlikely) reason we have got weak hashes only for individual targets we will show a warning to this effect (again, befor downloading and failing the index itself). Closes: 806459
2015-12-11Convert package names from Packages files to lower caseJulian Andres Klode
dpkg does that when reading package files, so we should do the same. This only deals with parsing names from binary package paragraphs, it does not look at source package names and/or the list of binaries in a dsc file. Closes: #807012
2015-12-07Avoid overflow when summing up file sizesJulian Andres Klode
We need to pass 0llu instead of 0 as the init value, otherwise std::accumulate will calculate with ints. Reported-by: Raphaël Hertzog
2015-12-01require explicit paths to dsc/control as we do for deb filesDavid Kalnischkies
Otherwise a user is subject to unexpected content-injection depending on which directory she happens to start apt in. This also cleans up the code requiring less implementation details in build-dep which is always good. Technically, this is an ABI break as we override virtual methods, but that they weren't overridden was a mistake resulting in pure classes, which shouldn't be pure, so they were unusable – and as they are new in 1.1 nobody is using them yet (and hopefully ever as they are borderline implementation details). Closes: 806693
2015-11-27add messages to our deprecation warnings in libaptDavid Kalnischkies
Git-Dch: Ignore
2015-11-21review of new/changed translatable program stringsJustin B Rye
Reference mail: https://lists.debian.org/debian-l10n-english/2015/11/msg00006.html
2015-11-05apply various suggestions made by cppcheckDavid Kalnischkies
Reported-By: cppcheck Git-Dch: Ignore
2015-11-05do not use read() returned value unconditionallyDavid Kalnischkies
A slightly unlikely bug, but lets fix it while slightly reworking this whole function to be slightly saner to look at, even if still not good. Git-Dch: Ignore
2015-11-04wrap every unlink call to check for != /dev/nullDavid Kalnischkies
Unlinking /dev/null is bad, we shouldn't do that. Also, we should print at least a warning if we tried to unlink a file but didn't manage to pull it of (ignoring the case were the file is /dev/null or doesn't exist in the first place). This got triggered by a relatively unlikely to cause problem in pkgAcquire::Worker::PrepareFiles which would while temporary uncompressed files (which are set to keep compressed) figure out that to files are the same and prepare for sharing by deleting them. Bad move. That also shows why not printing a warning is a bad idea as this hide the error for in non-root test runs. Git-Dch: Ignore
2015-11-04ignore newlines in dpkg-deb control output for installing debsDavid Kalnischkies
Leading or trailing newlines can be confusing for our parser as it expects two newlines to start/stop a new stanza. To solve this the lines we wanna add are printed first, ignore any leading newlines and then add the stanza as provided by dpkg-deb with or without trailing newlines as the parser will look at the first stanza only anyway and removing trailing newlines is considerably harder to do. Closes: 802553
2015-11-04support arch:all data e.g. in separate Packages fileDavid Kalnischkies
Based on a discussion with Niels Thykier who asked for Contents-all this implements apt trying for all architecture dependent files to get a file for the architecture all, which is treated internally now as an official architecture which is always around (like native). This way arch:all data can be shared instead of duplicated for each architecture requiring the user to download the same information again and again. There is one problem however: In Debian there is already a binary-all/ Packages file, but the binary-any files still include arch:all packages, so that downloading this file now would be a waste of time, bandwidth and diskspace. We therefore need a way to decide if it makes sense to download the all file for Packages in Debian or not. The obvious answer would be a special flag in the Release file indicating this, which would need to default to 'no' and every reasonable repository would override it to 'yes' in a few years time, but the flag would be there "forever". Looking closer at a Release file we see the field "Architectures", which doesn't include 'all' at the moment. With the idea outlined above that 'all' is a "proper" architecture now, we interpret this field as being authoritative in declaring which architectures are supported by this repository. If it says 'all', apt will try to get all, if not it will be skipped. This gives us another interesting feature: If I configure a source to download armel and mips, but it declares it supports only armel apt will now print a notice saying as much. Previously this was a very cryptic failure. If on the other hand the repository supports mips, too, but for some reason doesn't ship mips packages at the moment, this 'missing' file is silently ignored (= that is the same as the repository including an empty file). The Architectures field isn't mandatory through, so if it isn't there, we assume that every architecture is supported by this repository, which skips the arch:all if not listed in the release file.
2015-11-04sanify API to get 'the' candidate versionDavid Kalnischkies
This was discussed a while ago on #debian-apt and now that I see myself making this mistake lets bite the bullet and fix it in the easy way out version: Using a new name which fits with a similar named setter and deprecate the old method instead of 'hostily' changing API. Closes: #803471
2015-11-04show progress info while 'downloading' a local .deb fileDavid Kalnischkies
Showing just "Get: [1234 B]" looks very strange, so we now print the filename and as usual the package name, version and architecture.
2015-11-04provide public interface to hold/unhold packagesDavid Kalnischkies
We had this code lying around in apt-mark for a while now, but other frontends need this (and similar) functionality as well, so its high time that we provide a public interface in libapt for this stuff.
2015-11-04refactor dpkg execution in deb/debsystemDavid Kalnischkies
We have a few places and there will be a few more still where we have to call dpkg to detect/set certain features or settings. Centralizing the calling infrastructure now seems like a good idea before we add another.
2015-11-04implement a public pkgSystem::ArchitecturesSupportedDavid Kalnischkies
2015-11-04implement a public pkgSystem::MultiArchSupportedDavid Kalnischkies
Some codepaths need to check if the system (in our case usually dpkg) supports MultiArch or not. We had copy-pasted the check so far into these paths, but having it as a system check is better for reusability.
2015-10-30dpkgpm: Convert users of localtime() to localtime_r()Julian Andres Klode
The former is not thread-safe, whereas the latter is. Gbp-Dch: ignore
2015-10-30dpkgpm: Use ptsname_r() instead of ptsname() to be thread-safeJulian Andres Klode
This function only exists on a limited number of platforms, so we add a configure check to make sure it exists. Gbp-Dch: ignore
2015-10-30dpkgpm: Convert ctime() call to ctime_r()Julian Andres Klode
ctime() is not thread-safe, ctime_r() is. Gbp-Dch: ignore
2015-10-23deblistparser: Make PrioList constJulian Andres Klode
More safety, less writeable memory.
2015-09-14add by-hash sources.list option and document all of by-hashDavid Kalnischkies
This changes the semantics of the option (which is renamed too) to be a yes/no value with the special additional value "force" as this allows by-hash to be disabled even if the repository indicates it would be supported and is more in line with our other yes/no options like pdiff which disable themselves if no support can be detected. The feature wasn't documented so far and hasn't reached a (un)stable release yet, so changing it without trying too hard to keep compatibility seems okay.
2015-09-14avoid using global PendingError to avoid failing too often too soonDavid Kalnischkies
Our error reporting is historically grown into some kind of mess. A while ago I implemented stacking for the global error which is used in this commit now to wrap calls to functions which do not report (all) errors via return, so that only failures in those calls cause a failure to propergate down the chain rather than failing if anything (potentially totally unrelated) has failed at some point in the past. This way we can avoid stopping the entire acquire process just because a single source produced an error for example. It also means that after the acquire process the cache is generated – even if the acquire process had failures – as we still have the old good data around we can and should generate a cache for (again). There are probably more instances of this hiding, but all these looked like the easiest to work with and fix with reasonable (aka net-positive) effects.
2015-09-14implement dpkgs vision of interpreting pkg:<arch> dependenciesDavid Kalnischkies
How the Multi-Arch field and pkg:<arch> dependencies interact was discussed at DebConf15 in the "MultiArch BoF". dpkg and apt (among other tools like dose) had a different interpretation in certain scenarios which we resolved by agreeing on dpkg view – and this commit realizes this agreement in code. As was the case so far libapt sticks to the idea of trying to hide MultiArch as much as possible from individual frontends and instead translates it to good old SingleArch. There are certainly situations which can be improved in frontends if they know that MultiArch is upon them, but these are improvements – not necessary changes needed to unbreak a frontend. The implementation idea is simple: If we parse a dependency on foo:amd64 the dependency is formed on a package 'foo:amd64' of arch 'any'. This package is provided by package 'foo' of arch 'amd64', but not by 'foo' of arch 'i386'. Both of those foo packages provide each other through (assuming foo is M-A:foreign) to allow a dependency on 'foo' to be satisfied by either foo of amd64 or i386. Packages can also declare to provide 'foo:amd64' which is translated to providing 'foo:amd64:any' as well. This indirection over provides was chosen as the alternative would be to teach dependency resolvers how to deal with architecture specific dependencies – which violates the design idea of avoiding resolver changes, especially as architecture-specific dependencies are a cornercase with quite a few subtil rules. Handling it all over versioned provides as we already did for M-A in general seems much simpler as it just works for them. This switch to :any has actually a "surprising" benefit as well: Even frontends showing a package name via .Name() [which doesn't show the architecture] will display the "architecture" for dependencies in which it was explicitely requested, while we will not show the 'strange' :any arch in FullName(true) [= pretty-print] either. Before you had to specialcase these and by default you wouldn't get these details shown. The only identifiable disadvantage is that this complicates error reporting and handling. apt-get's ShowBroken has existing problems with virtual packages [it just shows the name without any reason], so that has to be worked on eventually. The other case is that detecting if a package is completely unknown or if it was at least referenced somewhere needs to acount for this "split" – not that it makes a practical difference which error is shown… but its one of the improvements possible.
2015-09-14M-A: allowed pkgs of unconfigured archs do not statisfy :anyDavid Kalnischkies
We parse all architectures we encounter recently, which means we also parse packages from architectures which are neither native nor foreign, but still came onto the system somehow (usually via heavy force).
2015-09-14store ':any' pseudo-packages with 'any' as architectureDavid Kalnischkies
Previously we had python:any:amd64, python:any:i386, … in the cache and the dependencies of an amd64 package would be on python:any:amd64, of an i386 on python:any:i386 and so on. That seems like a relatively pointless endeavor given that they will all be provided by the same packages and therefore also a waste of space. Git-Dch: Ignore
2015-08-31fix some unused parameter/variable warningsDavid Kalnischkies
Reported-By: gcc Git-Dch: Ignore
2015-08-30detect and deal with indextarget duplicatesDavid Kalnischkies
Multiple targets downloading the same file is bad™ as it leads us to all sorts of problems like the acquire system breaking or simply a problem of which settings to use for them. Beside that this is most likely a mistake and silently ignoring it doesn't help the user realizing his mistake… On the other hand, we have 'duplicates' which are 'created' by how we create indextargets, so we have to prevent those from being created to but do not emit a warning for them as this is an implementation detail. And then, there is the absolute and most likely user mistake: Having the same target(s) activated in multiple entries.
2015-08-30implement $(NATIVE_ARCHITECTURE) substvar for indextargetsDavid Kalnischkies
2015-08-29implement indextargets option 'DefaultEnabled'David Kalnischkies
Some targets like Contents-udeb are special-needs targets. Shipping the configuration snippet for them is okay, but they shouldn't be downloaded by default. Forcing the user to enable targets by uncommenting targets is wrong and this would still not really solve the problem completely as even if you want to download some -udebs it will probably not be for all sources you have enabled, so having the possibility of disabling a target by default, but giving the user the option to enable it on a per-source entry basis is better.
2015-08-29use c++11 algorithms to avoid strange compiler warningsDavid Kalnischkies
Nobody knows what makes the 'unable to optimize loop' warning to appear in the sourceslist minus-options parsing, especially if we use a foreach loop, but we can replace it with some nice c++11 algorithm+lambda usage, which also helps in making even clearer what happens here. And as this would be a lonely change, lets do it for a few more loops as well where I might or might not have seen the warning at some point in time, too. Git-Dch: Ignore
2015-08-28implement PDiff patching for compressed filesDavid Kalnischkies
Some additional files like 'Contents' are very big and should therefore kept compressed on the disk, which apt-file did in the past. It also implemented pdiff patching of these files by un- and recompressing these files on-the-fly, with this commit we can do the same – but we can do this in both pdiff patching styles (client and server merging) and secured by hashes. Hashes are in so far slightly complicated as we can't compare the hashes of the compressed files as we might compress them differently than the server would (different compressor versions, options, …), so we must compare the hashes of the uncompressed content. While this commit has changes in public headers, the classes it changes are marked as hidden, so nobody can use them directly, which means the ABI break is internal only.
2015-08-27sources.list and indextargets option for pdiffsDavid Kalnischkies
Disabling pdiffs can be useful occasionally, like if you have a fast local mirror where the download doesn't matter, but still want to use it for non-local mirrors. Also, some users might prefer it to only use it for very big indextargets like Contents.
2015-08-27Do not parse Status fields from remote sourcesJulian Andres Klode
This could allow an attacker to mark a package as installed in a remote package index, as long as the package was not listed in the dpkg status file. This way, an attacker could force the installation of a package during a dist-upgrade, by providing two packages in an index, an older marked as installed, and a newer - apt would "upgrade" to the newer version.
2015-08-27allow explicit dis/enable of IndexTargets in sources optionsDavid Kalnischkies
While Target{,-Add,-Remove} is available for configuring IndexTargets already, allow Targets to be mentioned explicitely as yes/no options as well, so that the Target 'Contents' can be disabled via 'Contents: no' as well as 'Target-Remove: Contents'.
2015-08-27not all targets are deb-src targetsDavid Kalnischkies
Sometimes too much refactoring can have bad effects. Thanks: Niels Thykier for reporting on IRC Git-Dch: Ignore
2015-08-17Fix all the wrong removals of includes that iwyu got wrongMichael Vogt
Git-Dch: ignore
2015-08-17Cleanup includes after running iwyuMichael Vogt
2015-08-12Do not set unhonored DPKG_NO_TSTP variable for dpkgGuillem Jover
Support for that variable was removed in dpkg in 1.15.6, in commit 6f037003e8b96878b485efb7cbd1f846e3bf4e97. Closes: #765366
2015-08-11Annotate more methods with APT_OVERRIDEJulian Andres Klode
Gbp-Dch: ignore Reported-By: g++ -Wsuggest-override Thanks: g++ -Wsuggest-override