summaryrefslogtreecommitdiff
path: root/test/integration
AgeCommit message (Collapse)Author
2015-08-31do delay the test for http, too, to make it more reliableDavid Kalnischkies
The file method was already slowed down and somehow I thought I had done the same for http, but it turns out that I didn't. Giving it the same delay as file should help in making this test slower and therefore more likely to successfully test what it is supposed to test. Git-Dch: Ignore
2015-08-31if file is inaccessible for _apt, disable privilege drop in acquireDavid Kalnischkies
We had a very similar method previously for our own private usage, but with some generalisation we can move this check into the acquire system proper so that all frontends profit from this compatibility change. As we are disabling a security feature here a warning is issued and frontends are advised to consider reworking their download logic if possible. Note that this is implemented as an all or nothing situation: We can't just (not) drop privileges for a subset of the files in a fetcher, so in case you have to download some files with and some without you need to use two fetchers.
2015-08-30detect and deal with indextarget duplicatesDavid Kalnischkies
Multiple targets downloading the same file is bad™ as it leads us to all sorts of problems like the acquire system breaking or simply a problem of which settings to use for them. Beside that this is most likely a mistake and silently ignoring it doesn't help the user realizing his mistake… On the other hand, we have 'duplicates' which are 'created' by how we create indextargets, so we have to prevent those from being created to but do not emit a warning for them as this is an implementation detail. And then, there is the absolute and most likely user mistake: Having the same target(s) activated in multiple entries.
2015-08-29try xz instead of bz2 first for compressed filesDavid Kalnischkies
xz has pretty much won "the compressor war" and e.g. the Debian archive doesn't even distribute bz2 anymore in favor of 'xz' and 'gz', so by changing the default order we have a more realistic --print-uris behavior as it will always show the first compressor. In practice this effects repositories without a Release file (very bad, we don't want to support them anymore anyhow) as xz will be tried before bz2 now [which is probably not available, but so might be bz2…] AND repositories which provide both, bz2 and xz (which isn't too common) in sofar as apt will now download xz instead of bz2. Users with special needs can stick with bz2 as first compressor tried with Acquire::CompressionTypes::Order:: "bz2"; (see man apt.conf) – but users with special needs usually prefer "gz" anyhow, so the realworld change is expected to be very low.
2015-08-29implement indextargets option 'DefaultEnabled'David Kalnischkies
Some targets like Contents-udeb are special-needs targets. Shipping the configuration snippet for them is okay, but they shouldn't be downloaded by default. Forcing the user to enable targets by uncommenting targets is wrong and this would still not really solve the problem completely as even if you want to download some -udebs it will probably not be for all sources you have enabled, so having the possibility of disabling a target by default, but giving the user the option to enable it on a per-source entry basis is better.
2015-08-28implement PDiff patching for compressed filesDavid Kalnischkies
Some additional files like 'Contents' are very big and should therefore kept compressed on the disk, which apt-file did in the past. It also implemented pdiff patching of these files by un- and recompressing these files on-the-fly, with this commit we can do the same – but we can do this in both pdiff patching styles (client and server merging) and secured by hashes. Hashes are in so far slightly complicated as we can't compare the hashes of the compressed files as we might compress them differently than the server would (different compressor versions, options, …), so we must compare the hashes of the uncompressed content. While this commit has changes in public headers, the classes it changes are marked as hidden, so nobody can use them directly, which means the ABI break is internal only.
2015-08-27Fix test-security-no-remote-statusJulian Andres Klode
Gbp-Dch: ignore
2015-08-27Do not parse Status fields from remote sourcesJulian Andres Klode
This could allow an attacker to mark a package as installed in a remote package index, as long as the package was not listed in the dpkg status file. This way, an attacker could force the installation of a package during a dist-upgrade, by providing two packages in an index, an older marked as installed, and a newer - apt would "upgrade" to the newer version.
2015-08-27install: Set a local deb as the candidate for that packageJulian Andres Klode
This ensures that we can install .deb files that are not the candidate for a given package.
2015-08-27allow explicit dis/enable of IndexTargets in sources optionsDavid Kalnischkies
While Target{,-Add,-Remove} is available for configuring IndexTargets already, allow Targets to be mentioned explicitely as yes/no options as well, so that the Target 'Contents' can be disabled via 'Contents: no' as well as 'Target-Remove: Contents'.
2015-08-27use always priv-dropping for changelog download as rootDavid Kalnischkies
First of, the temporary directory we download the changelog to needs to be owned by _apt, but that also means that we don't need to check if we could/should drop privs as the download happens to a dedicated tempdir and only after that it is moved to its final location by a privileged user.
2015-08-27fix various typos reported by codespellDavid Kalnischkies
Reported-By: codespell
2015-08-27just-in-time removal of broken essential packagesDavid Kalnischkies
We deal with Conflicts in SmartUnpack in pretty much the same way, but Breaks weren't handled in SmartConfigure so that the remove was sheduled after the configuration of the package breaking the to-be-removed. Closes: 796070
2015-08-27test for #796070: downgrades involving essential packagesJulian Andres Klode
Git-Dch: Ignore
2015-08-27tests: check apt-get source release name switching with -tDavid Kalnischkies
Git-Dch: Ignore
2015-08-19Support tabs in sources.list filesJulian Andres Klode
Also support vertical tabs, as isspace() does the same. Closes: #796067
2015-08-18Add support for "apt-cache showsrc --only-source srcpkgname"Michael Vogt
Thanks: Steve Slangasek for the suggestion Closes: 695633
2015-08-17Fix the test suite harderJulian Andres Klode
Gbp-Dch: ignore
2015-08-17Fix the test suite againJulian Andres Klode
Gbp-Dch: ignore
2015-08-14Add integration test for Pin-Priority range checksJulian Andres Klode
Gbp-Dch: ignore
2015-08-14Replace --force-yes by various options starting with --allowJulian Andres Klode
This enables more fine grained control over such exceptions.
2015-08-14Merge branch 'debian/experimental' of https://github.com/DonKult/apt into ↵Julian Andres Klode
debian/experimental
2015-08-13Remove an invalid fi from a testcaseJulian Andres Klode
Gbp-Dch: ignore
2015-08-13Fix integration tests for the removal of the Package pin outputJulian Andres Klode
This should make them work again.
2015-08-12Replace all "press enter" occurrences with "press [Enter]"Luca Bruno
Thanks: Andre Felipe Machado for initial patch Closes: 414848
2015-08-12replace direct calls to egrep with grep -EDaniel Hartwig
The rest of the initial patch is not needed or incorrect in our usage. Big changes for the dselect scripts seem unneeded as well as those are hardly used by anyone anymore… [commit message written by commiter] Closes: 255577 Thanks: David Weinehall for initial patch
2015-08-12travis: add ppa:ubuntu-toolschain-r/test as source for gcc-5David Kalnischkies
This makes travis-ci able to run our tests again. Sometimes. If it doesn't spontaneously fails with internal gcc errors… Git-Dch: Ignore
2015-08-10move manual-bit from 'oldlibs' pkg to its dependenciesDavid Kalnischkies
oldlibs used to be in APT::Never-MarkAuto-Sections so that old transition packages can be removed without causing the then (autoinstalled) renamed package to be autoremoved. It isn't ideal through as ideally you want the oldlibs package to be removed after nothing depends on it anymore regardless of if you have once installed it by hand or not – and if you had the package talking over (the dependencies) should carry the manual bit now as they are the real deal now. As an added bonus if the package has no dependencies because it is an oldlibs without a direct replacement you should move away from (like lib1 and lib2 are currently in the archive, but there will hopefully only be lib2 in the release) you get a lib1 marked as auto. If the user still needs the oldlibs package for some reason all he has to do is mark it as manual once as this move is only performed if a installed package changes its section from a not-Move-Autobit-Sections to a Move-Autobit-Sections.
2015-08-10move APT::Never-MarkAuto-Sections handling to MarkDeleteDavid Kalnischkies
Having the handling in MarkInstall means that it just effects installation of the metapackage, but if the dependencies change the new dependencies aren't protected (and the old dependencies are still protected for no 'reason'). Having it in MarkDelete means that if a metapackage is sheduled for removal all its currently installed dependencies are marked as manual, which helps against both as in this case there is no new/old and additionally if a user decides the installation of a metapackage was wrong he can just remove it explicitely avoid the manual marking entirely.
2015-08-10no value for MultiArch field is 'no', not 'none'David Kalnischkies
Git-Dch: Ignore
2015-08-10mark again deps of pkgs in APT::Never-MarkAuto-Sections as manualDavid Kalnischkies
In 50ef3344c3afaaf9943142906b2f976a0337d264 (and similar for other branches), while 'fixing' the edgecase of a package being in multiple sections (e.g. moved from libs to oldlibs in newer releases) I accidently broke the feature itself completely by operating on the package itself and no longer on its dependencies… The behaviour isn't ideal in multiple ways, which we are hopefully able to fix with new ideas as mentioned in the buglog, but until then the functionality of this "hack" should be restored. Reported-By: Raphaël Hertzog <hertzog@debian.org> Tested-By: Adam Conrad <adconrad@ubuntu.com> Closes: 793360 LP: 1479207 Thanks: Raphaël Hertzog and Adam Conrad for detailed reports and initial patches
2015-08-10drop extra newline in 'Failed to fetch' and 'GPG error' messageDavid Kalnischkies
I never understood why there is an extra newline in those messages, so now is as good time as any to drop them. Lets see if someone complains with a good reason to keep it…
2015-08-10enhance "hit paywall" error message to mention the probable causeDavid Kalnischkies
Reporting errors from Done() is bad for progress reporting and such, so factoring this out is a good idea and we start with moving the supposed- to-be clearsigned file isn't clearsigned out first – improving the error message in the process as we use the same message for a similar case (NODATA) as this is what I have to look at with the venue wifi at DebCamp and the old errormessage doesn't really say anything.
2015-08-10parse packages from all architectures into the cacheDavid Kalnischkies
Now that we can dynamically create dependencies and provides as needed rather than requiring to know with which architectures we will deal before running we can allow the listparser to parse all records rather than skipping records of "unknown" architectures. This can e.g. happen if a user has foreign architecture packages in his status file without dpkg knowing about this architecture (or apt configured in this way). A sideeffect is that now arch:all packages are (correctly) recorded as available from any Packages file, not just from the native one – which has its downsides for the resolver as mixed-arch source packages can appear in different architectures at different times, but that is the problem of the resolver and dealing with it in the parser is at best a hack (and also depends on a helpful repository). Another sideeffect is that his allows :none packages to appear in Packages files again as we don't do any kind of checks now, but given that they aren't really supported (anymore) by anyone we can live with that.
2015-08-10elimate duplicated code in pkgIndexFile subclassesDavid Kalnischkies
Trade deduplication of code for a bunch of new virtuals, so it is actually visible how the different indexes behave cleaning up the interface at large in the process. Git-Dch: Ignore
2015-08-10add volatile sources support in libapt-pkgDavid Kalnischkies
Sources are usually defined in sources.list (and co) and are pretty stable, but once in a while a frontend might want to add an additional "source" like a local .deb file to install this package (No support for 'real' sources being added this way as this is a multistep process). We had a hack in place to allow apt-get and apt to pull this of for a short while now, but other frontends are either left in the cold by this and/or the code for it looks dirty with FIXMEs plastering it and has on top of this also some problems (like including these 'volatile' sources in the srcpkgcache.bin file). So the biggest part in this commit is actually the rewrite of the cache generation as it is now potentially a three step process. The biggest problem with adding support now through is that this makes a bunch of previously mostly unusable by externs and therefore hidden classes public, so a bit of further tuneing on this now public API is in order…
2015-08-10just-in-time creation for (explicit) negative depsDavid Kalnischkies
Now that we deal with provides in a more dynamic fashion the last remaining problem is explicit dependencies like 'Conflicts: foo' which have to apply to all architectures, but creating them all at the same time requires us to know all architectures ending up in the cache which isn't needed to be the same set as all foreign architectures. The effect is visible already now through as this prevents the creation of a bunch of virtual packages for arch:all packages and as such also many dependencies, just not very visible if you don't look at the stats… Git-Dch Ignore
2015-08-10just-in-time creation for (implicit) ProvidesDavid Kalnischkies
Expecting the worst is easy to code, but has its disadvantages e.g. by creating package structures which otherwise would have never existed. By creating the provides instead at the time a package structure is added we are well prepared for the introduction of partial architectures, massive amounts of M-A:foreign (and :allowed) and co as far as provides are concerned at least. We have something relatively similar for dependencies already. Many tests are added for both M-A states and the code cleaned to properly support implicit provides for foreign architectures and architectures we 'just' happen to parse. Git-Dch: Ignore
2015-08-10hide implicit deps in apt-cache again by defaultDavid Kalnischkies
Before MultiArch implicits weren't a thing, so they were hidden by default by definition. Adding them for MultiArch solved many problems, but having no reliable way of detecting which dependency (and provides) is implicit or not causes problems everytime we want to output dependencies without confusing our observers with unneeded implementation details. The really notworthy point here is actually that we keep now a better record of how a dependency came to be so that we can later reason about it more easily, but that is hidden so deep down in the library internals that change is more the problems it solves than the change itself.
2015-08-10link DependencyData structs togetherDavid Kalnischkies
Cache generation needs a way of quickly iterating over the unique potion of the dependencies to be able to share them. By linking them together we can reduce the speed penality (~ 80%) with only a small reduction in saved size (~ 20%). Git-Dch: Ignore
2015-08-10bunch of micro-optimizations for depcacheDavid Kalnischkies
DepCache functions are called a lot, so if we can squeeze some drops out of them for free we should do so. Takes also the opportunity to remove some whitespace errors from these functions. Git-Dch: Ignore
2015-08-10show or-groups in not-installed recommends and suggests listsDavid Kalnischkies
Further abstracting our new ShowList allows to use it for containers of strings as well giving us the option to implement an or-groups display for the recommends and suggests lists which is a nice trick given that it also helps with migrating the last remaining other cases of old ShowList.
2015-08-10implement a more generic ShowList methodDavid Kalnischkies
apt-get is displaying various lists of package names, which until now it was building as a string before passing it to ShowList, which inserted linebreaks at fitting points and showed a title if needed, but it never really understood what it was working with. With the help of C++11 the new generic knows not only what it works with, but generates the list on the fly rather than asking for it and potentially discarding parts of the input (= the non-default verbose display). It also doubles as a test for how usable the CacheSets are with C++11. (Not all callers are adapted yet.) Git-Dch: Ignore
2015-08-10rename 'apt-get files' to 'apt-get indextargets'David Kalnischkies
'files' is a bit too generic as a name for a command usually only used programmatically (if at all) by developers, so instead of "wasting" this generic name for this we use "indextargets" which is actually the name of the datastructure the displayed data is stored in. Along with this rename the config options are renamed accordingly.
2015-08-10handle site-changing redirects as mirror changesDavid Kalnischkies
Redirectors like httpredir.debian.org orchestra the download from multiple (hopefully close) mirrors while having only a single central sources.list entry by using redirects. This has the effect that the progress report always shows the source it started with, not the mirror it ends up fetching from, which is especially problematic for error reporting as having a report for a "Hashsum mismatch" for the redirector URI is next to useless as nobody knows which URI it was really fetched from (regardless of it coming from a user or via the report script) from this output alone. You would need to enable debug output and hope for the same situation to arise again… We hence reuse the UsedMirror field of the mirror:// method and detect redirects which change the site and declare this new site as the UsedMirrror (and adapt the description). The disadvantage is that there is no obvious mapping anymore (it is relatively easy to guess through with some experience) from progress lines to sources.list lines, so error messages need to take care to use the Target description (rather than current Item description) if they want to refer to the sources.list entry.
2015-08-10implement Signed-By without using gpg for verificationDavid Kalnischkies
The previous commit returns to the possibility of using just gpgv for verification proposes. There is one problem through: We can't enforce a specific keyid without using gpg, but our acquire method can as it parses gpgv output anyway, so it can deal with good signatures from not expected signatures and treats them as unknown keys instead. Git-Dch: Ignore
2015-08-10merge keyrings with cat instead of gpg in apt-keyDavid Kalnischkies
If all keyrings are simple keyrings we can merge the keyrings with cat rather than doing a detour over gpg --export | --import (see #790665), which means 'apt-key verify' can do without gpg and just use gpgv as before the merging change. We declare this gpgv usage explicit now in the dependencies. This isn't a new dependency as gnupg as well as debian-archive-keyring depend on and we used it before unconditionally, just that we didn't declare it. The handling of the merged keyring needs to be slightly different as our merged keyring can end up containing the same key multiple times, but at least currently gpg does remove only the first occurrence with --delete-keys, so we move the handling to a if one is gone, all are gone rather than an (implicit) quid pro quo or even no effect. Thanks: Daniel Kahn Gillmor for the suggestion
2015-08-10support gpg 2.1.x in apt-keyDavid Kalnischkies
The output of gpg slightly changes in 2.1 which breaks the testcase, but the real problem is that this branch introduces a new default keyring format (which is called keybox) and mixing it with simple keyrings (the previous default format) has various problems like failing in the keybox to keyring import (#790665) or [older] gpgv versions not being able to deal with keyboxes (and newer versions as well currently: https://bugs.gnupg.org/gnupg/issue2025). We fix this by being a bit more careful in who creates keyrings (aka: we do it or we take a simple keyring as base) to ensure we always have a keyring instead of a keybox. This way we can ensure that any version combination of gpv/gpgv2 and gnupg/gnupg2 without doing explicit version checks and use the same code for all of them. Closes: 781042
2015-08-10allow individual targets to be kept compressedDavid Kalnischkies
There is an option to keep all targets (Packages, Sources, …) compressed for a while now, but the all-or-nothing approach is a bit limited for our purposes with additional targets as some of them are very big (Contents) and rarely used in comparison, so keeping them compressed by default can make sense, while others are still unpacked. Most interesting is the copy-change maybe: Copy is used by the acquire system as an uncompressor and it is hence expected that it returns the hashes for the "output", not the input. Now, in the case of keeping a file compressed, the output is never written to disk, but generated in memory and we should still validated it, so for compressed files copy is expected to return the hashes of the uncompressed file. We used to use the config option to enable on-the-fly decompress in the method, but in reality copy is never used in a way where it shouldn't decompress a compressed file to get its hashes, so we can save us the trouble of sending this information to the method and just do it always.
2015-08-10implement Signed-By option for sources.listDavid Kalnischkies
Limits which key(s) can be used to sign a repository. Not immensely useful from a security perspective all by itself, but if the user has additional measures in place to confine a repository (like pinning) an attacker who gets the key for such a repository is limited to its potential and can't use the key to sign its attacks for an other (maybe less limited) repository… (yes, this is as weak as it sounds, but having the capability might come in handy for implementing other stuff later).