summaryrefslogtreecommitdiff
path: root/test/integration
AgeCommit message (Collapse)Author
2016-08-25treat .ddeb files like .deb, especially for dpkgDavid Kalnischkies
Ubuntu uses *.ddeb files for their debug packages, but the interface we are using since f495992428a396e0f98886c9a761a804aa161c68 to talk to dpkg isn't supporting *.ddeb files. This used to work previously as apt itself isn't caring about the filenames at all and if they are explicitly mentioned dpkg will accept all, too. It might or might not be a good idea to patch dpkg, too, but regardless of it happening, we don't want to couple us to closely to dpkg for this minor feature but testing for this at runtime as it would delay shipping the fix for the too long commandlines further. It is also questionable if it is really a good idea to allow any file extension to be used here (like .foobar in the testcase), but we used to and we tend to avoid breaking existing usecases if we can help it. As a bonus, this also allows the installation of ddeb files directly from the commandline as you can with deb files already. We continue to ignore udeb through as the user-mistake to useful ratio is too high. LP: #1616909
2016-08-25show apt-key warnings in apt updateDavid Kalnischkies
In 105503b4b470c124bc0c271bd8a50e25ecbe9133 we got a warning implemented for unreadable files which greatly improves the behavior of apt update already as everything will work as long as we don't need the keys included in these files. The behavior if they are needed is still strange through as update will fail claiming missing keys and a manual test (which the user will likely perform as root) will be successful. Passing the new warning generated by apt-key through to apt is a bit strange from an interface point of view, but basically duplicating the warning code in multiple places doesn't feel right either. That means we have no translation for the message through as apt-key has no i18n yet. It also means that if the user has a bunch of sources each of them will generate a warning for each unreadable file which could result in quite a few duplicated warnings, but "too many" is better than none. Closes: 834973
2016-08-25apt-key: warn instead of fail on unreadable keyringsDavid Kalnischkies
apt-key has inconsistent behaviour if it can't read a keyring file: Commands like 'list' skipped silently over such keyrings while 'verify' failed hard resulting in apt to report cconfusing gpg errors (#834973). As a first step we teach apt-key to be more consistent here skipping in all commands over unreadable keyrings, but issuing a warning in the process, which is as usual for apt commands displayed at the end of the run.
2016-08-24do not restore selections for already purged packagesDavid Kalnischkies
In most cases apt was already skipping the (re)setting of packages as to be removed/purged if dpkg had told us that it already did, but we haven't dealt with it in the most obvious of the cases: Selections set for packages we touched in this operation which either restores selections even dpkg would have overridden or e.g. tries to restore a purge selection for a package which was just purged – does not happen with apt itself as it isn't using selections in this way, but higher frontends like aptitude do. The result in the later case is a warning printed by dpkg that we try to set selections for an unknown package, which is harmless per se, but can be confusing for users and we really shouldn't cause warnings in dpkg if we can help it. Reported-By: Guillem Jover on IRC
2016-08-24do fail on weakhash/loop earlier in acquireDavid Kalnischkies
The bugreport shows a segfault caused by the code not doing the correct magical dance to remove an item from inside a queue in all cases. We could try hard to fix this, but it is actually better and also easier to perform these checks (which cause instant failure) earlier so that they haven't entered queue(s) yet, which in return makes cleanup trivial. The result is that we actually end up failing "too early" as if we wouldn't be careful download errors would be logged before that process was even started. Not a problem for the acquire system, but likely to confuse users and programs alike if they see the download process producing errors before apt was technically allowed to do an acquire (it didn't, so no violation, but it looks like it to the untrained eye). Closes: 835195
2016-08-23do dpkg --configure before --remove/--purge --pendingDavid Kalnischkies
Commit 7ec343309b7bc6001b465c870609b3c570026149 got us most of the way, but the last mile was botched by having the pending calls in the wrong order as this way we potentially 'force' dpkg to remove/purge a package it doesn't want to as another package still depends on it and the replacement isn't fully installed yet. So what we do now is a configure before remove and purge (all with --no-triggers) and finishing off with another configure pending call to take care of the triggers. Note that in the bugreport example our current planner is forcing dpkg to remove the package earlier via --force-depends which we could do for the pending calls as well and could be used as a workaround, but we want to do less forcing eventually. Closes: 835094
2016-08-17methods: read config in most to least specific orderDavid Kalnischkies
The implementation of the generic config fallback did the fallback in the wrong order so that the least specific option wasn't the last value picked but in fact the first one… doh! So in the bugreports case http -> https -> http::<hostname> -> https::<hostname> while it should have been the reverse as before. Regression-In: 30060442025824c491f58887ca7369f3c572fa57 Closes: 834642
2016-08-17add --with-source option and Packages/Sources supportDavid Kalnischkies
We support "./foobar.deb" as a way to install a deb file directly. Recently .changes files were added. This highlights a problem as you can't add the changes file without also trying to install all of them. Now, it could also be handy to add entire Packages/Sources files to perhaps get a bunch of packages in without installing them all implicitly. This commit introduces --with-source which allows to add *.deb, *.changes, *.dsc, source-dirs, Packages & Sources files (the later can also be compressed) without also installing them.
2016-08-17allow spaces in fingerprints for 'apt-key del'David Kalnischkies
Fingerprints tend to be displayed in space-separated octet pairs so be nice and allow delete to remove a key based on such a string rather than requiring that the user is deleting all the spaces manually.
2016-08-17add the gpg-classic variant to the gpgv/gnupg or-groupDavid Kalnischkies
We need to support partial upgrades anyhow, so we have to deal with the different versions and your tests try to ensure that we do, so we shouldn't make any explicit higher requirements.
2016-08-17tests: update self-sigs on our test keysDavid Kalnischkies
gpg upstream committed "gpgv: Tweak default options for extra security." applied on the 1.x and 2.x branches: http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=e32c575e0f3704e7563048eea6d26844bdfc494b This commit includes "[…], but we should validate the key by its self signature for primary key, and back signature for subkey." Our testkeys are old and do not really considered best practices in the last years, so their most recent self-signature is SHA1-only which with this gpg commit and our testcases defaulting to --weak-digest sha1 are refused. So what we do here is just applying some of the recent best practices on top of our testcase keys. Gbp-Dch: Ignore
2016-08-17tests: set source directory for gdbDavid Kalnischkies
Helps interactive gdb calls find the source code. Gbp-Dch: Ignore
2016-08-17support compression and by-hash for .diff/Index filesDavid Kalnischkies
In af81ab9030229b4ce6cbe28f0f0831d4896fda01 by-hash got implemented as a special compression type for our usual index files like Packages. Missing in this scheme was the special .diff/Index index file containing the info about individual patches for this index file. Deriving from the index file class directly we inherent the compression handling infrastructure and in this way also by-hash nearly for free. Closes: #824926
2016-08-17retry without same redirection mirror on 404 errorsDavid Kalnischkies
If 9b8034a9fd40b4d05075fda719e61f6eb4c45678 serves the Release files from a partial mirror we will end up getting 404 for some of the indexes. Instead of giving up, we will instead ignore our same redirection mirror constrain and ask the redirection service as a potential hashsum mismatch is better than keeping the certain 404 error.
2016-08-16don't sent Range requests if we know its not acceptedDavid Kalnischkies
If the server told us in a previous request that it isn't supporting Ranges with bytes via an Accept-Ranges header missing bytes, we don't try to formulate requests using Ranges.
2016-08-13http(s): allow empty values for header fieldsDavid Kalnischkies
It seems completely pointless from a server-POV to sent empty header fields, so most of them don't do it (simply proven by this limitation existing since day one) – but it is technically allowed by the RFC as the surounding whitespaces are optional and Github seems to like sending "X-Geo-Block-List:\r\n" since recently (bug reports in other http clients indicate July) at least sometimes as the reporter claims to have seen it on https only even through it can happen with both. Closes: 834048
2016-08-11tests: copy 01autoremove from the right placeDavid Kalnischkies
With cmake using BUILDDIRECTORY at this place is not only as wrong as it was before, but it might not even work always copying the system provided one which might or might not be current and hence fails tests needing it to be current like ./test-apt-move-and-forget-manual-sections We don't want to always use the one from the source directory through either like in autopkgtests. Gbp-Dch: Ignore
2016-08-11Merge branch 'feature/apt-dpkg-comm'David Kalnischkies
2016-08-11block direct connections to .onion domains (RFC7687)David Kalnischkies
Doing a direct connect to an .onion address (if you don't happen to use it as a local domain, which you shouldn't) is bound to fail and does leak the information that you do use Tor and which hidden service you wanted to connect to to a DNS server. Worse, if the DNS is poisoned and actually resolves tricking a user into believing the setup would work correctly… This does block also the usage of wrappers like torsocks with apt, but with native support available and advertised in the error message this shouldn't really be an issue. Inspired-by: https://bugzilla.mozilla.org/show_bug.cgi?id=1228457
2016-08-10disable explicit configuration of all packages at the endDavid Kalnischkies
With b4450f1dd6bca537e60406b2383ab154a3e1485f we dropped what we calculated here later on and now that we don't need it in the meantime either we can just skip the busy work by default and expect dpkg to do the right thing dropping also our little "last explicit configures" removal trick introduced in b4450f1dd6bca537e60406b2383ab154a3e1485f. This enables the last of a bunch of previously experimental options, some of them existing still, but are very special and hence not really worth documenting anymore (especially as it would need to be rewritten now entirely) which is why the documentation is nearly completely dropped. The order of configuration stanzas in the simulation code changes slightly as it isn't concerning itself with finding the 'right' order, but any order is valid anyhow as long as the entire set happens in the same call.
2016-08-10try to avoid removal of crossgraded packagesDavid Kalnischkies
The user has to approve the removal of a crossgraded package as it might be needed to remove it (temporarily) in the process, but in most cases we can happily avoid it and let dpkg unpack over it skipping the remove. This has some effects on progress reporting and how deal with selections through which makes this a tiny bit complicated.
2016-08-10allow methods to be disabled and redirected via configDavid Kalnischkies
To prevent accidents like adding http-sources while using tor+http it can make sense to allow disabling methods. It might even make sense to allow "redirections" and adding "symlinked" methods via configuration. This could e.g. allow using different options for certain sources by adding and configuring a "virtual" new method which picks up the config based on the name it was called with like e.g. http does if called as tor+http.
2016-08-10implement socks5h proxy support for http methodDavid Kalnischkies
Socks support is a requested feature in sofar that the internet is actually believing Acquire::socks::Proxy would exist. It doesn't and this commit isn't adding it as that isn't how our configuration works, but it allows Acquire::http::Proxy="socks5h://…". The HTTPS method was changed already to support socks proxies (all versions) via curl. This commit implements only SOCKS5 (RFC1928) with no auth or pass&user auth (RFC1929), but not GSSAPI which is required by the RFC. The 'h' in the protocol name further indicates that DNS resolution is delegated to the socks proxy rather than performed locally. The implementation works and was tested with Tor as socks proxy for which implementing socks5h only can actually be considered a feature. Closes: 744934
2016-08-10implement generic config fallback for methodsDavid Kalnischkies
The https method implemented for a long while now a hardcoded fallback to the same options in http, which, while it works, is rather inflexible if we want to allow the methods to use another name to change their behavior slightly, like apt-transport-tor does to https – most of the diff being s#https#tor#g which then fails to do the full circle fallthrough tor -> https -> http for https sources. With this config infrastructure this could be implemented now.
2016-08-10use the same redirection handling for http and httpsDavid Kalnischkies
cURL which backs our https implementation can handle redirects on its own, but by dealing with them on our own we gain finer control over which redirections will be performed (we don't like https → http) and by whom so that redirections to other hosts correctly spawn a new https method dealing with these instead of letting the current one deal with it.
2016-08-10detect redirection loops in acquire instead of workersDavid Kalnischkies
Having the detection handled in specific (http) workers means that a redirection loop over different hostnames isn't detected. Its also not a good idea have this implement in each method independently even if it would work
2016-08-10fail on unsupported http/https proxy settingsDavid Kalnischkies
Closes: #623443
2016-08-10don't purge directly, but remove and do purge at the endDavid Kalnischkies
If we want a package to be purged from the system tell dpkg in the ordering (if it has to touch it explicitly) to remove it and cover the purging of the config files at the end with a --purge --pending call. That should help packages move conffiles around between packages correctly even if the user is purging packages directly in big actions like dist-upgrades involving many packages.
2016-08-10select remove/purge packages early on for dpkgDavid Kalnischkies
Telling dpkg early on that we are going to remove these packages later helps it with auto-deconfiguration decisions and its another area where a planner can ignore the nitty gritty details and let dpkg decide the course of action if there are no special requirements.
2016-08-10save and restore selection states before/after calling dpkgDavid Kalnischkies
dpkg decides certain things on its own based on selections and especially if we want to call --pending on purge/remove actions, we need to ensure a clean slate or otherwise we surprise the user by removing packages we weren't allowed to remove by the user in this run (the selection might be an overarching plan for the not-yet "future"). Ideally dpkg would have some kind of temporal selection interface for this case, but it hasn't, so we make it temporal with the risk of loosing state if we don't manage to restore them.
2016-08-10use dpkg --unpack --recursive to avoid long cmdlinesDavid Kalnischkies
Having long commandlines split into two is a huge problem if it happens and additionally if we want to introduce planners which perform less micromanagment its a good idea to leave the details for dpkg to decide. In practice this doesn't work yet unconditionally as a bug is hiding in the ordering code of dpkg, but it works if apt imposes its ordering so this commit allows for now at least to solve the first problem.
2016-08-10pass --force-remove-essential to dpkg only if neededDavid Kalnischkies
APT (usually) knows which package is essential or not, so we can avoid passing this force flag to dpkg unconditionally if the user hasn't chosen a non-default essential handling obscuring the information.
2016-08-10Do not set the binary dir in run-tests, it breaks stuffJulian Andres Klode
This breaks -j and does all sort of other weird stuff I did not notice in the previous (non-parallel) runs. Gbp-Dch: ignore
2016-08-10test: Automatically discover CMake build directoryJulian Andres Klode
Look at the project root, and all directories directly below it and pick the directory with the newest CMakeCache.txt file. Gbp-Dch: ignore
2016-08-10Get rid of the old buildsystemJulian Andres Klode
Bye, bye, old friend.
2016-08-06CMake: Switch integration tests and travis overJulian Andres Klode
This early support seems a bit hacky, but it's a hard switch: The integration tests do not understand the old build system anymore afterwards. I don't really like that.
2016-08-06test: Pass -maxdepth 1 when running find in methods dirJulian Andres Klode
This fixes a test failures in the cmake branch which contains sub directories in the methods output dir.
2016-07-27(error) va_list 'args' was opened but not closed by va_end()David Kalnischkies
Reported-By: cppcheck Gbp-Dch: Ignore
2016-07-27rred: truncate result file before writing to itDavid Kalnischkies
If another file in the transaction fails and hence dooms the transaction we can end in a situation in which a -patched file (= rred writes the result of the patching to it) remains in the partial/ directory. The next apt call will perform the rred patching again and write its result again to the -patched file, but instead of starting with an empty file as intended it will override the content previously in the file which has the same result if the new content happens to be longer than the old content, but if it isn't parts of the old content remain in the file which will pass verification as the new content written to it matches the hashes and if the entire transaction passes the file will be moved the lists/ directory where it might or might not trigger errors depending on if the old content which remained forms a valid file together with the new content. This has no real security implications as no untrusted data is involved: The old content consists of a base file which passed verification and a bunch of patches which all passed multiple verifications as well, so the old content isn't controllable by an attacker and the new one isn't either (as the new content alone passes verification). So the best an attacker can do is letting the user run into the same issue as in the report. Closes: #831762
2016-07-22use a configurable location for apport report storageDavid Kalnischkies
Hardcoding /var/crash means we can't test it properly and it isn't really our style.
2016-07-22support dpkg debug mode in APT::StateChangesDavid Kalnischkies
Gbp-Dch: Ignore
2016-07-22create non-existent files in edit-sources with 644 instead of 640David Kalnischkies
If the sources file we want to edit doesn't exist yet GetLock will create it with 640, which for a generic lockfile might be okay, but as this is a sources file more relaxed permissions are in order – and actually required as it wont be readable for unprivileged users causing warnings/errors in apt calls. Reported-By: J. Theede (musca) on IRC
2016-07-22tests: avoid time-dependent rebuild of cachesDavid Kalnischkies
The tests changes the sources.list and the modification time of this file is considered while figuring out if the cache can be good. Usually this isn't an issue, but in that case we have the cache generation produce warnings which appear twice in this case. Gbp-Dch: Ignore
2016-07-22clean up default-stanzas from extended_states on writeDavid Kalnischkies
The existing cleanup was happening only for packages which had a status change (install -> uninstalled) which is the most frequent but no the only case – you can e.g. set autobits explicitly with apt-mark. This would leave stanzas in the states file declaring a package to be manually installed – which is the default value for a package not listed at all, so we can just as well drop it from the file.
2016-07-22tests: skip over -flags for first option in autotestsDavid Kalnischkies
Otherwise calls like "apt -q install" end up calling "aptautotest_apt_q" instead of "aptautotest_apt_install" Gbp-Dch: Ignore
2016-07-22support "install ./foo.changes"David Kalnischkies
We support installing ./foo.deb (and ./foo.dsc for source) for a while now, but it can be a bit clunky to work with those directly if you e.g. build packages locally in a 'central' build-area. The changes files also include hashsums and can be signed, so this can also be considered an enhancement in terms of security as a user "just" has to verify the signature on the changes file then rather than checking all deb files individually in these manual installation procedures.
2016-07-22allow arch=all to override No-Support-for-Architecture-allDavid Kalnischkies
If a user explicitly requests the download of arch:all apt shouldn't get in the way and perform its detection dance if arch:all packages are (also) in arch:any files or not. This e.g. allows setting arch=all on a source with such a field (or one which doesn't support all at all, but has the arch:all files like Debian itself ATM) to get only the arch:all packages from there instead of behaving like a no-op. Reported-By: Helmut Grohne on IRC
2016-07-19don't hardcode /var/lib/dpkg/status as dir::state::statusDavid Kalnischkies
Theoretically it should be enough to change the Dir setting and have apt pick the dpkg/status file from that. Also, it should be consistently effected by RootDir. Both wasn't really the case through, so a user had to explicitly set it too (or ignore it and have or not have expected sideeffects caused by it). This commit tries to guess better the location of the dpkg/status file by setting dir::state::status to a naive "../dpkg/status", just that this setting would be interpreted as relative to the CWD and not relative to the dir::state directory. Also, the status file isn't really relative to the state files apt has in /var/lib/apt/ as evident if we consider that apt/ could be a symlink to someplace else and "../dpkg" not effected by it, so what we do here is an explicit replace on apt/ – similar to how we create directories if it ends in apt/ – with dpkg/. As this is a change it has the potential to cause regressions in so far as the dpkg/status file of the "host" system is no longer used if you set a "chroot" system via the Dir setting – but that tends to be intended and causes people to painfully figure out that they had to set this explicitly before, so that it now works more in terms of how the other Dir settings work (aka "as expected"). If using the host status file is really intended it is in fact easier to set this explicitely compared to setting the new "magic" location explicitely.
2016-07-15tests: activate dpkg multi-arch even if test is single archDavid Kalnischkies
Most tests are either multiarch, do not care for the specific architecture or do not interact with dpkg, so really effect by this is only test-external-installation-planner-protocol, but its a general issue that while APT can be told to treat any architecture as native dpkg has the native architecture hardcoded so if we run tests we must make sure that dpkg knows about the architecture we will treat as "native" in apt as otherwise dpkg will refuse to install packages from such an architecture. This reverts f883d2c3675eae2700e4cd1532c1a236cae69a4e as it complicates the test slightly for no practical gain after the generic fix.
2016-07-15Use native arch in test-external-installation-planner-protocolJulian Andres Klode
Hardcoding amd64 broke the tests.