summaryrefslogtreecommitdiff
path: root/test/integration
AgeCommit message (Collapse)Author
2019-01-18SECURITY UPDATE: content injection in http method (CVE-2019-3462)Julian Andres Klode
This fixes a security issue that can be exploited to inject arbritrary debs or other files into a signed repository as followed: (1) Server sends a redirect to somewhere%0a<headers for the apt method> (where %0a is \n encoded) (2) apt method decodes the redirect (because the method encodes the URLs before sending them out), writting something like somewhere\n <headers> into its output (3) apt then uses the headers injected for validation purposes. Regression-Of: c34ea12ad509cb34c954ed574a301c3cbede55ec LP: #1812353
2017-06-30Reset failure reason when connection was successfulJulian Andres Klode
When APT was trying multiple addresses, any later error somewhere else would be reported with ConnectionRefused or ConnectionTimedOut as the FailReason because that was set by early connect attempts. This causes APT to handle the failures differently, leading to some weirdly breaking test cases (like the changed one). Add debugging to the previously failing test case so we can find out when something goes wrong there again. (cherry picked from commit d3a70c3e5ae68a0e5a3d4667dd1d0fc0887e6263)
2017-06-28travis: ignore profiling warning in progress linesDavid Kalnischkies
On Travis CI running tests with code coverage enabled sometimes generates profiling lines, which we filter out for a while now, but that misses lines generated showing progress still causing test failures, so more sed logic is added in the hopes to ignore them. Extends: 58608941e6b58a46109b7cd875716b3d8054c4bf Gbp-Dch: Ignore (cherry picked from commit fc251c8c9e2a76ab5c350900e9e032830c81e2b3)
2017-05-31Fix parsing of or groups in build-deps with ignored packagesJulian Andres Klode
If the last alternative(s) of an Or group is ignored, because it does not match an architecture list, we would end up keeping the or flag, effectively making the next AND an OR. For example, when parsing (on amd64): debhelper (>= 9), libnacl-dev [amd64] | libnacl-dev [i386] => debhelper (>= 9), libnacl-dev | Which can cause python-apt to crash. Even worse: debhelper (>= 9), libnacl-dev [amd64] | libnacl-dev [i386], foobar => debhelper (>= 9), libnacl-dev [amd64] | foobar By setting the previous alternatives Or flag to the current Or flag if the current alternative is ignored, we solve the issue. LP: #1694697
2017-03-19Ignore AutomaticRemove conffile option in upgradeDavid Kalnischkies
We are in a dilemma here: The regression of sorts was introduced in 2013 with commit d8a8f9d7f0 allowing pkg modifiers for the upgrade commands. That calls the autoremover as a sideeffect through and with it comes the option to remove the garbage packages in these commands (similar to aptitude). Having the option on the commandline is no problem – people aren't going to request what they don't want (or so I hope), but the documentation explicitly states that this option only effects install/remove and mentions a config knob users might use and expect to not suddenly apply (especially without documentation) to more commands. Just reverting the commit is out of question, completely ignoring the option breaks the workflow of every user who happened to use --autoremove on the commandline for upgrade and expects that to work given that it was accepted and worked in a stable release. Changing the documentation to reflect reality while perhaps the simplest and cleanest option contradicts freeze and is a surprising change we tend to avoid like the plague while just leaving it be confuses all users who end up believing the documentation even if was different in the last 3 years. So what we do is a tricky compromise: The configuration option if read from a file does apply only for install/remove as documented, while if the option is encountered on the commandline it is accepted and applies to the upgrade which should make 99% of the users happy. The rest has to wait for us to figure out for buster how to get that documented and implemented in a saner way. Closes: #855891
2017-02-10Do not package names representing .dsc/.deb/... filesJulian Andres Klode
In the case of build-dep and other commands where a file can be passed we must make sure not to normalize the path name as that can have odd side effects, or well, cause the operation to do nothing. Test for build-dep-file is adjusted to perform the vcard check once as "vcard" and once as "VCard", thus testing that this solves the reported bug. We inline the std::transform() and optimize it a bit to not write anything in the common case (package names are defined to be lowercase, the whole transformation is just for names that should not exist...) to counter the performance hit of the added find() call (it's about 0.15% more instructions than with the existing transform, but we save about 0.67% in writes...). Closes: #854794
2017-02-09don't test with "too early for 32bit" yearsDavid Kalnischkies
$ uname -m i686 $ date -d '0-12-25' date: invalid date '0-12-25' Test-Regression-In: 25a14d4ccfceb2698edce01092bc6a1dbe9fb217
2017-01-24test suite: Do not exit 0 in trap for QUITJulian Andres Klode
This hides errors in the test suite because it will exit with 0 here. Instead, just do exit 1 in most traps, and do just the cleanup in the QUIT hook. This fixes a regression introduced with the caching of the GPG home directory in 4ce2f35248123ff2366c8c365ad6a94945578d66.
2017-01-19fix various typos reported by spellintianDavid Kalnischkies
Most of them in (old) code comments. The two instances of user visible string changes the po files of the manpages are fixed up as well. Gbp-Dch: Ignore Reported-By: spellintian
2017-01-19make the moo reproducibleDavid Kalnischkies
Normal cows moo every time they feel like it and it might be a "moo", "moo!" or "moo?". This is completely unacceptable behaviour in our super cow through as as a superior being it has to show its superiority over the common cows and the meager easter eggs by being fully reproducible! The second version of Chris' patch is modified to include an array of tests for this feature – which doubles as explanation for some of the moo lines by giving more exact dates – and falling back to current time if the environment is invalid + passing time around instead of having an invalid environment be an unrecoverable error (aka: Guru Meditation) as that is more inline with how apt usually behaves: The wisdom of super cow should be available to everyone, even to the most misfortune users not capable of having a valid environment variable. That makes the code slightly more ugly, so instead of requiring a young follower to produce a third version a high priest of the cult applied the finishing touches as he is used to the pain by now – and another round with the slowpoke high priest would have been a serious threat to the Debian release schedule which the cow would not approve. Closes: #848721 Signed-off-by: Super Cow Thanks: Chris Lamb for initial patch and guru meditation
2017-01-19remove 'old' FAILED files in the next acquire callDavid Kalnischkies
If apt renames a file to .FAILED it leaves its namespace and is never touched again – expect since 1.1~exp4 in which "apt clean" will remove those files. The usefulness of these files rapidly degrades if you don't keep the update log itself (together with debug output in the best case) through and on 99% of all system they will be kept around forever just to collect dust over time and eat up space. With this commit an update call will remove all FAILED files of previous runs, so that the FAILED files you have on disk are always only the ones related to the last apt run stopping apt from hoarding files. Closes: 846476
2017-01-19fix 'install --no-download' modeDavid Kalnischkies
The mode wasn't working at all if not used together with --fix-missing which while likely to come in pairs its legal to use standalone. Regression-in: eb1f04dda07c2b69549ad9fd793cca0e91841b3e
2017-01-19don't show update stats if cache generation is disabledDavid Kalnischkies
Unlikely that anyone is actually running into this, but if we asked to not generate a cache and avoid it in the first step we shouldn't create one implicitly anyway by displaying the statistics.
2017-01-17test: use downloadfile instead of apthelper download-fileJulian Andres Klode
This prevents CI failures from happening in 1.3 and 1.2 and might actually be more complete. Gbp-Dch: ignore
2017-01-17https: Quote path in URL before passing it to curlJulian Andres Klode
Curl requires URLs to be urlencoded. We are however giving it undecoded URLs. This causes it go completely nuts if there is a space in the URI, producing requests like: GET /a file HTTP/1.1 which the servers then interpret as a GET request for "/a" with HTTP version "file" or some other non-sense. This works around the issue by encoding the path component of the URL. I'm not sure if we should encode other parts of the URL as well, this one seems to do the trick for the actual issue at hand. A more correct fix is to avoid the dequoting and (re-)quoting of URLs when a redirect occurs / a new request is sent. That's been on the radar for probably a year or two now, but nobody bothered implementing that yet. LP: #1651923
2017-01-17Read dpkg tables to handle architecture wildcardsJulian Andres Klode
Our implementation of wildcards was rudimentary. It worked for some common ones, but it was also broken: For example, armel matched any-armel, but should match any-arm. With this commit, we load the correct tables from dpkg. Supported are both triplets and quadruplet tables (the latter introduced in dpkg 1.18.11). There are some odd things we have to deal with in the cache filter for historical and API reasons: * The character "*" must be accepted as an alternative to any - in fact it may appear anywhere in the wildcard as we also allow fnmatch() style wildcard matching on the commandline. * The code might get passed an arch with a minus at the end, for example the cmdline "install apt:any-arm-" will first try to check if any-arm- is a valid architecture. We deal with this by rejecting any wildcard ending in a minus. * Triplets are actually implemented by extending them to faux quadruplets - by prepending a "base" component for the architecture tuple, and "any" if there is a wildcard component. Once we have constructed a wildcard, it is transformed into an fnmatch() expression for historical reasons. In the future, we should really get a tuple class and implement matching in a better, more explicit way. This does for now though - it passes all the test cases and accepts all things it should accept. Closes: #748936 Thanks: James Clarke <jrtc27@jrtc27.com> for the initial patch
2016-12-31allow warning generation for non-whitelisted optionsDavid Kalnischkies
The idea is simple: Each¹ Find*( call starts with a call check if the given option (with the requested type) exists in the whitelist. The whitelist is specified via our configure-index file so that we have a better chance at keeping it current. the whitelist is loaded via a special (undocumented for now) configuration stanza and if none is loaded the empty whitelist will make it so that no warnings are shown. Much needs to be done still, but that is as good a time as any to take a snapshot of the current state and release it into the wild given that it found some bugs already and has no practical effect on users. ¹ not all in this iteration, but many
2016-12-31expand -f to --fix-broken in error messagesDavid Kalnischkies
Users end up believing that this is a --force mode as -f is common for that, but apt doesn't have such a mode and --fix-broken is really not about forcing something but actually trying to fix the breakage which tends to be the result of a user forcing something on its system via low-level forced dpkg calls. Example: The "common" pattern of "dpkg -i ./foo.deb; apt install -f" is nowadays far better dealt with via "apt install ./foo.deb". And while at it the two places handing out this suggestion are changed to use the same strings to avoid needless translation work in the future and the suggestion uses 'apt' instead of 'apt-get' as this will be run interactively by a user, so its a good opportunity to showcase what we can do and will allow us to be more helpful to the user. Closes: #709092 Thanks: Kristian Glass for initial patch!
2016-12-31allow default build-essentials to be overriddenDavid Kalnischkies
The config list APT::Build-Essential gets a similar treatment to other lists now by having the value of the option itself be an override for the list allowing to disable build-essentials entirely as well as adding/overriding as usual by now in other lists. Reported-By: Johannes 'josch' Schauer on IRC
2016-12-31add --indep-only for build-dep commandDavid Kalnischkies
The implementation is quite different compared to --arch-only due to ABI reasons but functionality wise they are similar and usually both available for symmetry at least. Closes: #845775
2016-12-31ensure generation of valid EDSP error stanzasDavid Kalnischkies
The crude way of preparing a message to be a multiline value failed at generation valid deb822 in case the error message ended with a new line like the resolving errors from apt do. apt itself can parse these, but other tools like grep-dctrl choke on it, so be nice and print valid. Reported-By: Johannes 'josch' Schauer on IRC
2016-12-31warn if clearsigned file has ignored content partsDavid Kalnischkies
Clearsigned files like InRelease, .dsc, .changes and co can potentially include unsigned or additional messages blocks ignored by gpg in verification, but a potential source of trouble in our own parsing attempts – and an unneeded risk as the usecases for the clearsigned files we deal with do not reasonably include unsigned parts (like emails or some such). This commit changes the silent ignoring to warnings for now to get an impression on how widespread unintended unsigned parts are, but eventually we want to turn these into hard errors.
2016-12-21tests: cache the apt-key homedir used for Release signingDavid Kalnischkies
Importing a new secret key into gpg(2) can be increadibly slow which prolongs the test runs significantly – by caching the homedir we gain a significant speedbonus as reimporting already present keys seems like a far less costly operation. Git-Dch: Ignore
2016-12-16let {dsc,tar,diff}-only implicitly enable download-onlyDavid Kalnischkies
That was the case already for tar-only and diff-only, but in a more confusing way and without a message while dsc "worked" before resulting in a dpkg-source error shortly after as tar/diff files aren't available…
2016-11-25optional write aptwebserver log to client specific filesDavid Kalnischkies
The test test-handle-redirect-as-used-mirror-change serves multiple clients at the same time, so the order of the output is undefined and once in a while the two clients will intermix their lines causing the grep we perform on it later to fail making our tests fail. Solved by introducing client-specific logfiles which we all grep and sort the result to have the results more stable. Git-Dch: Ignore
2016-11-25add apt-key support for armored GPG key files (*.asc)David Kalnischkies
Having binary files in /etc is kinda annoying – not that the armored files are much better – but it is hard to keep tabs on which format the file has ("simple" or "keybox") and different gnupg versions have different default binary formats which can be confusing for users to work with (beside that it is binary). Adding support for this now will enable us in some distant future to move to armored later on, much like we added trusted.gpg.d years before the world picked it up.
2016-11-24don't perform implicit crossgrades involving M-A:sameDavid Kalnischkies
dpkg stumbles over these (#844300) and we haven't dropped 'easier' removes to be implicit and to be scheduled by dpkg by default so far so we shouldn't push the decision in such cases to dpkg either.
2016-11-24improve arch-unqualified dpkg-progress parsingDavid Kalnischkies
Our old idea was to look for the first package which would be "touched" and take this as the package dpkg is talking about, but that is incorrect in complicated situations like a package upgraded to/from multiple M-A:same siblings installed. As we us the progress report to decide what is still needed we have to be reasonabily right about the package dpkg is talking about, so we jump to quite a few loops to get it.
2016-11-23correct cross & disappear progress detectionDavid Kalnischkies
Given that we use the progress information to skip over actions dpkg has already done like not purging a package which was already removed and had no config files or not acting on disappeared packages and such it is important that apt and dpkg agree on which states the package has to pass through. To ensure that we keep tabs on this in the future a warning is added at the end if apt hasn't seen all the action it was supposed to see. I can't wait for the first bugreporters to wonder about this…
2016-11-11apt-ftparchive: Support NotAutomatic and ButAutomaticUpgrades fieldsJames Clarke
This also changes Acquire-By-Hash to be "yes" rather than "true", so it is consistent with dak's output. Closes: #272557
2016-11-11show distribution mismatch for changed codenamesDavid Kalnischkies
We have the last Release file around for other checks, so its trivial to look if the new Release file contains a new codename (e.g. the user has "testing" in the sources and it flipped from stretch to buster). Such a change can be okay and expected, but also be a hint of problems, so a warning if we see it happen seems okay. We can only print it once anyhow and frontends and co are likely to ignore/hide it.
2016-11-11don't warn if untransformed distribution matchesDavid Kalnischkies
A suite or codename entry in the Release file is checked against the distribution field in the sources.list entry that lead to the download of that Release file. This distribution entry can contain slashes in the distribution field: deb http://security.debian.org/debian wheezy/updates main However, the Release file may only contain "wheezy" in the Codename field and not "wheezy/updates". So a transformation needs to take place that removes the last / and everything that comes after (e.g. "/updates"). This fails, however, for valid cases like a reprepro snapshot where the given Codename contains slashes but is perfectly fine and doesn't need to be transformed. Since that transformation is essentially just a workaround for special cases like the security repository, it should be checked if the literal Codename without any transformations happened is valid and only if isn't the dist should be checked against the transformated one. This way special cases like security.debian.org are handled and reprepro snapshots work too. The initial patch was taken as insperationto move whole transformation to CheckDist() which makes this method more accepting & easier to use (but according to codesearch.d.n we are the only users anyhow). Thanks: Lukas Anzinger for initial patch Closes: 644610
2016-11-11add hidden config to set packages as Essential/ImportantDavid Kalnischkies
You can pretty much achieve the same with a local dummy package if you want to, but libapt has an inbuilt setting for essential: "apt" which can be overridden with this option as well – it could be helpful in quick tests and what not so adding this alternative shouldn't really hurt much. We aren't going to document them much through as care must be taken in regards to the binary caches as they aren't invalidated by config options alone, so the effects of old settings could still be in them, similar to the other already existing pkgCacheGen option(s). Closes: 767891 Thanks: Anthony Towns for initial patch
2016-11-10travis: ignore profiling warning in test diffsDavid Kalnischkies
On Travis CI running tests with code coverage enabled sometimes generates lines like: profiling:/path/to/file.gcda:Merge mismatch for function 257 It would be nice if we could resolve this somehow as it garbles the statistics, but until then it is far more annoying that this causes test failures for no good reason. Gbp-Dch: Ignore
2016-11-10support 'apt build-dep .' (aka: without /)David Kalnischkies
Reported-By: Christoph Berg (Myon) on IRC
2016-11-10improve SOCKS error messages for http slightlyDavid Kalnischkies
The 0.0.0.0:0 tor reports is pretty useless by itself, but even if an IP would be reported it is better to show the user the hostname we wanted the proxy to connect to in the same error message. We improve upon it further by looking for this bind address in particular and remap error messages slightly to give users a better chance of figuring out what went wrong. Upstream Tor can't do that as it is technically wrong.
2016-11-09rename Checksum-FileSize to Filesize in hashsum mismatchDavid Kalnischkies
Some people do not recognize the field value with such an arcane name and/or expect it to refer to something different (e.g. #839257). We can't just rename it internally as its an avoidance strategy as such fieldname existed previously with less clear semantics, but we can spare the general public from this implementation detail.
2016-11-09show the conflicting distribution warning againDavid Kalnischkies
Sometimes you should really act upon your todos. Especially if you have placed them directly in the code. Closes: 841874
2016-11-09reset HOME, USER(NAME), TMPDIR & SHELL in DropPrivilegesDavid Kalnischkies
We can't cleanup the environment like e.g. sudo would do as you usually want the environment to "leak" into these helpers, but some variables like HOME should really not have still the value of the root user – it could confuse the helpers (USER) and HOME isn't accessible anyhow. Closes: 842877
2016-11-09add a testcase for support of various build-dependency typesDavid Kalnischkies
See also c5f22e483cc0f31f2938874370ac776e40792069 Gbp-Dch: Ignore
2016-11-09fix testcase expecting incorrect remove log from dpkgDavid Kalnischkies
dpkg 1.18.11 includes: * Do not log nor print duplicate dpkg removal action. We print “Removing <package> (<version>)” lines and log remove action twice when purging a package from frontends, because they usually first call --remove and then --purge sequentially. When purging a package which is already in config-files (i.e. it has been removed before), do not print nor log the remove action.
2016-11-02keep Release.gpg on untrusted to trusted IMS-HitDavid Kalnischkies
A user relying on the deprecated behaviour of apt-get to accept a source with an unknown pubkey to install a package containing the key expects that the following 'apt-get update' causes the source to be considered as trusted, but in case the source hadn't changed in the meantime this wasn't happening: The source kept being untrusted until the Release file was changed. This only effects sources not using InRelease and only apt-get, the apt binary downright refuses this course of actions, but it is a common way of adding external sources. Closes: 838779
2016-11-02don't install new deps of candidates for kept back pkgsDavid Kalnischkies
In effect this is an extension of the 6 years old commit a8dfff90aa740889eb99d00fde5d70908d9fd88a which uses the autoremover to remove packages again from the solution which are no longer needed to be there. Commonly these are dependencies of packages we end up not installed due to problem resolver decisions. Slightly less common is the situation we deal with here: a package which we wanted to upgrade sporting a new dependency, but ended up holding back. The problem is that all versions of an installed reverse dependencies can bring back a "garbage" package – we need to do this as there is nothing inherently wrong in having garbage packages installed or upgrade them, which itself would have garbage dependencies, so just blindly killing all new garbage packages would prevent the upgrade (and actually generate errors). What we should be doing is looking only at the version we will have on the system, disregarding all old/new reverse dependencies. Reported-By: Stuart Prescott (themill) on IRC
2016-09-18VersionHash: Do not skip too long dependency linesJulian Andres Klode
If the dependency line does not contain spaces in the repository but does in the dpkg status file (because dpkg normalized the dependency list), the dpkg line might be longer than the line in the repository. If it now happens to be longer than 1024 characters, it would be skipped, causing the hashes to be out of date. Note that we have to bump the minor cache version again as this changes the format slightly, and we might get mismatches with an older src cache otherwise. Fixes Debian/apt#23
2016-09-07test: Always install dpkg into our tests, regardless of MAJulian Andres Klode
Even if we only configure a single architecture, install dpkg, so dpkg can assert multi arch correctly. This also has the nice side effect of making single architecture and multiple architecture test cases more uniform. This fixes a regression from f878d3a862128bc1385616751ae1d78246b1bd01 ("test: Assert multi-arch in the chroot").
2016-09-07test: framework: Ensure copied status files have trailing linesJulian Andres Klode
If we copied one of the existing status files, we might not have a trailing newline, so let's add one. Gbp-Dch: ignore
2016-09-07edsp: try 2 to read responses even if writing failedDavid Kalnischkies
Commit b60c8a89c281f2bb945d426d2215cbf8f5760738 improved the situation, but due to inconsistency mostly for planners, not for solvers. As the idea of hiding errors if we show another error is a bit scary (as the extern error might be a followup of our intern error, rather than the reason for our intern error as it is at the moment) we don't discard the errors, but if we got an extern error we show them directly removing them from the error list at the end of the run – that list will contain the extern error which hopefully gives us the best of both worlds. The problem itself is the same as before: The externals exiting before apt is done talking to them. Reported-By: Johannes 'josch' Schauer on IRC
2016-09-02acquire: Use priority queues and a 3 stage pipeline designJulian Andres Klode
Employ a priority queue instead of a normal queue to hold the items; and only add items to the running pipeline if their priority is the same or higher than the priority of items in the queue. The priorities are designed for a 3 stage pipeline system: In stage 1, all Release files and .diff/Index files are fetched. This allows us to determine what files remain to be fetched, and thus ensures a usable progress reporting. In stage 2, all Pdiff patches are fetched, so we can apply them in parallel with fetching other files in stage 3. In stage 3, all other files are fetched (complete index files such as Contents, Packages). Performance improvements, mainly from fetching the pdiff patches before complete files, so they can be applied in parallel: For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update of Debian unstable and testing with Contents and appstream for amd64 and i386, update time reduced from 37 seconds to 24-28 seconds. Previously, apt would first download new DEP11 icon tarballs and metadata files, causing the CPU to be idle. By fetching the diffs in stage 2, we can now patch our contents and Packages files while we are downloading the DEP11 stuff.
2016-09-01support long keyid and fingerprint in gpgv's GOODSIGDavid Kalnischkies
In gpgv1 GOODSIG (and the other messages of status-fd) are documented as sending the long keyid. In gpgv2 it is documented to be either long keyid or the fingerprint. At the moment it is still the long keyid, but the documentation hints at the possibility of changing this. We care about this for Signed-By support as we detect this way if the right fingerprint has signed this file (or not). The check itself is done via VALIDSIG which always is a fingerprint, but there must also be a GOODSIG (as expired sigs are valid, too) found to be accepted which wouldn't be found in the fingerprint-case and the signature hence refused.
2016-09-01test-apt-cdrom: Fix for gnupg 2.1.15Julian Andres Klode
gpg annoyingly changed its output and broke our test suite again by adding two extra lines about key type and issuer. Really annoying. Those lines also have more than one space after the colon, so let's use \s* there - and also change the other lines to support variable length whitespace in case gpg decides to break things there too.