summaryrefslogtreecommitdiff
path: root/test/integration
AgeCommit message (Collapse)Author
2017-07-26use FileFd to parse all apt configuration filesDavid Kalnischkies
Using different ways of opening files means we have different behaviour and error messages for them, so by the same for all we can have more uniformity for users and apt developers alike.
2017-07-26fail early in http if server answer is too small as wellDavid Kalnischkies
Failing on too much data is good, but we can do better by checking for exact filesizes as we know with hashsums how large a file should be, so if we get a file which has a size we do not expect we can drop it directly, regardless of if the file is larger or smaller than what we expect which should catch most cases which would end up as hashsum errors later now a lot sooner.
2017-07-26fail earlier if server answers with too much dataDavid Kalnischkies
We tend to operate on rather large static files, which means we usually get Content-Length information from the server. If we combine this information with the filesize we are expecting (factoring in pipelining) we can avoid reading a bunch of data we are ending up rejecting anyhow by just closing the connection saving bandwidth and time both for the server as well as the client.
2017-07-26don't try to parse all fields starting with HTTP as status-lineDavid Kalnischkies
It is highly unlikely to encounter fields which start with HTTP in practice, but we should really be a bit more restrictive here.
2017-07-26don't move failed pdiff indexes out of partialDavid Kalnischkies
The comment says this is intended, but looking at the history reveals that the comment comes from a different era. Nowadays we don't really need it anymore (and even back then it was disputeable) as we haven't used that file for our update in the end and nothing really needs this file after the update. Triggered is this by 188f297a2af4c15cb1d502360d1e478644b5b810 which moves various error conditions forward including this code expecting the file to exist – but it doesn't need to as download could have failed. We could fix that by simple checking if the file exists and only stage it if it does, but instead we don't stage it and instead even rename it out of the way with our conventional FAILED name (if it exists). That restores support for partial mirrors (= in this case mirrors which don't ship pdiff files). Note that apt heals itself even if only such a mirror is used as the update is successful even if that error is shown. Closes: 869425
2017-07-12Handle supported components with slashes in sources.listApollon Oikonomopoulos
Commit d7c92411dc1f4c6be098d1425f9c1c075e0c2154 parses the Components section of (In)Release and attempts to detect the distribution's supported components. While doing so, it handles component names with slashes in a special manner, assuming that the actual component is only the part after the final slash. This is done to handle security.debian.org, which usually appears in sources.list as follows: deb http://s.d.o/debian-security stretch/updates main contrib non-free while the actual release file has: Codename: stretch Components: updates/main updates/contrib updates/non-free While this special handing on APTs part indeed works for debian-security, it emits spurious warnings on repositories that actually use slashes in the component names *and* appear so in sources.list. We fix this by adding both component versions (whole and final part) to the SupportedComponents array. Closes: #868127
2017-07-01Ignore download order in test-apt-update-failure-propagationJulian Andres Klode
This caused spurious test failures.
2017-06-30Switch to 'http' as the default https methodJulian Andres Klode
The old curl based method is still available as 'curl', 'curl+http', and 'curl+https'.
2017-06-30Reset failure reason when connection was successfulJulian Andres Klode
When APT was trying multiple addresses, any later error somewhere else would be reported with ConnectionRefused or ConnectionTimedOut as the FailReason because that was set by early connect attempts. This causes APT to handle the failures differently, leading to some weirdly breaking test cases (like the changed one). Add debugging to the previously failing test case so we can find out when something goes wrong there again.
2017-06-28support tor+https being handled by httpDavid Kalnischkies
The apt-transport-tor package operates via simple symlinks which can result in 'http' being called as 'tor+https', so it must pick up the right configuration pieces and trigger https support also in plus names.
2017-06-28Strip 0: epochs from the version hashJulian Andres Klode
This should fix some issues with dpkg normalizing such values. Suprisingly enough apt treats the Version: field the same, even with epoch vs without, but not when searching, and does not strip the 0: from the output.
2017-06-28show a Release-Notes URI if infos were changedDavid Kalnischkies
This gives the repository owner a chance to explain why this change was needed – e.g. explaining the organisational changes or simply detailing the changes in the new release made. Note that this URI is also shown if the change is accepted, so it also draws attention to release notes of minor updates (if users watch apt output closely).
2017-06-28error in update on Release information changesDavid Kalnischkies
The value of Origin, Label, Codename and co can be used in user configuration from apts own pinning to unattended upgrades. A repository changing this values can therefore have serious effects on the behaviour of apt and other tools using these values. In a first step we will generate error messages for these changes now explaining the need for explicit confirmation and provide config options and commandline flags to accept them.
2017-06-28fail instead of warn on insecure repositories in apt-getDavid Kalnischkies
The exception was made to give (script) users a one-release grace period to adapt their setup to deal with apt enforcing signing of repositories. As we are now at the start of a new release cycle its as good a time as any to lift it now. Removes-Exception: 952ee63b0af14a534c0aca00c11d1a99be6b22b2
2017-06-28Merge branch 'feature/http-https'Julian Andres Klode
2017-06-28Skip test-apt-download-progressJulian Andres Klode
The test keeps failing continously on Ubuntu, so let's fix it for now.
2017-06-28Fix test suite and enable non-curl testing on travis, shippableJulian Andres Klode
Gbp-Dch: ignore
2017-06-28Fix https->http redirect issuesDavid Kalnischkies
Gbp-Dch: ignore
2017-06-27travis: ignore profiling warning in progress linesDavid Kalnischkies
On Travis CI running tests with code coverage enabled sometimes generates profiling lines, which we filter out for a while now, but that misses lines generated showing progress still causing test failures, so more sed logic is added in the hopes to ignore them. Extends: 58608941e6b58a46109b7cd875716b3d8054c4bf Gbp-Dch: Ignore
2017-06-26deal with 3xx httpcodes as required by HTTP/1.1 specDavid Kalnischkies
An unknown code should be handled the same as the x00 code of this group, but for redirections we used to treat 300 (and a few others) as an error while unknown codes were considered redirections. Instead we check now explicitly for the redirection codes we support for redirecting (and add the 308 defined in RFC 7538) to avoid future problems if new 3xx codes are added expecting certain behaviours. Potentially strange would have been e.g. "305 Use Proxy" sending a Location for the proxy to use – which wouldn't have worked and resulted in an error anyhow, but probably confused users in the process.
2017-06-26fail InRelease on non-404 HTTP errorcodesDavid Kalnischkies
There are very many HTTP errorcodes which indicate that the repository isn't available at the moment or the connection has some kind of problem. Given that we do not require Release files the result was that these errors were ignored and the user presented with a message like "Repository is no longer signed" which sends the user in the wrong direction. Instead of trying to figure out which http errorcodes indicate a global problem we accept only 404 for ignoring and consider all the rest as hard errors now causing us to stop instantly after the InRelease file and print the errorcode (with short description from server) received.
2017-06-26show .diff/Index properly as ignored if we fallbackDavid Kalnischkies
Moving the code responsible for parsing the Index file from ::Done into the slightly earlier ::VerifyDone allows us to still "fail" the download if we can't make use of the Index for whatever reason, so that the progress log correctly displays "Ign" instead of "Get" for the file. This also makes quiet a few debug messages proper error messages (but those are still hidden by default for Ign lines).
2017-06-26warn if an expected file can't be acquiredDavid Kalnischkies
If we couldn't find an entry for a Sources file we would generate an error while for a Packages file we would silently skip it due to assuming it is missing because it is empty. We can do better by checking if the repository declares that it supports a component we want to get the file from and if not say so and hint at the user making a typo. An example were this helps is mozilla.debian.net which dropped the firefox-aurora component (as upstream did) meaning no upgrades until the user notices manually that the repository doesn't provide packages anymore. With this commit warnings are raised hopefully causing the user to investigate what is wrong (sooner).
2017-06-26make the create-test-data script great againDavid Kalnischkies
Changes in the past to the buildsystem and the testing framework broke this little helper script – lets fix those problems to restore functionality. Gbp-Dch: Ignore
2017-06-26Show permission error if ProxyAutoDetect cmd can't be executedDavid Kalnischkies
As the proxy commands are not executed as root, a user can run into permission errors (s)he isn't expecting – as our switching is an implementation detail – so the error message in that case should really be better than a generic "error code 100" sending the user in the wrong direction as that implies the command was executed, but errored out. Closes: 857885
2017-06-26Call update from apt-key test for a strange path testDavid Kalnischkies
We setup a "horrible" environment in the apt-key testcase to check all kinds of things, but we really should be making also at least a simple apt update call, as that in turn will call apt-key which is how apt-key is used in the non-testcase world, so that calling should be able to deal with such environments as well. Gbp-Dch: Ignore
2017-06-26Add a few more Auto-Detect-Proxy testsDavid Kalnischkies
Gbp-Dch: Ignore
2017-06-26tests: fix gpg-agent killing in testcasesDavid Kalnischkies
We want to kill the agent if its home directory exists at that location, not if it isn't there (leaving an army of processes around). Gbp-Dch: Ignore
2017-05-31Fix parsing of or groups in build-deps with ignored packagesJulian Andres Klode
If the last alternative(s) of an Or group is ignored, because it does not match an architecture list, we would end up keeping the or flag, effectively making the next AND an OR. For example, when parsing (on amd64): debhelper (>= 9), libnacl-dev [amd64] | libnacl-dev [i386] => debhelper (>= 9), libnacl-dev | Which can cause python-apt to crash. Even worse: debhelper (>= 9), libnacl-dev [amd64] | libnacl-dev [i386], foobar => debhelper (>= 9), libnacl-dev [amd64] | foobar By setting the previous alternatives Or flag to the current Or flag if the current alternative is ignored, we solve the issue. LP: #1694697
2017-03-19Ignore AutomaticRemove conffile option in upgradeDavid Kalnischkies
We are in a dilemma here: The regression of sorts was introduced in 2013 with commit d8a8f9d7f0 allowing pkg modifiers for the upgrade commands. That calls the autoremover as a sideeffect through and with it comes the option to remove the garbage packages in these commands (similar to aptitude). Having the option on the commandline is no problem – people aren't going to request what they don't want (or so I hope), but the documentation explicitly states that this option only effects install/remove and mentions a config knob users might use and expect to not suddenly apply (especially without documentation) to more commands. Just reverting the commit is out of question, completely ignoring the option breaks the workflow of every user who happened to use --autoremove on the commandline for upgrade and expects that to work given that it was accepted and worked in a stable release. Changing the documentation to reflect reality while perhaps the simplest and cleanest option contradicts freeze and is a surprising change we tend to avoid like the plague while just leaving it be confuses all users who end up believing the documentation even if was different in the last 3 years. So what we do is a tricky compromise: The configuration option if read from a file does apply only for install/remove as documented, while if the option is encountered on the commandline it is accepted and applies to the upgrade which should make 99% of the users happy. The rest has to wait for us to figure out for buster how to get that documented and implemented in a saner way. Closes: #855891
2017-02-10Do not package names representing .dsc/.deb/... filesJulian Andres Klode
In the case of build-dep and other commands where a file can be passed we must make sure not to normalize the path name as that can have odd side effects, or well, cause the operation to do nothing. Test for build-dep-file is adjusted to perform the vcard check once as "vcard" and once as "VCard", thus testing that this solves the reported bug. We inline the std::transform() and optimize it a bit to not write anything in the common case (package names are defined to be lowercase, the whole transformation is just for names that should not exist...) to counter the performance hit of the added find() call (it's about 0.15% more instructions than with the existing transform, but we save about 0.67% in writes...). Closes: #854794
2017-02-09don't test with "too early for 32bit" yearsDavid Kalnischkies
$ uname -m i686 $ date -d '0-12-25' date: invalid date '0-12-25' Test-Regression-In: 25a14d4ccfceb2698edce01092bc6a1dbe9fb217
2017-01-24test suite: Do not exit 0 in trap for QUITJulian Andres Klode
This hides errors in the test suite because it will exit with 0 here. Instead, just do exit 1 in most traps, and do just the cleanup in the QUIT hook. This fixes a regression introduced with the caching of the GPG home directory in 4ce2f35248123ff2366c8c365ad6a94945578d66.
2017-01-19fix various typos reported by spellintianDavid Kalnischkies
Most of them in (old) code comments. The two instances of user visible string changes the po files of the manpages are fixed up as well. Gbp-Dch: Ignore Reported-By: spellintian
2017-01-19make the moo reproducibleDavid Kalnischkies
Normal cows moo every time they feel like it and it might be a "moo", "moo!" or "moo?". This is completely unacceptable behaviour in our super cow through as as a superior being it has to show its superiority over the common cows and the meager easter eggs by being fully reproducible! The second version of Chris' patch is modified to include an array of tests for this feature – which doubles as explanation for some of the moo lines by giving more exact dates – and falling back to current time if the environment is invalid + passing time around instead of having an invalid environment be an unrecoverable error (aka: Guru Meditation) as that is more inline with how apt usually behaves: The wisdom of super cow should be available to everyone, even to the most misfortune users not capable of having a valid environment variable. That makes the code slightly more ugly, so instead of requiring a young follower to produce a third version a high priest of the cult applied the finishing touches as he is used to the pain by now – and another round with the slowpoke high priest would have been a serious threat to the Debian release schedule which the cow would not approve. Closes: #848721 Signed-off-by: Super Cow Thanks: Chris Lamb for initial patch and guru meditation
2017-01-19remove 'old' FAILED files in the next acquire callDavid Kalnischkies
If apt renames a file to .FAILED it leaves its namespace and is never touched again – expect since 1.1~exp4 in which "apt clean" will remove those files. The usefulness of these files rapidly degrades if you don't keep the update log itself (together with debug output in the best case) through and on 99% of all system they will be kept around forever just to collect dust over time and eat up space. With this commit an update call will remove all FAILED files of previous runs, so that the FAILED files you have on disk are always only the ones related to the last apt run stopping apt from hoarding files. Closes: 846476
2017-01-19fix 'install --no-download' modeDavid Kalnischkies
The mode wasn't working at all if not used together with --fix-missing which while likely to come in pairs its legal to use standalone. Regression-in: eb1f04dda07c2b69549ad9fd793cca0e91841b3e
2017-01-19don't show update stats if cache generation is disabledDavid Kalnischkies
Unlikely that anyone is actually running into this, but if we asked to not generate a cache and avoid it in the first step we shouldn't create one implicitly anyway by displaying the statistics.
2017-01-17test: use downloadfile instead of apthelper download-fileJulian Andres Klode
This prevents CI failures from happening in 1.3 and 1.2 and might actually be more complete. Gbp-Dch: ignore
2017-01-17https: Quote path in URL before passing it to curlJulian Andres Klode
Curl requires URLs to be urlencoded. We are however giving it undecoded URLs. This causes it go completely nuts if there is a space in the URI, producing requests like: GET /a file HTTP/1.1 which the servers then interpret as a GET request for "/a" with HTTP version "file" or some other non-sense. This works around the issue by encoding the path component of the URL. I'm not sure if we should encode other parts of the URL as well, this one seems to do the trick for the actual issue at hand. A more correct fix is to avoid the dequoting and (re-)quoting of URLs when a redirect occurs / a new request is sent. That's been on the radar for probably a year or two now, but nobody bothered implementing that yet. LP: #1651923
2017-01-17Read dpkg tables to handle architecture wildcardsJulian Andres Klode
Our implementation of wildcards was rudimentary. It worked for some common ones, but it was also broken: For example, armel matched any-armel, but should match any-arm. With this commit, we load the correct tables from dpkg. Supported are both triplets and quadruplet tables (the latter introduced in dpkg 1.18.11). There are some odd things we have to deal with in the cache filter for historical and API reasons: * The character "*" must be accepted as an alternative to any - in fact it may appear anywhere in the wildcard as we also allow fnmatch() style wildcard matching on the commandline. * The code might get passed an arch with a minus at the end, for example the cmdline "install apt:any-arm-" will first try to check if any-arm- is a valid architecture. We deal with this by rejecting any wildcard ending in a minus. * Triplets are actually implemented by extending them to faux quadruplets - by prepending a "base" component for the architecture tuple, and "any" if there is a wildcard component. Once we have constructed a wildcard, it is transformed into an fnmatch() expression for historical reasons. In the future, we should really get a tuple class and implement matching in a better, more explicit way. This does for now though - it passes all the test cases and accepts all things it should accept. Closes: #748936 Thanks: James Clarke <jrtc27@jrtc27.com> for the initial patch
2016-12-31allow warning generation for non-whitelisted optionsDavid Kalnischkies
The idea is simple: Each¹ Find*( call starts with a call check if the given option (with the requested type) exists in the whitelist. The whitelist is specified via our configure-index file so that we have a better chance at keeping it current. the whitelist is loaded via a special (undocumented for now) configuration stanza and if none is loaded the empty whitelist will make it so that no warnings are shown. Much needs to be done still, but that is as good a time as any to take a snapshot of the current state and release it into the wild given that it found some bugs already and has no practical effect on users. ¹ not all in this iteration, but many
2016-12-31expand -f to --fix-broken in error messagesDavid Kalnischkies
Users end up believing that this is a --force mode as -f is common for that, but apt doesn't have such a mode and --fix-broken is really not about forcing something but actually trying to fix the breakage which tends to be the result of a user forcing something on its system via low-level forced dpkg calls. Example: The "common" pattern of "dpkg -i ./foo.deb; apt install -f" is nowadays far better dealt with via "apt install ./foo.deb". And while at it the two places handing out this suggestion are changed to use the same strings to avoid needless translation work in the future and the suggestion uses 'apt' instead of 'apt-get' as this will be run interactively by a user, so its a good opportunity to showcase what we can do and will allow us to be more helpful to the user. Closes: #709092 Thanks: Kristian Glass for initial patch!
2016-12-31allow default build-essentials to be overriddenDavid Kalnischkies
The config list APT::Build-Essential gets a similar treatment to other lists now by having the value of the option itself be an override for the list allowing to disable build-essentials entirely as well as adding/overriding as usual by now in other lists. Reported-By: Johannes 'josch' Schauer on IRC
2016-12-31add --indep-only for build-dep commandDavid Kalnischkies
The implementation is quite different compared to --arch-only due to ABI reasons but functionality wise they are similar and usually both available for symmetry at least. Closes: #845775
2016-12-31ensure generation of valid EDSP error stanzasDavid Kalnischkies
The crude way of preparing a message to be a multiline value failed at generation valid deb822 in case the error message ended with a new line like the resolving errors from apt do. apt itself can parse these, but other tools like grep-dctrl choke on it, so be nice and print valid. Reported-By: Johannes 'josch' Schauer on IRC
2016-12-31warn if clearsigned file has ignored content partsDavid Kalnischkies
Clearsigned files like InRelease, .dsc, .changes and co can potentially include unsigned or additional messages blocks ignored by gpg in verification, but a potential source of trouble in our own parsing attempts – and an unneeded risk as the usecases for the clearsigned files we deal with do not reasonably include unsigned parts (like emails or some such). This commit changes the silent ignoring to warnings for now to get an impression on how widespread unintended unsigned parts are, but eventually we want to turn these into hard errors.
2016-12-21tests: cache the apt-key homedir used for Release signingDavid Kalnischkies
Importing a new secret key into gpg(2) can be increadibly slow which prolongs the test runs significantly – by caching the homedir we gain a significant speedbonus as reimporting already present keys seems like a far less costly operation. Git-Dch: Ignore
2016-12-16let {dsc,tar,diff}-only implicitly enable download-onlyDavid Kalnischkies
That was the case already for tar-only and diff-only, but in a more confusing way and without a message while dsc "worked" before resulting in a dpkg-source error shortly after as tar/diff files aren't available…
2016-11-25optional write aptwebserver log to client specific filesDavid Kalnischkies
The test test-handle-redirect-as-used-mirror-change serves multiple clients at the same time, so the order of the output is undefined and once in a while the two clients will intermix their lines causing the grep we perform on it later to fail making our tests fail. Solved by introducing client-specific logfiles which we all grep and sort the result to have the results more stable. Git-Dch: Ignore