summaryrefslogtreecommitdiff
path: root/test
AgeCommit message (Collapse)Author
2015-05-13detect Releasefile IMS hits even if the server doesn'tDavid Kalnischkies
Not all servers we are talking to support If-Modified-Since and some are not even sending Last-Modified for us, so in an effort to detect such hits we run a hashsum check on the 'old' compared to the 'new' file, we got the hashes for the 'new' already for "free" from the methods anyway and hence just need to calculated the old ones. This allows us to detect hits even with unsupported servers, which in turn means we benefit from all the new hit behavior also here.
2015-05-12detect 416 complete file in partial by expected hashDavid Kalnischkies
If we have the expected hashes we can check with them if the file we have in partial we got a 416 for is the expected file. We detected this with same-size before, but not every server sends a good Content-Range header with a 416 response.
2015-05-11rewrite all TFRewrite instances to use the new pkgTagSection::WriteDavid Kalnischkies
While it is mostly busywork to rewrite all instances it actually fixes bugs as the data storage used by the new method is std::string rather than a char*, the later mostly created by c_str() from a std::string which the caller has to ensure keeps in scope – something apt-ftparchive actually didn't ensure and relied on copy-on-write behavior instead which c++11 forbids and hence the new default gcc abi doesn't use it.
2015-05-11implement a more c++-style TFRewrite alternativeDavid Kalnischkies
TFRewrite is okay, but it has obscure limitations (256 Tags), even more obscure bugs (order for renames is defined by the old name) and the interface is very c-style encouraging bad usage like we do it in apt-ftparchive passing massive amounts of c_str() from std::string in. The old-style is marked as deprecated accordingly. The next commit will fix all places in the apt code to not use the old-style anymore.
2015-05-11sync TFRewrite*Order arrays with dpkg and dakDavid Kalnischkies
dpkg and dak know various field names and order them in their output, while we have yet another order and have to play catch up with them as we are sitting between chairs here and neither order is ideal for us, too. A little testcase is from now on supposed to help ensureing that we do not derivate to far away from which fields dpkg knows and orders.
2015-05-11remove available file to have same dpkg -l behaviorDavid Kalnischkies
dpkg -l < 1.16.2 loads the available file and hence sees a package which later versions do not see, leading to failures on travis-ci. The different versions also have slightly different messages. Git-Dch: Ignore
2015-05-11remove unused and strange default-value for pinsDavid Kalnischkies
If the pin for a generic pin is 0, it get a value by strange looking rules, if the pin is specific the rules are at least not strange, but the value 989 is a magic number without any direct meaning… but both never happens in practice as the parsing skips such entries with a warning, so there always is a priority != 0 and the code therefore never used.
2015-05-11a pin of 1000 always means downgrade allowedDavid Kalnischkies
The documentation says this, but the code only agreed while evaluating specific packages, but not generics. These needed a pin above 1000 to have the same effect. The code causing this makes references to a 'second pesduo status file', but nowhere is explained what this might stand for and/or what it was, so we do the only reasonable thing: Remove all references and do as documented.
2015-05-11do not require installed libapt-pkg-dev for gtestDavid Kalnischkies
Git-Dch: Ignore
2015-05-11improve partial/ cleanup in abort and failure casesDavid Kalnischkies
Especially pdiff-enhanced downloads have the tendency to fail for various reasons from which we can recover and even a successful download used to leave the old unpatched index in partial/. By adding a new method responsible for making the transaction of an individual file happen we can at specialisations especially for abort cases to deal with the cleanup. This also helps in keeping the compressed indexes around if another index failed instead of keeping the decompressed files, which we wouldn't pick up in the next call.
2015-04-19Merge branch 'debian/jessie' into debian/experimentalDavid Kalnischkies
Conflicts: apt-pkg/acquire-item.cc cmdline/apt-key.in methods/https.cc test/integration/test-apt-key test/integration/test-multiarch-foreign
2015-04-19a hit on Release files means the indexes will be hits tooDavid Kalnischkies
If we get a IMSHit for the Transaction-Manager (= the InRelease file or as its still supported fallback Release + Release.gpg combo) we can assume that every file we would queue based on this manager, but already have locally is current and hence would get an IMSHit, too. We therefore save us and the server the trouble and skip the queuing in this case. Beside speeding up repetative executions of 'apt-get update' this way we also avoid hitting hashsum errors if the indexes are in fact already updated, but the Release file isn't yet as it is the case on well behaving mirrors as Release files is updated last. The implementation is a bit harder than the theory makes it sound as we still have to keep reverifying the Release files (e.g. to detect now expired once to avoid an attacker being able to silently stale us) and have to handle cases in which the Release file hits, but some indexes aren't present (e.g. user added a new foreign architecture).
2015-04-19ensure lists/ files have correct permissions after apt-cdrom addDavid Kalnischkies
Its a bit unpredictable which permissons and owners we will encounter on a CD-ROM (or a USB stick, as apt-cdrom is responsible for those too), so we have to ensure in this codepath as well that everything is nicely setup without waiting for a 'apt-get update' to fix up the (potential) mess.
2015-04-19calculate hashes while downloading in httpsDavid Kalnischkies
We do this in HTTP already to give the CPU some exercise while the disk is heavily spinning (or flashing?) to store the data avoiding the need to reread the entire file again later on to calculate the hashes – which happens outside of the eyes of progress reporting, so you might ended up with a bunch of https workers 'stuck' at 100% while they were busy calculating hashes. This is a bummer for everyone using apt as a connection speedtest as the https method works slower now (not really, it just isn't reporting done too early anymore).
2015-04-19calculate only expected hashes in methodsDavid Kalnischkies
Methods get told which hashes are expected by the acquire system, which means we can use this list to restrict what we calculate in the methods as any extra we are calculating is wasted effort as we can't compare it with anything anyway. Adding support for a new hash algorithm is therefore 'free' now and if a algorithm is no longer provided in a repository for a file, we automatically stop calculating it. In practice this results in a speed-up in Debian as we don't have SHA512 here (so far), so we practically stop calculating it.
2015-04-19handle servers closing encoded connections correctlyDavid Kalnischkies
Servers who advertise that they close the connection get the 'Closes' encoding flag, but this conflicts with servers who response with a transfer-encoding (e.g. encoding) as it is saved in the same flag. We have a better flag for the keep-alive (or not) of the connection anyway, so we check this instead of the encoding. This is in practice not much of a problem as real servers we talk to are HTTP1.1 servers (with keep-alive) and there isn't much point in doing chunked encoding if you are going to close anyway, but our simple testserver stumbles over this if pressed and its a bit cleaner, too. Git-Dch: Ignore
2015-04-19send Alt-* info for uncompressed based on any compressionsDavid Kalnischkies
file sends information about the uncompressed file if it can find it as well as for the compressed file. This was done only for gzip so far, but we support more compression types. That this information isn't used a lot is a different story. Git-Dch: Ignore
2015-04-19improve https method queue progress reportingDavid Kalnischkies
The worker expects that the methods tell him when they start or finish downloading a file. Various information pieces are passed along in this report including the (expected) filesize. https was using a "global" struct for reporting which made it 'reuse' incorrect values in some cases like a non-existent InRelease fallbacking to Release{,.gpg} resulting in a size-mismatch warning. Reducing the scope and redesigning the setting of the values we can fix this and related issues. Closes: 777565, 781509 Thanks: Robert Edmonds and Anders Kaseorg for initial patchs
2015-04-19demote missing gtest to a buildtime warningDavid Kalnischkies
We just need it for unit tests and our debian/rules file actually skips calling them if nocheck is given… but this fails anyhow as we declared a hard-dependency on it. Demoting the error to a warning in configuration and adding a test in the 'make test' path with a friendly message allows nocheck to be useful again. (Running unit tests is fully encouraged of course, but bootstrappers and co do not need to be burdened with this stuff)
2015-04-12parse specific-arch dependencies correctly on single-arch systemsDavid Kalnischkies
On single-arch the parsing was creating groupnames like 'apt:amd64' even through it should be 'apt' and a package in it belonging to architecture amd64. The result for foreign architectures was as expected: The dependency isn't satisfiable, but for native architecture it means the wrong package (ala apt:amd64:amd64) is linked so this is also not satisfiable, which is very much not expected. No longer excluding single-arch from this codepath allows the generation of the correct links, which still link to non-exisiting packages for foreign dependencies, but natives link to the expected native package just as if no architecture was given. For negative arch-specific dependencies ala Conflicts this matter was worse as apt will believe there isn't a Conflict to resolve, tricking it into calculating a solution dpkg will refuse. Architecture specific positive dependencies are rare in jessie – the only one in amd64 main is foreign –, negative dependencies do not even exist. Neither class has a native specimen, so no package in jessie is effected by this bug, but it might be interesting for stretch upgrades. This also means the regression potential is very low. Closes: 777760
2015-04-10test for or-group choice behaviour in upgradeDavid Kalnischkies
In #780028 we were discussing how the or-group order should be more important than keep-back decisions of 'upgrade'. We have this behaviour, but to ensure it stays this way lets add a test for it. Git-Dch: Ignore
2015-04-10add a simple unit test for acquire progressDavid Kalnischkies
This isn't testing much of the 'complex' parts, but its better than nothing for now. Git-Dch: Ignore
2015-04-10reimplement the last uses of sprintfDavid Kalnischkies
Working with strings c-style is complicated and error-prune, so by converting to c++ style we gain some simplicity and avoid buffer overflows by later extensions. Git-Dch: Ignore
2015-04-07keyids in "apt-key del" should be case-insensitiveDavid Kalnischkies
gnupg is case-insensitive about keyids, so back then apt-key called it directly any keyid was accepted, but now that we work more with the keyid ourself we regressed to require uppercase keyids by accident. This is also inconsistent with other apt-key commands which still use gnupg directly. A single case-insensitive grep and we are fine again. Closes: 781696
2015-03-16stop displaying time of build in online helpJérémy Bobbio
As part of the “reproducible builds” effort [1], we have noticed that apt could not be built reproducibly. One issue is that it uses the __DATE__ and __TIME__ macros of the C preprocessor to display the time of build in the online help. We believe this information not to be really useful to users as they can always look at the package data and metadata to figure it out. The attached patch simply removes this information. All non-documentation packages can then be built reproducibly with our current experimental framework. [David: changed the string slightly to be untranslateable as well] Closes: 774342
2015-03-16test exitcode as well as string equalityDavid Kalnischkies
We use test{success,failure} now all over the place in the framework, so its only consequencial to do this in the situations in which we test for a specific output as well. Git-Dch: Ignore
2015-03-16derive more of https from http methodDavid Kalnischkies
Bug #778375 uncovered that https wasn't properly integrated in the class family tree of http as it was supposed to be leading to a NULL pointer dereference. Fixing this 'properly' was deemed to much diff for practically no gain that late in the release, so commit 0c2dc43d4fe1d026650b5e2920a021557f9534a6 just fixed the synptom, while this commit here is fixing the cause plus adding a test.
2015-03-16merge debian/sid into debian/experimentalDavid Kalnischkies
2015-03-08rework dpkg-wrapping in test frameworkDavid Kalnischkies
We are able to run maintainer scripts if needed before, but we need to set ADMINDIR to be able to call dpkg tools like dpkg-trigger inside of them. Git-Dch: Ignore
2015-01-10award points for positive dependencies againDavid Kalnischkies
Commit 9ec748ff103840c4c65471ca00d3b72984131ce4 from Feb 23 last year adds a version check after 8daf68e366fa9fa2794ae667f51562663856237c added 8 days earlier negative points for breaks/conflicts with the intended that only dependencies which are satisfied propagate points (aka: old conflicts do not). The implementation was needlessly complex and flawed through preventing positive dependencies from gaining points like they did before these commits making library transitions harder instead of simpler. It worked out anyhow most of the time out of pure 'luck' (and other ways of gaining points) or got miss attributed to being a temporary hick-up. Closes: 774924
2015-01-10128 KiB DSC files ought to be enough for everyoneDavid Kalnischkies
Your mileage may vary, but don't worry: There is more than one way to do it, but our one size fits all is not a bigger hammer, but an entire roundhouse kick! So brace yourself for the tl;dr: The limit is gone.* Beware: This fixes also the problem that a double newline is unconditionally added 'later' which is an overcommitment in case the dsc filesize is limit-2 <= x <= limit. * limited to numbers fitting into an unsigned long long. Closes: 774893
2015-01-06Add regression test for the previous commitMichael Vogt
The issue was that https.cc never called URIStart(), one way to detect this is that no download progress is generated without this call. The test now checks for this and as a side-effect will also ensure that we do not break download progress reporting and Acquire::{http,https}::Dl-Limit accidently.
2014-12-23pass-through stdin fd instead of content if not a terminalDavid Kalnischkies
Commit 299aea924ccef428219ed6f1a026c122678429e6 fixes the problem of not logging terminal in case stdin & stdout are not a terminal. The problem is that we are then trying to pass-through stdin content by reading from the apt-process stdin and writing it to the stdin of the child (dpkg), which works great for users who can control themselves, but pipes and co are a bit less forgiving causing us to pass everything to the first child process, which if the sending part of the pipe is e.g. 'yes' we will never see the end of it (as the pipe is full at some point and further writing blocks). There is a simple solution for that of course: If stdin isn't a terminal, we us the apt-process stdin as stdin for the child directly (We don't do this if it is a terminal to be able to save the typed input in the log). Closes: 773061
2014-12-23always run 'dpkg --configure -a' at the end of our dpkg callingsDavid Kalnischkies
dpkg checks now for dependencies before running triggers, so that packages can now end up in trigger states (especially those we are not touching at all with our calls) after apt is done running. The solution to this is trivial: Just tell dpkg to configure everything after we have (supposely) configured everything already. In the worst case this means dpkg will have to run a bunch of triggers, usually it will just do nothing though. The code to make this happen was already available, so we just flip a config option here to cause it to be run. This way we can keep pretending that triggers are an implementation detail of dpkg. --triggers-only would supposely work as well, but --configure is more robust in regards to future changes to dpkg and something we will hopefully make use of in future versions anyway (as it was planed at the time this and related options were implemented). Note that dpkg currently has a workaround implemented to allow upgrades to jessie to be clean, so that the test works before and after. Also note that test (compared to the one in the bug) drops the await test as its is considered a loop by dpkg now. Closes: 769609
2014-12-23do not make PTY slave the controlling terminalDavid Kalnischkies
If we have no controlling terminal opening a terminal will make this terminal our controller, which is a serious problem if this happens to be the pseudo terminal we created to run dpkg in as we will close this terminal at the end hanging ourself up in the process… The offending open is the one we do to have at least one slave fd open all the time, but for good measure, we apply the flag also to the slave fd opening in the child process as we set the controlling terminal explicitely here. This is a regression from 150bdc9ca5d656f9fba94d37c5f4f183b02bd746 with the slight twist that this usecase was silently broken before in that it wasn't logging the output in term.log (as a pseudo terminal wasn't created). Closes: 772641
2014-12-22dispose http(s) 416 error page as non-contentDavid Kalnischkies
Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. (The tests and their binary helpers had to be slightly modified to apply, but the patch to fix the issue itself is unchanged.) Closes: 768797
2014-12-10do not make PTY slave the controlling terminalDavid Kalnischkies
If we have no controlling terminal opening a terminal will make this terminal our controller, which is a serious problem if this happens to be the pseudo terminal we created to run dpkg in as we will close this terminal at the end hanging ourself up in the process… The offending open is the one we do to have at least one slave fd open all the time, but for good measure, we apply the flag also to the slave fd opening in the child process as we set the controlling terminal explicitely here. This is a regression from 150bdc9ca5d656f9fba94d37c5f4f183b02bd746 with the slight twist that this usecase was silently broken before in that it wasn't logging the output in term.log (as a pseudo terminal wasn't created). Closes: 772641
2014-12-09dispose http(s) 416 error page as non-contentDavid Kalnischkies
Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. Closes: 768797
2014-12-07always run 'dpkg --configure -a' at the end of our dpkg callingsDavid Kalnischkies
dpkg checks now for dependencies before running triggers, so that packages can now end up in trigger states (especially those we are not touching at all with our calls) after apt is done running. The solution to this is trivial: Just tell dpkg to configure everything after we have (supposely) configured everything already. In the worst case this means dpkg will have to run a bunch of triggers, usually it will just do nothing though. The code to make this happen was already available, so we just flip a config option here to cause it to be run. This way we can keep pretending that triggers are an implementation detail of dpkg. --triggers-only would supposely work as well, but --configure is more robust in regards to future changes to dpkg and something we will hopefully make use of in future versions anyway (as it was planed at the time this and related options were implemented). Closes: 769609
2014-12-07correct architecture detection for 'rc' packages for purgeDavid Kalnischkies
We were already considering these cases, but the code was flawed, so that packages changing architectures are incorrectly handled and hence the wrong architecture is used to call dpkg with, so that dpkg says the package isn't installed (which it isn't for the requested architecture). Closes: 770898
2014-12-07properly handle already reinstall pkgs in orderingDavid Kalnischkies
The bugreport itself describes the case of the ordering code detecting a loop where none is present, but the testcase finds also cases in which there is actually a loop and we fail to realize it. --reinstall can be considered an interactive command through and it usually doesn't encounter such "hard" problems (= looping essentials), so this is less serious than it sounds at first. Closes: 770291
2014-11-28support long keyids in "apt-key del" instead of ignoring themJames McCoy
apt-key given a long keyid reports just "OK" all the time, but doesn't delete the mentioned key as it doesn't find the key. Note: In debian/experimental this was closed with 29f1b977100aeb6d6ebd38923eeb7a623e264ffe which just added the testcase as the rewrite of apt-key had fixed this as well. Closes: 754436
2014-11-28fix PTY interaction on linux and kfreebsdDavid Kalnischkies
We run dpkg on its own pty, so we can log its output and have our own output around it (like the progress bar), while also allowing debconf and configfile prompts to happen. In commit 223ae57d468fdcac451209a095047a07a5698212 we changed to constantly reopening the slave for kfreebsd. This has the sideeffect though that in some cases slave and master will lose their connection on linux, so that no output is passed along anymore. We fix this by having always an fd referencing the slave open (linux), but we don't use it (kfreebsd). Failing to get our PTY up and running has many (bad) consequences including (not limited to, nor all at ones or in any case) garbled ouput, no output, no logging, a (partial) mixture of the previous items, … This commit is therefore also reshuffling quiet a bit of the creation code to get especially the output part up and running on linux and the logging for kfreebsd. Note that the testcase tries to cover some cases, but this is an interactivity issue so only interactive usage can really be a good test. Closes: 765687
2014-11-28close leaking slave fd after setting up pty magicDavid Kalnischkies
The fd moves out of scope here anyway, so we should close it properly instead of leaking it which will tickle down to dpkg maintainer scripts. Closes: 767774
2014-11-19use dpkg --merge-avail only if needed in apt-markDavid Kalnischkies
Only "recent" versions of dpkg support stdin for merge instead of a file, so as a quick fix we delay calling it until we really need it which fixes most of the problem already. Checking for a specific dpkg version here is deemed too much work, just like using a temporary file here and depends a too high requirement for this minor usecase. After all, it didn't work at all before, so we break nobody here and can fix it if someone complains (with a patch).
2014-11-19fix test to support non-multiarch dpkg versionsDavid Kalnischkies
On travis we work with a pre-multiarch version of dpkg, so the output is slightly different in regards to package names. Git-Dch: Ignore
2014-11-19fix PTY interaction on linux and kfreebsdDavid Kalnischkies
We run dpkg on its own pty, so we can log its output and have our own output around it (like the progress bar), while also allowing debconf and configfile prompts to happen. In commit 223ae57d468fdcac451209a095047a07a5698212 we changed to constantly reopening the slave for kfreebsd. This has the sideeffect though that in some cases slave and master will lose their connection on linux, so that no output is passed along anymore. We fix this by having always an fd referencing the slave open (linux), but we don't use it (kfreebsd). Failing to get our PTY up and running has many (bad) consequences including (not limited to, nor all at ones or in any case) garbled ouput, no output, no logging, a (partial) mixture of the previous items, … This commit is therefore also reshuffling quiet a bit of the creation code to get especially the output part up and running on linux and the logging for kfreebsd. Note that the testcase tries to cover some cases, but this is an interactivity issue so only interactive usage can really be a good test. Closes: 765687
2014-11-18close leaking slave fd after setting up pty magicDavid Kalnischkies
The fd moves out of scope here anyway, so we should close it properly instead of leaking it which will tickle down to dpkg maintainer scripts. Closes: 767774
2014-11-18fix file ownership tests to work on kfreebsdDavid Kalnischkies
While on linux files are created in /tmp with $USER:$USER, on my kfreebsd testmachine they are created with $USER:root, so we pull some strings here to make it work on both.
2014-11-18create our cache and lib directory always with mode 755David Kalnischkies
We autocreate for a while now the last two directories in /var/lib/apt/lists (similar for /var/cache/apt/archives) which is very nice for systems having any of those on tmpfs or other non-persistent storage. This also means though that this creation is effected by the default umask, so for people with aggressive umasks like 027 the directories will be created with 750, which means all non-root users are left out, which is usually exactly what we want then this umask is set, but the cache and lib directories contain public knowledge. There isn't any need to protect them from viewers and they render apt completely useless if not readable.