summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2015-05-12implement VerifyFile as all-hashes checkDavid Kalnischkies
It isn't used much compared to what the methodname suggests, but in the remaining uses it can't hurt to check more than strictly necessary by calculating and verifying with all hashes we can compare with rather than "just" the best known hash.
2015-05-12detect 416 complete file in partial by expected hashDavid Kalnischkies
If we have the expected hashes we can check with them if the file we have in partial we got a 416 for is the expected file. We detected this with same-size before, but not every server sends a good Content-Range header with a 416 response.
2015-05-11rewrite all TFRewrite instances to use the new pkgTagSection::WriteDavid Kalnischkies
While it is mostly busywork to rewrite all instances it actually fixes bugs as the data storage used by the new method is std::string rather than a char*, the later mostly created by c_str() from a std::string which the caller has to ensure keeps in scope – something apt-ftparchive actually didn't ensure and relied on copy-on-write behavior instead which c++11 forbids and hence the new default gcc abi doesn't use it.
2015-05-11implement a more c++-style TFRewrite alternativeDavid Kalnischkies
TFRewrite is okay, but it has obscure limitations (256 Tags), even more obscure bugs (order for renames is defined by the old name) and the interface is very c-style encouraging bad usage like we do it in apt-ftparchive passing massive amounts of c_str() from std::string in. The old-style is marked as deprecated accordingly. The next commit will fix all places in the apt code to not use the old-style anymore.
2015-05-11stop depending on copy-on-write for std::stringDavid Kalnischkies
In 66c3875df391b1120b43831efcbe88a78569fbfe we workaround/fixed a problem where the code makes the assumption that the compiler uses copy-on-write implementations for std::string. Turns out that for c++11 compatibility gcc >= 5 will stop doing this by default.
2015-05-11sync TFRewrite*Order arrays with dpkg and dakDavid Kalnischkies
dpkg and dak know various field names and order them in their output, while we have yet another order and have to play catch up with them as we are sitting between chairs here and neither order is ideal for us, too. A little testcase is from now on supposed to help ensureing that we do not derivate to far away from which fields dpkg knows and orders.
2015-05-11fix 'Source' to 'Package' rename in apt-ftparchiveDavid Kalnischkies
This rename with value is ordered by the 'old' name 'Source', but should be ordered by the new name… by splitting the operation in a delete and a new field we can easily fix this problem locally for now.
2015-05-11drop incorrect parameter implicitely converted to boolDavid Kalnischkies
The helper expects to be told if it should generate messages, not where these messages should be printed – as it isn't printing such messages, but puts them in _error. apt-get uses in other methods a helper specialisation which does also print stuff to a stream through, so this is likely a copy&paste error. Git-Dch: Ignore
2015-05-11fix macro definition for very old GCC < 3David Kalnischkies
Git-Dch: Ignore
2015-05-11show non-matching m-a:same versions in debug messageDavid Kalnischkies
Slightly rewriting the code to ensure we only use two sources for the versions as it could otherwise be confusing to look at.
2015-05-11remove available file to have same dpkg -l behaviorDavid Kalnischkies
dpkg -l < 1.16.2 loads the available file and hence sees a package which later versions do not see, leading to failures on travis-ci. The different versions also have slightly different messages. Git-Dch: Ignore
2015-05-11remove unused and strange default-value for pinsDavid Kalnischkies
If the pin for a generic pin is 0, it get a value by strange looking rules, if the pin is specific the rules are at least not strange, but the value 989 is a magic number without any direct meaning… but both never happens in practice as the parsing skips such entries with a warning, so there always is a priority != 0 and the code therefore never used.
2015-05-11a pin of 1000 always means downgrade allowedDavid Kalnischkies
The documentation says this, but the code only agreed while evaluating specific packages, but not generics. These needed a pin above 1000 to have the same effect. The code causing this makes references to a 'second pesduo status file', but nowhere is explained what this might stand for and/or what it was, so we do the only reasonable thing: Remove all references and do as documented.
2015-05-11do not require installed libapt-pkg-dev for gtestDavid Kalnischkies
Git-Dch: Ignore
2015-05-11improve partial/ cleanup in abort and failure casesDavid Kalnischkies
Especially pdiff-enhanced downloads have the tendency to fail for various reasons from which we can recover and even a successful download used to leave the old unpatched index in partial/. By adding a new method responsible for making the transaction of an individual file happen we can at specialisations especially for abort cases to deal with the cleanup. This also helps in keeping the compressed indexes around if another index failed instead of keeping the decompressed files, which we wouldn't pick up in the next call.
2015-04-19Merge branch 'debian/jessie' into debian/experimentalDavid Kalnischkies
Conflicts: apt-pkg/acquire-item.cc cmdline/apt-key.in methods/https.cc test/integration/test-apt-key test/integration/test-multiarch-foreign
2015-04-19hide first pdiff merge failure debug messageDavid Kalnischkies
The sibling of this message are all guarded as debug messages, just this one had it missing an subsequently causes display issues if triggered. Git-Dch: Ignore
2015-04-19a hit on Release files means the indexes will be hits tooDavid Kalnischkies
If we get a IMSHit for the Transaction-Manager (= the InRelease file or as its still supported fallback Release + Release.gpg combo) we can assume that every file we would queue based on this manager, but already have locally is current and hence would get an IMSHit, too. We therefore save us and the server the trouble and skip the queuing in this case. Beside speeding up repetative executions of 'apt-get update' this way we also avoid hitting hashsum errors if the indexes are in fact already updated, but the Release file isn't yet as it is the case on well behaving mirrors as Release files is updated last. The implementation is a bit harder than the theory makes it sound as we still have to keep reverifying the Release files (e.g. to detect now expired once to avoid an attacker being able to silently stale us) and have to handle cases in which the Release file hits, but some indexes aren't present (e.g. user added a new foreign architecture).
2015-04-19refactor calculation of final lists/ name from URIDavid Kalnischkies
Calculating the final name of an item which it will have after everything is done and verified successfully is suprisingly complicated as while they all follow a simple pattern, the URI and where it is stored varies between the items. With some (abibreaking) redesign we can abstract this similar to how it is already down for the partial file location. Git-Dch: Ignore
2015-04-19unsigned Release files can expire, tooDavid Kalnischkies
Checking Valid-Until on an unsigned Release file doesn't give us any security brownie points as an attacker could just change the date and in practice repositories with unsigned Release files will very likely not have a Valid-Until date, but for symetry and the fact that being unsigned is currently just a warning, while expired is a fatal error.
2015-04-19ensure lists/ files have correct permissions after apt-cdrom addDavid Kalnischkies
Its a bit unpredictable which permissons and owners we will encounter on a CD-ROM (or a USB stick, as apt-cdrom is responsible for those too), so we have to ensure in this codepath as well that everything is nicely setup without waiting for a 'apt-get update' to fix up the (potential) mess.
2015-04-19calculate hashes while downloading in httpsDavid Kalnischkies
We do this in HTTP already to give the CPU some exercise while the disk is heavily spinning (or flashing?) to store the data avoiding the need to reread the entire file again later on to calculate the hashes – which happens outside of the eyes of progress reporting, so you might ended up with a bunch of https workers 'stuck' at 100% while they were busy calculating hashes. This is a bummer for everyone using apt as a connection speedtest as the https method works slower now (not really, it just isn't reporting done too early anymore).
2015-04-19Italian program translation updateMilo Casagrande
Closes: 782122
2015-04-19calculate only expected hashes in methodsDavid Kalnischkies
Methods get told which hashes are expected by the acquire system, which means we can use this list to restrict what we calculate in the methods as any extra we are calculating is wasted effort as we can't compare it with anything anyway. Adding support for a new hash algorithm is therefore 'free' now and if a algorithm is no longer provided in a repository for a file, we automatically stop calculating it. In practice this results in a speed-up in Debian as we don't have SHA512 here (so far), so we practically stop calculating it.
2015-04-19if we can, use gccs __builtin_swap methodsDavid Kalnischkies
Git-Dch: Ignore
2015-04-19use SHA2 unrolled code as its a bit fasterDavid Kalnischkies
Upstream claims its faster if combined with an optimizing compiler and I can confirm that in some tests, so lets see how it works out in practice. Git-Dch: Ignore
2015-04-19handle servers closing encoded connections correctlyDavid Kalnischkies
Servers who advertise that they close the connection get the 'Closes' encoding flag, but this conflicts with servers who response with a transfer-encoding (e.g. encoding) as it is saved in the same flag. We have a better flag for the keep-alive (or not) of the connection anyway, so we check this instead of the encoding. This is in practice not much of a problem as real servers we talk to are HTTP1.1 servers (with keep-alive) and there isn't much point in doing chunked encoding if you are going to close anyway, but our simple testserver stumbles over this if pressed and its a bit cleaner, too. Git-Dch: Ignore
2015-04-19send Alt-* info for uncompressed based on any compressionsDavid Kalnischkies
file sends information about the uncompressed file if it can find it as well as for the compressed file. This was done only for gzip so far, but we support more compression types. That this information isn't used a lot is a different story. Git-Dch: Ignore
2015-04-19remove duplicated check for same file copyDavid Kalnischkies
Git-Dch: Ignore
2015-04-19improve https method queue progress reportingDavid Kalnischkies
The worker expects that the methods tell him when they start or finish downloading a file. Various information pieces are passed along in this report including the (expected) filesize. https was using a "global" struct for reporting which made it 'reuse' incorrect values in some cases like a non-existent InRelease fallbacking to Release{,.gpg} resulting in a size-mismatch warning. Reducing the scope and redesigning the setting of the values we can fix this and related issues. Closes: 777565, 781509 Thanks: Robert Edmonds and Anders Kaseorg for initial patchs
2015-04-19do not unlink https file on general errorDavid Kalnischkies
It might be quite interesting which file (content) made curl freak out and other methods keep the file around as well. Git-Dch: Ignore
2015-04-19demote missing gtest to a buildtime warningDavid Kalnischkies
We just need it for unit tests and our debian/rules file actually skips calling them if nocheck is given… but this fails anyhow as we declared a hard-dependency on it. Demoting the error to a warning in configuration and adding a test in the 'make test' path with a friendly message allows nocheck to be useful again. (Running unit tests is fully encouraged of course, but bootstrappers and co do not need to be burdened with this stuff)
2015-04-13release 1.0.9.8David Kalnischkies
2015-04-12parse specific-arch dependencies correctly on single-arch systemsDavid Kalnischkies
On single-arch the parsing was creating groupnames like 'apt:amd64' even through it should be 'apt' and a package in it belonging to architecture amd64. The result for foreign architectures was as expected: The dependency isn't satisfiable, but for native architecture it means the wrong package (ala apt:amd64:amd64) is linked so this is also not satisfiable, which is very much not expected. No longer excluding single-arch from this codepath allows the generation of the correct links, which still link to non-exisiting packages for foreign dependencies, but natives link to the expected native package just as if no architecture was given. For negative arch-specific dependencies ala Conflicts this matter was worse as apt will believe there isn't a Conflict to resolve, tricking it into calculating a solution dpkg will refuse. Architecture specific positive dependencies are rare in jessie – the only one in amd64 main is foreign –, negative dependencies do not even exist. Neither class has a native specimen, so no package in jessie is effected by this bug, but it might be interesting for stretch upgrades. This also means the regression potential is very low. Closes: 777760
2015-04-10test for or-group choice behaviour in upgradeDavid Kalnischkies
In #780028 we were discussing how the or-group order should be more important than keep-back decisions of 'upgrade'. We have this behaviour, but to ensure it stays this way lets add a test for it. Git-Dch: Ignore
2015-04-10add a simple unit test for acquire progressDavid Kalnischkies
This isn't testing much of the 'complex' parts, but its better than nothing for now. Git-Dch: Ignore
2015-04-10reimplement the last uses of sprintfDavid Kalnischkies
Working with strings c-style is complicated and error-prune, so by converting to c++ style we gain some simplicity and avoid buffer overflows by later extensions. Git-Dch: Ignore
2015-04-07keyids in "apt-key del" should be case-insensitiveDavid Kalnischkies
gnupg is case-insensitive about keyids, so back then apt-key called it directly any keyid was accepted, but now that we work more with the keyid ourself we regressed to require uppercase keyids by accident. This is also inconsistent with other apt-key commands which still use gnupg directly. A single case-insensitive grep and we are fine again. Closes: 781696
2015-04-07demote VectorizeString gcc attribute from const to pureDavid Kalnischkies
g++-5 generates a slightly broken libapt which doesn't split architecture configurations correctly resulting in e.g. Packages files requested for the bogus architecture 'amd64,i386' instead of for amd64 and i386. The reason is an incorrectly applied attribute marking the function as const, while functions with pointer arguments are not allowed to be declared as such (note that char& is a char* in disguise). Demoting the attribute to pure fixes this issue – better would be dropping the & from char but that is an API change… Neither earlier g++ versions nor clang use this attribute to generate broken code, so we don't need a rebuild of dependencies or anything and g++-5 isn't even included in jessie, but the effect is so strange and apt popular enough to consider avoiding this problem anyhow.
2015-04-07fix crash in order writing in pkgDPkgPM::WriteApportReport()Michael Vogt
libapt can be configured to write various bits of information to a file creating a report via apport. This is disabled by default in Debian and apport residing only in /experimental so far, but Ubuntu and other derivatives have this (in some versions) enabled by default and there is no regression potentially here. The crash is caused by a mismatch of operations vs. strings for operations, so adding the missing strings for these operations solves the problem. [commit message by David Kalnischkies] LP: #1436626
2015-04-07avoid depends on std::string implementation for pkgAcquire::Item::ModeDavid Kalnischkies
In /experimental this is resolved by deprecating Mode and moving to a new std::string, but that breaks ABI of course, so that was out of question. We can't change to a malloc/free style c-string either as Mode is public and hence a library user could be setting this as well. std::string implementors actually helped us out here with copy-on-write which means that while the variable "obviously" runs out of scope here, in reality you get the correct result as the string we work with here comes from the configuration in which it is still valid. Such a dependency on magic is bad of course, but its still interesting that only python3 seems to have an issue with it… With some silly explicit if-else assigning we can sidestep this issue while retaining the same output for 99.99% of all users (= noone actually configures additional compression algorithms which are also provided by repositories…), but even for these 0.01% its just a small change in the display as Mode can not be used for anything else. Example: apt/aptitude uses it in its 'update' implementations in the one-line progress at the bottom for specific items. Closes: 781858
2015-04-07properly handle expected filesize in httpsDavid Kalnischkies
The worker expects that the methods tell him when they start or finish downloading a file. Various information pieces are passed along in this report including the (expected) filesize. https is using a "global" struct for reporting which made it 'reuse' incorrect values in some cases like a non-existent InRelease fallbacking to Release{,.gpg} resulting in an incorrect size-mismatch warning scaring and desensitizing users as well as being subject to a race between the write_data and progress callbacks generating incorrect progress reporting and potentially the same error message. Other branches as well as the bugreports contain 'better' fixes making the struct local and other sensible changes, but are larger as a result, so in this version we opted for short diff with minimal effect above else instead. Closes: 777565, 781509 Thanks: Robert Edmonds and Anders Kaseorg for initial patchs
2015-04-07fix another d(e)select-upgrade typoDavid Kalnischkies
You would think one instance of this is enough, but 80e8d923ebc8d5f3f84eb3f922b28ca309c25026 wasn't as globally applied as the commit message suggested… LP: #1399037
2015-03-16stop displaying time of build in online helpJérémy Bobbio
As part of the “reproducible builds” effort [1], we have noticed that apt could not be built reproducibly. One issue is that it uses the __DATE__ and __TIME__ macros of the C preprocessor to display the time of build in the online help. We believe this information not to be really useful to users as they can always look at the package data and metadata to figure it out. The attached patch simply removes this information. All non-documentation packages can then be built reproducibly with our current experimental framework. [David: changed the string slightly to be untranslateable as well] Closes: 774342
2015-03-16Italian manpage translation updateBeatrice Torracca
Closes: 776702
2015-03-16test exitcode as well as string equalityDavid Kalnischkies
We use test{success,failure} now all over the place in the framework, so its only consequencial to do this in the situations in which we test for a specific output as well. Git-Dch: Ignore
2015-03-16parse arch-qualified Provides correctlyHelmut Grohne
The underlying problem is that libapt-pkg does not correctly parse these provides. Internally, it creates a version named "baz:i386" with architecture amd64. Of course, such a package name is invalid and thus this version is completely inaccessible. Thus, this bug should not cause apt to accept a broken situation as valid. Nevertheless, it prevents using architecture qualified depends. Closes: 777071
2015-03-16set version before updating po filesDavid Kalnischkies
This way the 'correct' version is carried over into the po files to reflect which version they were built for rather than the version before the current one. Git-Dch: Ignore
2015-03-16(error) Same iterator is used with different containersDavid Kalnischkies
cppcheck reports this error, its not really a problem for us as the API can actually deal with it via implicit conversion, but being explicit can't hurt and the less reported errors the better. Git-Dch: Ignore
2015-03-16fix some new compiler warnings reported by gcc-5David Kalnischkies
Git-Dch: Ignore