summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2016-04-25use the same redirection mirror for all index filesDavid Kalnischkies
Redirection services like httpredir.debian.org tend to use a set of mirrors from which they pick a mirror at "random" for each requested file, which is usually benefitial for the download of debs, but for the index files this can quickly cause problems (aka hashsum mismatches) if the two (or more) mirrors involved are only slightly out-of-sync. This commit "resolves" this issue by using the mirror we ended up using to get the (signed) Release file directly to get the index files belonging to this Release file instead of asking the redirection service which eliminates the risk of hitting out-of-sync mirrors. As an obvious downside the redirection service can't serve partial mirrors anymore for indexes and the download of indexes indexed in the same Release file can't be done in parallel (from different mirrors). This does not effect the download of non-index files like deb-files as out-of-sync mirrors aren't a huge problem there, so the parallel download outweights a potentially 404 error (also because this causes no errenous downloads while hashsum mismatches download the entire file before finding out that it was pointless). The rational for this is that indexes are relative to the Release file. If we would be talking about a HTML page including images, such a behaviour is obvious and intended – not doing it means in the best case a bunch of "useless" requests which will all be answered with a redirect.
2016-04-25show more details for "Writing more data" errors, tooDavid Kalnischkies
They are the small brothers of the hashsum mismatch, so they deserve a similar treatment even through we have for architectual reasons not a much to display as for hashsum mismatches for now.
2016-04-25show more details for "Hash Sum mismatch" errorsDavid Kalnischkies
Users tend to report these errors with just this error message… not very actionable and hard to figure out if this is a temporary or 'permanent' mirror-sync issue or even the occasional apt bug. Showing the involved hashsums and modification times should help in triaging these kind of bugs – and eventually we will have less of them via by-hash. The subheaders aren't marked for translation for now as they are technical glibberish and probably easier to deal with if not translated. After all, our iconic "Hash Sum mismatch" is translated at least. These additions were proposed in #817240 by Peter Palfrader.
2016-04-25format multiline errors properly in acquire progressDavid Kalnischkies
Together with the GlobalError change this allows us to add errors spanning multiple lines, just that we control GlobalError while the acquire progress is dealt with potentially by individual clients which might or might not need to be adapted. This isn't critical through as it either just works as expected anyhow or is a minor styling thing (after all, all this commit does it add two spaces to indent the lines a bit…).
2016-04-25drop empty line from fetch errorDavid Kalnischkies
This is a duplicate of sorts of 0efb29eb36184bbe6de7b1013d1898796d94b171 which is the a lot more frequent case of this error – and also a duplicate of this error message, just without the \n at the end. Git-Dch: Ignore
2016-04-25properly format multiline error messagesDavid Kalnischkies
2016-04-25sanify unused ReportMirrorFailure a tiny bitDavid Kalnischkies
Calling the (non-existent) reporter multiple times for the same error with different codes for the same error (e.g. hashsum) is a bit strange. It also doesn't need to be a public API. Ideally that would all look and behave slightly different, but we will worry about that at the time this is actually (planed to be) used somewhere… Git-Dch: Ignore
2016-04-25don't ask server if we have entire file in partial/David Kalnischkies
We have this situation in cases were parts of the transaction are refused (e.g. in a hashsum mismatch) and rerun the update (e.g. in the hope that we get a mirror which is synced this time). Previously we would ask the server with an if-range and in the best case recieve a 416 in response (less featureful server might end up giving us the entire file again or we get the wrong file this time giving us a hashsum mismatch…), which is a waste of time if we know already by checking the hashsums that we got the complete and correct file.
2016-04-25add dep11 files to default Release patternsDavid Kalnischkies
2016-04-25make random acquire queues work less randomDavid Kalnischkies
Queues feeding workers like rred are created in a random pattern to get a few of them to run in parallel – but if we already have an idling queue we don't need to assign it to a (potentially new) random queue as that saves us the (agruably small) overhead of starting up a new queue, avoids adding jobs to an already busy queue while others idle and as a bonus reduces the size of debug logs a bit. We also keep starting new queues now until we reach our limit before we assign work at random to them, which should give us a more effective utilisation overall compared to potentially adding work to busy queues while we haven't reached our queue limit yet.
2016-04-25Release 1.2.111.2.11Julian Andres Klode
2016-04-14ensure outdated files are dropped without lists-cleanupDavid Kalnischkies
Tested via (newly) empty index files, but effects also files dropped from the repository or an otherwise changed repository config.
2016-04-14silently skip acquire of empty index filesDavid Kalnischkies
There is just no point in taking the time to acquire empty files – especially as it will be tiny non-empty compressed files usually.
2016-04-14allow uncompressed files to be empty in store againDavid Kalnischkies
With the previous fix for file applied we can again hit repositories which contain uncompressed empty files, which since the introduction of the central store: method wasn't accounted for anymore as we forbid empty compressed files.
2016-04-14fix Alt-Filename handling of file methodDavid Kalnischkies
A silly of-by-one error in the stripping of the extension to check for the uncompressed filename broken in an attempt to support all compressions in commit a09f6eb8fc67cd2d836019f448f18580396185e5. Fixing this highlights also mistakes in the handling of the Alt-Filename in libapt which would cause apt to remove the file from the repository (if root has the needed rights – aka the disk isn't readonly or similar)
2016-04-13recheck Pre-Depends satisfaction in SmartConfigureDavid Kalnischkies
Regression introduced in commit 590f1923121815b36ef889033c1c416a23cbe9a2 (2011!) causing apt to not check if Pre-Depends are satisfied before calling --configure. This managed to hide so perfectly well for years as Pre-Depends aren't that common, apt prefers upgrading these packages first and checks for satisfaction is already in SmartUnpack, so there is only a small window of oppertunity to break a pre-dependency relation (usually with an unpack). Verified by logchecking with two provided status files in the buglog. I would have liked to write a test, but I wasn't able to reach the needed complexity to get apt to fail – but the change is small and reasonable, so what could possible go wrong™, right? LP: #1569099
2016-04-13detect compressed status files on extension againDavid Kalnischkies
It handy to be able to point apt at reading a compressed dpkg/status file in debugging cases, which worked pre-1.1 but somewhere down the line in the massive refactoring. Restoring this behavior in a central place for all realfile index files instead of just for the status file. (This has no effect on index files acquired from an archive – those are handled by different classes and support compressed files just fine)
2016-04-13Hungarian program translation updateKelemen Gábor
Closes: 820638
2016-04-13do not require non-broken systems in 'upgrade'David Kalnischkies
There is a good chance that the attempt will fail, but if a user mentions certain packages explicitly on the commandline there is a chance that this will consist of a broken system which is resolved by upgrading more packages then just the mentioned. This limitation was not effecting external resolvers.
2016-04-13webserver: 416 errors aren't closing connectionsDavid Kalnischkies
Breaking here lets our handler die which a client will fix by reconnecting… but that eats time needlessly and is simple the wrong handling, too. Git-Dch: Ignore
2016-04-07stop handling items in doomed transactionsDavid Kalnischkies
With the previous commit we track the state of transactions, so we can now use our knowledge to avoid processing data for a transaction which was already closed (via an abort in this case). This is needed as multiple independent processes are interacting in the process, so there isn't a simple immediate full-engine stop and it would also be bad to teach each and every item how to check if its manager has failed subordinate and what to do in that case. In the pdiff case, which deals (potentially) with many items during its lifetime e.g. a hashsum mismatch in another file can abort the transaction the file we try to patch via pdiff belongs to. This causes some of the items (which are already done) to be aborted with it, but items still in the process of acquisition continue in the processing and will later try to use all the items together failing in strange ways as cleanup already happened. The chosen solution is to dry up the communication channels instead by ignoring new requests for data acquisition, canceling requests which are not assigned to a queue and not calling Done/Failed on items anymore. This means that e.g. already started or pending (e.g. pipelined) downloads aren't stopped and continue as normal for now, but they remain in partial/ and aren't processed further so the next update command will pick them up and put them to good use while the current process fails updating (for this transaction group) in an orderly fashion. Closes: 817240 Thanks: Barr Detwix & Vincent Lefevre for log files
2016-04-07ensure transaction states are changed only onceDavid Kalnischkies
We want to keep track of the state of a transaction overall to base future decisions on it, but as a pre-requirement we have to make sure that a transaction isn't commited twice (which happened if the download of InRelease failed and Release takes over). It also happened to create empty commits after a transaction was already aborted in cases in which the Release files were rejected. This isn't effecting security at the moment, but to ensure this isn't happening again and can never be bad a bunch of fatal error messages are added to make regressions on this front visible.
2016-04-05releasing package apt version 1.2.101.2.10Michael Vogt
2016-04-04Fix `OnCalendar=*-*-* 6,18:00` instead of `OnCalendar=*-*-* 6:00,18:00`Michael Vogt
See https://github.com/systemd/systemd/issues/2956 - it appears that `OnCalendar=*-*-* 6,18:00` and `OnCalendar=*-*-* 6:00,18:00` are quite different. Git-Dch: ignore
2016-04-04Japanese program translation updateTakuma Yamada
Closes: 819938
2016-04-03don't leak on error in listparser creationDavid Kalnischkies
Git-Dch: Ignore Reported-By: gcc -fsanitize=address
2016-04-03use buffered writing for InRelease splittingDavid Kalnischkies
Hardly noticeable, but given that we have the option to easily enable it, lets enable it as every newline in the message is written individually by the code.
2016-04-03consistently add APT_OVERRIDE as method markerDavid Kalnischkies
Some methods had it missing, some used the keyword directly, which isn't a problem as it is a cc file, but for consistency lets stick to our macro for now. Git-Dch: Ignore
2016-04-01Use systemd.timer instead of a cron jobMichael Vogt
The rational is that we need to spread the load on the mirrors that apt update and unattended-upgrades cause. To do so, we leverage the RandomizeDelay feature of systemd. The other advantage is that the timer is not run at a fixed daily.daily time but instead every 24h. This also fixes the problem that the randomized deplay in the current apt.cron.daily causes other cron jobs to be deplayed. A compatibility cron job is also provided for systems that do not use systemd. Note that the time is fired two times a day, but the logic inside of apt.systemd.daily will ensure (via stamp files) that the servers are hit at most every 24h. Firing two times a day helps with the worst case update time and it also helps with systems that are not always on. LP: #246381, #727685 Closes: #600262, #709675, #663290
2016-03-28Allow lowering trust level of a hash via configJulian Andres Klode
Introduces APT::Hashes::<NAME> with entries Untrusted and Weak which can be set to true to cause the hash to be treated as untrusted and/or weak.
2016-03-27test-apt-update-reporting: Make more use of frameworkJulian Andres Klode
Use msgtest and testsuccess with a function instead of failing with a simple exit 1. This looks nicer. Gbp-Dch: ignore
2016-03-27test-acquire-same-file-multiple-times: Delete files before retryingJulian Andres Klode
This gets rid of byte-range requests and 416 responses. Gbp-Dch: ignore
2016-03-27test-apt-download-progress: Use a larger file for testingJulian Andres Klode
This should make the test less flaky, as with a small file, we might have already received all the data before trying to apply rate limits which is a constant source of failure on the i386 Ubuntu autopkgtest.
2016-03-27zh_CN.po: update simplified Chinese translation. (100%)Zhou Mo
2016-03-27Release 1.2.91.2.9Julian Andres Klode
2016-03-27Do not mark packages for keep that we want to removeJulian Andres Klode
If the package is marked for removal, keep it marked for removal and do not mark it for keep. If we mark it for keep, we some how later get to a different stage where it is marked for unpack instead of removal. In the example in the bug report, we would get a: SmartUnPack maas-region-controller-min:amd64 (replace version 2.0.0~alpha3+bzr4810-0ubuntu1 with Segmentation fault maas-region-controller-min:amd64 was marked for removal, but we changed it to keep and somehow it thinks that this is to be replaced now instead of removed (probably because the InstallVer != CandidateVer [with InstallVer = 0]). This fixes a regression introduced in release 1.2.7, commit: 0390edd5452b081f8efcf412f96d535a1d959457 Reported-by: LaMont Jones on IRC LP: #1562402
2016-03-25drop confusing comma from no strong hash messageDavid Kalnischkies
2016-03-24Release 1.2.81.2.8Julian Andres Klode
2016-03-24Update symbols fileJulian Andres Klode
2016-03-24Set hidden visibility for StringViewJulian Andres Klode
This avoids templates using StringView to be exported, such as std::vector<StringView*>::emplace_back(). Gbp-Dch: ignore
2016-03-24Japanese manpage translation updateTakuma Yamada
Closes: 818950
2016-03-22handle gpgv's weak-digests ERRSIGDavid Kalnischkies
Our own gpgv method can declare a digest algorithm as untrusted and handles these as worthless signatures. If gpgv comes with inbuilt untrusted (which is called weak in official terminology) which it e.g. does for MD5 in recent versions we should handle it in the same way. To check this we use the most uncommon still fully trusted hash as a configureable one via a hidden config option to toggle through all of the three states a hash can be in.
2016-03-21properly check for "all good sigs are weak"David Kalnischkies
Using erase(pos) is invalid in our case here as pos must be a valid and derefenceable iterator, which isn't the case for an end-iterator (like if we had no good signature). The problem runs deeper still through as VALIDSIG is a keyid while GOODSIG is just a longid so comparing them will always fail. Closes: 818910
2016-03-19refactor loading of previous release fileDavid Kalnischkies
There is really no need to have the same code three times. Git-Dch: Ignore
2016-03-19tests: reenable basic auth test and add @ in usernameDavid Kalnischkies
On launchpad #1558484 a user reports that @ in the authentication tokens parsing of sources.list isn't working in an older (precise) version. It isn't the recommended way of specifying passwords and co (auth.conf is), but we can at least test for regressions (and in this case test at all… who was that "clever" boy disabling a test with exit……… oh, nevermind. Git-Dch: Ignore
2016-03-19zh_CN.po: update simplified Chinese translation. (Closes: #818639)Zhou Mo
100% translated. Note: this commit contains a message from the future. #818639
2016-03-19cachefile: Only set members that were initialized successfullyJulian Andres Klode
Otherwise, things will just start failing later down the stack, because (a) the lazy getters do not check if building was successful and (b) any further getter call would return the invalid object anyway. Also initialize VS in pkgCache to nullptr by default. Closes: #818628
2016-03-19zh_CN.po: update simplified chinese translation.Zhou Mo
Note, 0 untran, 1 fuzzy, and the fuzzy string remains at Bug#818639.
2016-03-17test framework: Pass -n to lsof to speed up finding the https portJulian Andres Klode
There is no point in resolving all addresses to their names, this just seriously slows the setup phase down. So just pass -n to not resolve names anymore. Gbp-Dch: ignore
2016-03-17test-acquire-same-file-multiple-times: Run failing test up to 10 timesJulian Andres Klode
This should make the test less flaky and hopefully fix the failure on Ubuntu's armhf CI nodes. Gbp-Dch: ignore