summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2015-12-26Get rid of memmove() in our read bufferingJulian Andres Klode
This further improves our performance, and rred on uncompressed files now spents 78% of its time in writing. Which means that we should really look at buffering those.
2015-12-26rred: Allow passing files as arguments for compressor testingJulian Andres Klode
This introduces a -t mode in which the first argument is input, the second is output and the remaining are diffs. This allows us to test patching compressed files, which are detected using their file extension.
2015-12-26Use a hardcoded buffer size of 4096 to fix performanceJulian Andres Klode
The code uses memmove() to move parts of the buffer to the front when the buffer is only partially read. By simply reading one page at a time, the maximum size of bytes that must be moved has a hard limit, and performance improves: In one test case, consisting of a 430 MB Contents file, and a 75K PDiff, applying the PDiff previously took about 48 seconds and now completes in 2 seconds. Further speed up can be achieved by buffering writes, they account for about 60% of the run-time now.
2015-12-24Mark all FileFdPrivate classes as hidden1.1.6Julian Andres Klode
Gbp-Dch: ignore
2015-12-241.1.6 Christmas releaseJulian Andres Klode
2015-12-23fix new[] vs delete mismatch introduced by b3db9d81David Kalnischkies
And as we are at it lets fix the 'style' issue I introduced with the filefd changes as well. Reported-By: gcc -fsanitize's & cppcheck Git-Dch: Ignore
2015-12-23ensure we got a lock in clean operationDavid Kalnischkies
We try to acquired the locks, but we didn't stop if we failed to get it… Closes: 808561
2015-12-23use a dynamic buffer for ReadLineDavid Kalnischkies
We don't need the buffer that often - only for ReadLine - as it is only occasionally used, so it is actually more efficient to allocate it if needed instead of statically by default. It also allows the caller to influence the buffer size instead of hardcoding it. Git-Dch: Ignore
2015-12-23implement a buffer system for FileFd::ReadLineDavid Kalnischkies
The default implementation of ReadLine was very naive by just reading each character one-by-one. That is kinda okay for libraries implementing compression as they have internal buffers (but still not great), but while working with files directly or via a pipe as there is no buffer there so all those reads are in fact system calls. This commit introduces an internal buffer in the FileFd implementation which is only used by ReadLine. The more low-level Read and all other actions remain unbuffered – they just changed to deal with potential "left-overs" in the buffer correctly. Closes: 808579
2015-12-22parse xz-compression level from configurationDavid Kalnischkies
If we use the library to compress xz, still try to understand and pick up the arguments we would have used to call xz to figure out which level the user wants us to use instead of defaulting to level 6 (which is the default level of xz).
2015-12-22follow dpkg and xz and use CRC64 for xz compressionDavid Kalnischkies
dpkg switched from CRC32 to CRC64 in 777915108d9d36d022dc4fc4151a615fc95e5032 with the message: | This is the default CRC used by the xz command-line tool, align with | it and switch from CRC32 to CRC64. It should provide slightly better | detection against damaged data, at a negligible speed difference.
2015-12-22shuffle compressor-specific code into private subclassesDavid Kalnischkies
This isn't implementing any new features, it is "just" moving code around from FileFd methods which decided on each call how to handle the request by including all logic for all possible compressor backends in the method body to a model in which backend-specifics are implemented in a FileFdPrivate subclass. This avoids a big chunk of #ifdef's and should make it a tiny bit more obvious which backend uses which code. The execution of the idea is slightly uglified by the need to preserve ABI and API which causes liberal befriending. Git-Dch: Ignore
2015-12-19tests: support gpg2 properly in all testcasesDavid Kalnischkies
The output changes slightly between different versions, which we already dealt with in the main testcase for apt-key, but there are two more which do not test both versions explicitly and so still had gpg1 output to check against as this is the default at the moment. Git-Dch: Ignore
2015-12-19avoid triggering gpg2 migration in apt-keyDavid Kalnischkies
The presents (even of an empty) secring.gpg is indication enough for gpg2 to tigger the migration code which not only produces a bunch of output on each apt-key call, but also takes a while to complete as an agent needs to be started and all that. We workaround the first part by forcing the migration to happen always in a call we forced into silence, but that leaves us with an agent to start all the time – with a bit of reordering we can make it so that we do not explicitly create the secring, but let gpg create it if needed, which prevents the migration from being triggered and we have at least a bit less of a need for an agent. Changes - even to public only keyrings - still require one, but such actions are infrequent in comparison to verification calls, so that should be a net improvement.
2015-12-19avoid evaluating shell in paths used in apt-keyDavid Kalnischkies
apt-key creates internally a script (since ~1.1) which it will call to avoid dealing with an array of different options in the code itself, but while writing this script it wraps the values in "", which will cause the shell to evaluate its content upon execution. To make 'use' of this either set a absolute gpg command or TMPDIR to something as interesting as: "/tmp/This is fü\$\$ing cràzy, \$(man man | head -n1 | cut -d' ' -f1)\$!" If such paths can be encountered in reality is a different question…
2015-12-19tests: support spaces in path and TMPDIRDavid Kalnischkies
This doesn't allow all tests to run cleanly, but it at least allows to write tests which could run successfully in such environments. Git-Dch: Ignore
2015-12-19buildsystem: deal with spaces in path to sourceDavid Kalnischkies
Git-Dch: Ignore
2015-12-19Do not try to read in FileFd::Read() if Size is 0Julian Andres Klode
There's no point trying to read 0 bytes, so let's just not do this and switch to a while loop like in Write(). Gbp-Dch: ignore
2015-12-19Do nothing in FileFd::Write() if Size is 0Julian Andres Klode
Turn the do-while loop into while loops, so it simply does nothing if the Size is already 0. This reverts commit c0b271edc2f6d9e5dea5ac82fbc911f0e3adfa7a which introduced a fix for a specific instance of the issue in the CopyFile() function. Closes: #808381
2015-12-19CopyFile: avoid failing on EOF on some systemsPino Toscano
On EOF, ToRead will be 0, which might trigger on some systems (e.g. on the Hurd) an error due to the invalid byte count passed to write(). The whole loop already checks for ToRead != 0, so perform the writing step only when there was actual data read. Closes: #808381
2015-12-19CopyFile: fix BufSize to a sane valuePino Toscano
Commit e977b8b9234ac5db32f2f0ad7e183139b988340d tries to make BufSize calculated based on the size of the buffer; the problem is that std::unique_ptr::size() returns a pointer to the data, so sizeof() equals to the size of a pointer (later divided by sizeof(char), which is 1). The result is that the CopyFile copies in chunks of 8 bytes, which is not exactly ideal... As solution, declare BufSize in advance, and use its value to allocate the Buf array. Closes: #808381
2015-12-19Fix FileUtlTest.GetTempDir failure when run as rootPino Toscano
Testing /usr as TMPDIR assumes that GetTempDir() cannot use it because it cannot write to it; this is true for non-root users, but not so much for root. Since root can access everything, perform this particular test case only when not running as root. Closes: #808383
2015-12-14Release 1.1.51.1.5Julian Andres Klode
2015-12-14test-apt-download-progress: Run the https test multiple timesJulian Andres Klode
This filters out errors due to timing issues. Early exits if enough pulses occured.
2015-12-14pkgcache: Make hash arch-independent using fixed size integerJulian Andres Klode
This helps writing test cases. Also adapt the test case that expected 64-bit. Nothing changes performance wise, the distribution of the hash values remains intact.
2015-12-14debian/apt.auto-removal.sh: Adjust -dbg exclusion for multi-archJulian Andres Klode
Allow an optional colon followed by anything at the end.
2015-12-14tagfile: Hardcode error message for out of range integer valuesJulian Andres Klode
This makes the test suite work on 32 bit-long platforms. Gbp-Dch: ignore
2015-12-14test framework: More noopchroot fixesJulian Andres Klode
Use asprintf() so we have easy error detection and do not depend on PATH_MAX. Do not add another separator to the generated path, in both cases the path inside the chroot is guaranteed to have a leading / already. Also pass -Wall to gcc.
2015-12-14test framework: Unset no_proxy as wellJulian Andres Klode
This caused test-bug-717891-abolute-uris-for-proxies to fail Gbp-Dch: ignore
2015-12-14test framework: Unset http proxy variables in setupenvironmentJulian Andres Klode
This breaks a lot of test cases Gbp-Dch: ignore
2015-12-14test framework: Correctly generate new paths in noopchrootJulian Andres Klode
The allocated buffer was one byte too small. Allocate a buffer of PATH_MAX instead and use snprintf(), as suggested by Martin Pitt.
2015-12-14test-apt-download-progress: Limit rate to 600kJulian Andres Klode
This reduces the chance that the test fails. Gbp-Dch: ignore
2015-12-14test-apt-download-progress: Allow smaller progress changeJulian Andres Klode
Instead of checking for [10%, 100%), check for (0%, 100%), that is everything < 100% and >0%. Gbp-Dch: ignore
2015-12-14test-sourceslist-lang-plusminus-options: Use amd64 architectureJulian Andres Klode
This should make the test work on non-amd64 systems Gbp-Dch: ignore
2015-12-14tests: Depend on libfile-fcntllock-perl to shut up dpkg-gencontrolJulian Andres Klode
Ubuntu's autopkgtest server always prints dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe which is somewhat annoying. Work around that by depending on that perl stuff for the test suite. Gbp-Dch: ignore
2015-12-14non-existing directories don't need to be cleanedDavid Kalnischkies
Trying to clean up directories which do not exist seems rather silly if you think about it, so let apt think about it and stop it. Depends a bit on the caller if this is fixing anything for them as they might try to acquire a lock or doing other clever things as apt does. Closes: 807477
2015-12-14support regex and co in 'apt-cache policy $pkg' againDavid Kalnischkies
Regression of 1e064088bf7b3e29cd36d30760fb3e4143a1a49a (1.1~exp4) which moved code around and renamed methods heavily ending up calling the wrong method matching packagenames only instead of calling the full array. Most commands work with versions, so this managed to fly under the radar for quite a while. Closes: 807870
2015-12-14show a more descriptive error for weak Release filesDavid Kalnischkies
If we can't work with the hashes we parsed from the Release file we display now an error message if the Release file includes only weak hashes instead of downloading the indexes and failing to verify them with "Hash Sum mismatch" even through the hashes didn't mismatch (they were just weak). If for some (unlikely) reason we have got weak hashes only for individual targets we will show a warning to this effect (again, befor downloading and failing the index itself). Closes: 806459
2015-12-14redirect which stderr to /dev/null for consistencyDavid Kalnischkies
The "standard" which (debianutils) has no output whatsoever on stderr, bash and dash which use this implementation therefore haven't either. In zsh 'which' is a shell built-in – and has no stderr output either, it does print an error message on stdout… So, realistically, a redirection isn't needed at all, but it also can't hurt (<- I have said that before in this context ->) so why not for consistency with… well, not with "command -v" as that hasn't an error message either. Lets say for consistency with my mental image of shell, as I am still a bit puzzled by zsh's which and now could imagine even more strange things in other shells. Closes: 807373
2015-12-14mention APT::AutoRemove::{Recommends,Suggests}Important optionAndreas Metzler
Closes: 807413
2015-12-14fix incorrect Korean translation of "manually" and "automatically"Seo Sanghyeon
Korean translation update is attached. I downloaded the original from https://www.debian.org/international/l10n/po/ko. Originally, "manually" and "automatically" were translated same, which was confusing. Mail-Reference: <CAPrzcnL_YynHQkCSE4xH29eW1O0Q6SPhoSf21KOFTu=Vv6-0Nw@mail.gmail.com> Mail-Archive: https://lists.debian.org/deity/2015/12/msg00108.html
2015-12-13parse .diff/Index hashes in reverse orderDavid Kalnischkies
Reversing the parsing order ensures that we parse weaker hashes (like SHA1) before we touch newer/stronger hashes (like SHA256) as the weaker ones will usually be there for a longer time already with data already present, which we would discard if we start with the strong one first. The discarding is visible in the debug logs: File X wasn't in the list for the first parsed hash! (history) File X wasn't in the list for the first parsed hash! (patches) which if file X is part of the patch-path means apt will not find a path and fallback to acquire the whole file instead needlessly. If file X isn't part of the patch-path that is no problem, so that effects only the update-call which updates with patches coming from before and after the addition of a new hash.
2015-12-13fix typos and docs in GlobalError documentationDavid Kalnischkies
Reported-By: Manuel A. Fernandez Montecelo <mafm@debian.org> Git-Dch: Ignore
2015-12-13update symbols files to 1.1.4 state of affairsDavid Kalnischkies
2015-12-13check library was built before comparing with symbols fileDavid Kalnischkies
Git-Dch: Ignore
2015-12-11.travis.yml: Switch to new trusty machinesJulian Andres Klode
Yay, newer server Gbp-Dch: ignore
2015-12-11mmap: Define _DEFAULT_SOURCE instead of _BSD_SOURCEJulian Andres Klode
Fixes a warning reported by gcc. Gbp-Dch: ignore
2015-12-11apt-internal-solver: Make ShowHelp() and GetCommands() staticJulian Andres Klode
This fixes a warning reported by gcc. Gbp-Dch: ignore
2015-12-11Bump cache minor version to 2 to trigger rebuildsJulian Andres Klode
With the package names now normalized to lower case, the caches of affected systems need to be rebuild. Adjust the minor version to trigger such a rebuild. Gbp-Dch: ignore
2015-12-11Convert package names from Packages files to lower caseJulian Andres Klode
dpkg does that when reading package files, so we should do the same. This only deals with parsing names from binary package paragraphs, it does not look at source package names and/or the list of binaries in a dsc file. Closes: #807012