summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2016-01-02add optional support for comments in pkgTagFileDavid Kalnischkies
APT usually deals with perfectly formatted files generated automatically be other programs – and as it has to parse multiple MBs of such files it tries to be fast rather than forgiving. This was always a problem if we reused this parser for files with a deb822 syntax which are mostly written by hand however, like apt_preferences or the deb822-style sources as these can include stray newlines and more importantly comments all over the place. As a stopgap we had pkgUserTagSection which deals at least with comments before and after a given stanza, but comments in between weren't really supported and now that we support parsing debian/control for e.g. build-dep we face the full comment problem e.g. with comments inbetween multi-line fields (like Build-Depends). We can't easily deal with this on the pkgTagSection level as the interface gives access to 'raw' char-pointers for performance reasons so we would need to optionally add a buffer here on which we could remove comments to hand out pointers into this buffer instead. The interface is quite large already and supports writing stanzas as well, which does not support comments at all either. So while in future it might make sense to have a parser setup which deals with and keeps comments in this commit we opt for the simpler solution for now: We officially declare that pkgTagSection does not support comments and instead expect the caller to deal with them, which in our case is pkgTagFile: pkgTagFile is extended with an additional mode which can deal with comments by dropping them from the buffer which will later form the input of pkgTagSection. The actual implementation is slightly more complex than this sentence suggests at first on one hand to have good performance and on the other to allow jumping directly to stanzas with offsets collected in a previous run (like our cache generation does it for example).
2016-01-02fail installing build-deps if parsing them failedDavid Kalnischkies
Git-Dch: Ignore
2016-01-02Italian manpages translation updateBeatrice Torracca
Closes: 809522
2015-12-30Release 1.1.101.1.10Julian Andres Klode
2015-12-29Do not sync the cache fileJulian Andres Klode
Integrity is taken care of by the checksum now.
2015-12-29Add support for calculating hashes over the entire cacheJulian Andres Klode
2015-12-29pkgCacheGenerator: Allow passing down an already created cacheJulian Andres Klode
If we already have opened a cache, there is no point in having to open it again.
2015-12-29pkgTagSection::Scan: Fix read of uninitialized valueJulian Andres Klode
We ignored the boundary of the buffer we were reading in while scanning for spaces.
2015-12-29strutl.cc: Add declarations for the compat _ascii() functionsJulian Andres Klode
This shuts up gcc Gbp-Dch: ignore
2015-12-29Fix test case that was broken by switch of hash functionJulian Andres Klode
This test relies on the ordering of the hash function.
2015-12-29Turn tolower_ascii() and isspace_ascii() into inline functionsJulian Andres Klode
To preserve compatibility, the new inline functions have _inline as a suffix, and a macro defines the old names to refer to the inline variants. The old functions are still preserved for binary compatibility. Also simplify the implementation of both functions.
2015-12-29Switch to DJB hashing and use prime number as table sizeJulian Andres Klode
On my testing system, consisting of unstable and experimental, this reduces the average chain from 6.5 to 4.5, and the longest chain from 17 to 15.
2015-12-28BufferedFileFdPrivate: Make InternalFlush() save against errorsJulian Andres Klode
Previously, if flush errored inside the loop, data could have already been written to the wrapped descriptor without having been removed from the buffer. Also try to work around EINTR here. A better solution might be to have the individual privates detect an interrupt and return 0 in such a case, instead of relying on errno being untouched in between the syscall and the return from InternalWrite.
2015-12-28aptconfiguration: Set default compression level to 6Julian Andres Klode
Since commit 7a68effcb904b4424b54a30e448b6f2560cd1078, the xz and lzma compressors read the level of compression they shall use. A default of -9 is too much for them, this will use 674 MB, according to the xz manual page. Level -6 on the other hand only needs 94 MB memory for compression. This causes autopkgtest failures in the test-compressed-indexes test, as not enough memory exists to proceed. Change the other compression levels to 6 as well: The gzip and bzip2 FileFd backends do not read them, and use their code's default level which is 6, so do the same for external methods.
2015-12-28apt-helper: Use CopyFile() for concatenating the filesJulian Andres Klode
There's no point in keeping using yet another read-then-write loop. Gbp-Dch: ignore
2015-12-28BufferedWriter: flushing: Check for written < size instead of <=Julian Andres Klode
This avoids some issues with InternalWrite returning 0 because it just cannot write stuff at the moment.
2015-12-27fix 3 typos/omissions in apt.conf(5)Beatrice Torracca
Closes: 809160
2015-12-27deal with empty values properly in deb822 parserDavid Kalnischkies
Regression introduced in 8710a36a01c0cb1648926792c2ad05185535558e, but such fields are unlikely in practice as it is just as simple to not have a field at all with the same result of not having a value. Closes: 808102
2015-12-27Release 1.1.91.1.9Julian Andres Klode
2015-12-27doc/acquire-additional-files.txt: Mention apt-helper cat-fileJulian Andres Klode
People should know that this is exists.
2015-12-27allow repositories to forbid arch:all for specific index targetsDavid Kalnischkies
Debian has a Packages file for arch:all already, but the arch:any files contain arch:all packages as well, so downloading it would be a total waste of resources. Getting this solved is on the list of things to do, but it is also the hardest part – for index targets like Contents the situation is much easier and less server/client implementations are involved so we might not want to stall them. A repository can now declare via: No-Support-for-Architecture-all: Packages that even if an arch:all Packages exists, it shouldn't be downloaded, so that support for Contents files can be added now. See also 1dd20368486820efb6ef4476ad739e967174bec4 for the implementation of downloading arch:all index targets, which this is limiting. The field uses the name of the target from the apt configuration for simplicity and is negative by design as this field is intended to be supported/needed only for a "short" time (one or two Debian releases). While this commit theoretically supports any target, its expected to only see "Packages" as a value in reality.
2015-12-27pkgcachegen.h: Hack around unordered_map not existing before C++11Julian Andres Klode
This is for public users only, which cannot use the class at all, except for the static methods.
2015-12-27rred: Use buffered writesJulian Andres Klode
Buffered writes improve performance a lot, given that we spent about 78% of the time in _write.
2015-12-27FileFd: Add a buffered writing modeJulian Andres Klode
This is somewhat experimental right now, and might not work for everyone, so it is on an opt-in basis.
2015-12-27FildFd: Introduce a Flush() function and call it from Close()Julian Andres Klode
The flush function can be used for buffered writers.
2015-12-27FileFdPrivate: Add getter and setter for fieldsJulian Andres Klode
We will soon implement a buffered writing decorator and we will need to forward attribute changes to those.
2015-12-27fileutl: simple_buffer: Add write() and full() methodsJulian Andres Klode
These can be used to implement write buffering Gbp-Dch: ignore
2015-12-27fileutl: simple_buffer: Mark accessors as constJulian Andres Klode
Suggested by David. Gbp-Dch: ignore
2015-12-27FileFdPrivate: Extract SimpleBuffer and mark it as hiddenJulian Andres Klode
Gbp-Dch: ignore
2015-12-27rred: Only call pkgInitConfig() in test modeJulian Andres Klode
This accidentally slipped in in a previous commit, but it should be used only for testing mode. Reported-By: David Kalnischkies <david@kalnischkies.de>
2015-12-27ParseDepends: Mark branches for build-dep parsing as unlikelyJulian Andres Klode
We do not see those branches at all during normal mode of operation (that is, during cache generation), so tell the compiler about it.
2015-12-27debListParser: Do not validate Description-md5 for correctness twiceJulian Andres Klode
The Set() method returns false if the input is no hex number, so simply use that.
2015-12-27Hex2Digit: Do not use isxdigit()Niels Thykier
We directly check if we are a hex digit in HexDigit, so use that information. [jak@debian.org: Commit message wording]
2015-12-27debListParser: ParseDepends: Only query native arch if neededJulian Andres Klode
This makes the code parsing architecture lists slower, but on the other hand, improves the more generic case of reading dependencies from Packages files.
2015-12-27pkgcachegen: Use std::unordered_map instead of std::mapJulian Andres Klode
std::unordered_map is faster than std::map in our use case, reducing cache generation time by about 10% in my benchmark.
2015-12-27Convert most callers of isspace() to isspace_ascii()Julian Andres Klode
This converts all callers that read machine-generated data, callers that might work with user input are not converted.
2015-12-27Introduce isspace_ascii() for use by parsersJulian Andres Klode
This is like isspace(), but ignores the current locale.
2015-12-26Release 1.1.81.1.8Julian Andres Klode
This release is made for Niels Thykier and apt-file.
2015-12-26apt-helper: Check that we can open stdoutJulian Andres Klode
Gbp-Dch: ignore
2015-12-26Refactor InternalReadLine to not unroll Size == 0 caseJulian Andres Klode
There is not much point and this is more readable. Gbp-Dch: ignore
2015-12-26Change InternalReadLine to always use buffer.read() return valueJulian Andres Klode
This is mostly a documentation issue, as the size we want to read is always less than or equal to the size of the buffer, so the return value will be the same as the size argument. Nonetheless, people wondered about it, and it seems clearer to just always use the return value.
2015-12-26apt-helper: Add a cat-file command for concatening filesJulian Andres Klode
This can automatically handle compressed files and is useful for stuff like apt-file.
2015-12-26Release 1.1.71.1.7Julian Andres Klode
2015-12-26Get rid of memmove() in our read bufferingJulian Andres Klode
This further improves our performance, and rred on uncompressed files now spents 78% of its time in writing. Which means that we should really look at buffering those.
2015-12-26rred: Allow passing files as arguments for compressor testingJulian Andres Klode
This introduces a -t mode in which the first argument is input, the second is output and the remaining are diffs. This allows us to test patching compressed files, which are detected using their file extension.
2015-12-26Use a hardcoded buffer size of 4096 to fix performanceJulian Andres Klode
The code uses memmove() to move parts of the buffer to the front when the buffer is only partially read. By simply reading one page at a time, the maximum size of bytes that must be moved has a hard limit, and performance improves: In one test case, consisting of a 430 MB Contents file, and a 75K PDiff, applying the PDiff previously took about 48 seconds and now completes in 2 seconds. Further speed up can be achieved by buffering writes, they account for about 60% of the run-time now.
2015-12-24Mark all FileFdPrivate classes as hidden1.1.6Julian Andres Klode
Gbp-Dch: ignore
2015-12-241.1.6 Christmas releaseJulian Andres Klode
2015-12-23fix new[] vs delete mismatch introduced by b3db9d81David Kalnischkies
And as we are at it lets fix the 'style' issue I introduced with the filefd changes as well. Reported-By: gcc -fsanitize's & cppcheck Git-Dch: Ignore
2015-12-23ensure we got a lock in clean operationDavid Kalnischkies
We try to acquired the locks, but we didn't stop if we failed to get it… Closes: 808561