| Age | Commit message (Collapse) | Author |
|---|
|
Since commit 7a68effcb904b4424b54a30e448b6f2560cd1078, the xz
and lzma compressors read the level of compression they shall
use.
A default of -9 is too much for them, this will use 674 MB,
according to the xz manual page. Level -6 on the other hand
only needs 94 MB memory for compression.
This causes autopkgtest failures in the test-compressed-indexes
test, as not enough memory exists to proceed.
Change the other compression levels to 6 as well: The gzip
and bzip2 FileFd backends do not read them, and use their
code's default level which is 6, so do the same for external
methods.
|
|
There's no point in keeping using yet another read-then-write
loop.
Gbp-Dch: ignore
|
|
This avoids some issues with InternalWrite returning 0 because
it just cannot write stuff at the moment.
|
|
Closes: 809160
|
|
Regression introduced in 8710a36a01c0cb1648926792c2ad05185535558e,
but such fields are unlikely in practice as it is just as simple to not
have a field at all with the same result of not having a value.
Closes: 808102
|
|
|
|
People should know that this is exists.
|
|
Debian has a Packages file for arch:all already, but the arch:any files
contain arch:all packages as well, so downloading it would be a total
waste of resources. Getting this solved is on the list of things to do,
but it is also the hardest part – for index targets like Contents the
situation is much easier and less server/client implementations are
involved so we might not want to stall them.
A repository can now declare via:
No-Support-for-Architecture-all: Packages
that even if an arch:all Packages exists, it shouldn't be downloaded, so
that support for Contents files can be added now.
See also 1dd20368486820efb6ef4476ad739e967174bec4 for the implementation
of downloading arch:all index targets, which this is limiting.
The field uses the name of the target from the apt configuration for
simplicity and is negative by design as this field is intended to be
supported/needed only for a "short" time (one or two Debian releases).
While this commit theoretically supports any target, its expected to
only see "Packages" as a value in reality.
|
|
This is for public users only, which cannot use the class at all,
except for the static methods.
|
|
Buffered writes improve performance a lot, given that we spent
about 78% of the time in _write.
|
|
This is somewhat experimental right now, and might not work
for everyone, so it is on an opt-in basis.
|
|
The flush function can be used for buffered writers.
|
|
We will soon implement a buffered writing decorator and we will
need to forward attribute changes to those.
|
|
These can be used to implement write buffering
Gbp-Dch: ignore
|
|
Suggested by David.
Gbp-Dch: ignore
|
|
Gbp-Dch: ignore
|
|
This accidentally slipped in in a previous commit, but it should
be used only for testing mode.
Reported-By: David Kalnischkies <david@kalnischkies.de>
|
|
We do not see those branches at all during normal mode of
operation (that is, during cache generation), so tell the
compiler about it.
|
|
The Set() method returns false if the input is no hex number,
so simply use that.
|
|
We directly check if we are a hex digit in HexDigit, so use that
information.
[jak@debian.org: Commit message wording]
|
|
This makes the code parsing architecture lists slower, but on
the other hand, improves the more generic case of reading
dependencies from Packages files.
|
|
std::unordered_map is faster than std::map in our use case,
reducing cache generation time by about 10% in my benchmark.
|
|
This converts all callers that read machine-generated data,
callers that might work with user input are not converted.
|
|
This is like isspace(), but ignores the current locale.
|
|
This release is made for Niels Thykier and apt-file.
|
|
Gbp-Dch: ignore
|
|
There is not much point and this is more readable.
Gbp-Dch: ignore
|
|
This is mostly a documentation issue, as the size we want to
read is always less than or equal to the size of the buffer,
so the return value will be the same as the size argument.
Nonetheless, people wondered about it, and it seems clearer
to just always use the return value.
|
|
This can automatically handle compressed files and is useful
for stuff like apt-file.
|
|
|
|
This further improves our performance, and rred on uncompressed
files now spents 78% of its time in writing. Which means that
we should really look at buffering those.
|
|
This introduces a -t mode in which the first argument is input,
the second is output and the remaining are diffs.
This allows us to test patching compressed files, which are
detected using their file extension.
|
|
The code uses memmove() to move parts of the buffer to the
front when the buffer is only partially read. By simply
reading one page at a time, the maximum size of bytes that
must be moved has a hard limit, and performance improves:
In one test case, consisting of a 430 MB Contents file,
and a 75K PDiff, applying the PDiff previously took about
48 seconds and now completes in 2 seconds.
Further speed up can be achieved by buffering writes, they
account for about 60% of the run-time now.
|
|
Gbp-Dch: ignore
|
|
|
|
And as we are at it lets fix the 'style' issue I introduced with the
filefd changes as well.
Reported-By: gcc -fsanitize's & cppcheck
Git-Dch: Ignore
|
|
We try to acquired the locks, but we didn't stop if we failed to get it…
Closes: 808561
|
|
We don't need the buffer that often - only for ReadLine - as it is only
occasionally used, so it is actually more efficient to allocate it if
needed instead of statically by default. It also allows the caller to
influence the buffer size instead of hardcoding it.
Git-Dch: Ignore
|
|
The default implementation of ReadLine was very naive by just reading
each character one-by-one. That is kinda okay for libraries implementing
compression as they have internal buffers (but still not great), but
while working with files directly or via a pipe as there is no buffer
there so all those reads are in fact system calls.
This commit introduces an internal buffer in the FileFd implementation
which is only used by ReadLine. The more low-level Read and all other
actions remain unbuffered – they just changed to deal with potential
"left-overs" in the buffer correctly.
Closes: 808579
|
|
If we use the library to compress xz, still try to understand and pick
up the arguments we would have used to call xz to figure out which level
the user wants us to use instead of defaulting to level 6 (which is the
default level of xz).
|
|
dpkg switched from CRC32 to CRC64 in
777915108d9d36d022dc4fc4151a615fc95e5032 with the message:
| This is the default CRC used by the xz command-line tool, align with
| it and switch from CRC32 to CRC64. It should provide slightly better
| detection against damaged data, at a negligible speed difference.
|
|
This isn't implementing any new features, it is "just" moving code
around from FileFd methods which decided on each call how to handle the
request by including all logic for all possible compressor backends in
the method body to a model in which backend-specifics are implemented in
a FileFdPrivate subclass. This avoids a big chunk of #ifdef's and should
make it a tiny bit more obvious which backend uses which code.
The execution of the idea is slightly uglified by the need to preserve
ABI and API which causes liberal befriending.
Git-Dch: Ignore
|
|
The output changes slightly between different versions, which we already
dealt with in the main testcase for apt-key, but there are two more
which do not test both versions explicitly and so still had gpg1 output
to check against as this is the default at the moment.
Git-Dch: Ignore
|
|
The presents (even of an empty) secring.gpg is indication enough for
gpg2 to tigger the migration code which not only produces a bunch of
output on each apt-key call, but also takes a while to complete as an
agent needs to be started and all that.
We workaround the first part by forcing the migration to happen always
in a call we forced into silence, but that leaves us with an agent to
start all the time – with a bit of reordering we can make it so that we
do not explicitly create the secring, but let gpg create it if needed,
which prevents the migration from being triggered and we have at least a
bit less of a need for an agent. Changes - even to public only keyrings
- still require one, but such actions are infrequent in comparison to
verification calls, so that should be a net improvement.
|
|
apt-key creates internally a script (since ~1.1) which it will call to
avoid dealing with an array of different options in the code itself, but
while writing this script it wraps the values in "", which will cause
the shell to evaluate its content upon execution.
To make 'use' of this either set a absolute gpg command or TMPDIR to
something as interesting as:
"/tmp/This is fü\$\$ing cràzy, \$(man man | head -n1 | cut -d' ' -f1)\$!"
If such paths can be encountered in reality is a different question…
|
|
This doesn't allow all tests to run cleanly, but it at least allows to
write tests which could run successfully in such environments.
Git-Dch: Ignore
|
|
Git-Dch: Ignore
|
|
There's no point trying to read 0 bytes, so let's just not
do this and switch to a while loop like in Write().
Gbp-Dch: ignore
|
|
Turn the do-while loop into while loops, so it simply does nothing
if the Size is already 0.
This reverts commit c0b271edc2f6d9e5dea5ac82fbc911f0e3adfa7a which
introduced a fix for a specific instance of the issue in the
CopyFile() function.
Closes: #808381
|
|
On EOF, ToRead will be 0, which might trigger on some systems (e.g.
on the Hurd) an error due to the invalid byte count passed to write().
The whole loop already checks for ToRead != 0, so perform the writing
step only when there was actual data read.
Closes: #808381
|
