| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Closes: #977938
|
|
Use encoded URIs in the acquire system
See merge request apt-team/apt!139
|
|
This commit potentially breaks code feeding apt an encoded URI using a
method which does not get URIs send encoded. The webserverconfig
requests in our tests are an example for this – but they only worked
before if the server was expecting a double encoding as that was what
was happening to an encoded URI: so unlikely to work as expected in
practice.
Now with the new methods we can drop this double encoding and rely on
the URI being passed properly (and without modification) between the
layers so that passing in encoded URIs should now work correctly.
|
|
Every method opts in to getting the encoded URI passed along while
keeping compat in case we are operated by an older acquire system.
Effectively this is just a change for the http-based methods as the
others just decode the URI as they work with files directly.
|
|
We do not deal a lot with URIs which need encoding, but then we do it is
a pain that we store it decoded in the acquire system as it means we
have to decode and reencode URIs eventually which is potentially giving
us slightly different URIs.
We see that in our own testing framework while setting up redirects as
the config options are effectively double-encoded and decoded to pass
them around successfully as otherwise %2f and / in an URI are treated
the same.
This commit adds the infrastructure for methods to opt into getting URIs
send in encoded form (and returning them to us in encoded form, too) so
that we eventually do not have to touch the URIs which is how it should
be. This means though that we have to deal with methods who do not
support this yet (aka: all at the moment) for which we decode and encode
while communicating with them.
|
|
Our http method encodes the URI again which results in the double
encoding we have unwrap in the webserver (we did already, but we skip
the filename handling now which does the first decode).
|
|
|
|
Unroll pkgCache::sHash 8 times and break up the dependency between
the iterations by expanding the calculation
H(n) = 33 * H(n-1) + c
8 times rather than performing it 8 times. This seems to yield about
a 0.4% performance improvement.
I tried unrolling 4 and 2 bytes as well, those only having 3 ifs at
the end rather than 1 small loop; but that was actually slower -
potentially the code got to large and the cache went bonkers.
I also tried unrolling 4 times instead of 8, thinking that smaller
code might yield better results overall then, but that was slower as
well.
|
|
|
|
XXH3 is faster than both our CRC32c implementation as well
as DJB hash for hash table hashing, so meh, let's switch to
it.
|
|
|
|
|
|
We now have over 100k package names, my Ubuntu system has 125k
arleady, so increase the hash table size to match, this will cost
us about a MB in cache size, but give a very nice speed up somewhere
around 3%-4% or so.
|
|
|
|
The integer overflow was detected by DonKult who added a check like this:
(std::numeric_limits<decltype(Itm.Size)>::max() - (2 * sizeof(Block)))
Which deals with the code as is, but also still is a fairly big limit,
and could become fragile if we change the code. Let's limit our file
sizes to 128 GiB, which should be sufficient for everyone.
Original comment by DonKult:
The code assumes that it can add sizeof(Block)-1 to the size of the item
later on, but if we are close to a 64bit overflow this is not possible.
Fixing this seems too complex compared to just ensuring there is enough
room left given that we will have a lot more problems the moment we will
be acting on files that large as if the item is that large, the (valid)
tar including it probably doesn't fit in 64bit either.
|
|
Like the code in arfile.cc, MemControlExtract also has buffer
overflows, in code allocating memory for parsing control files.
Specify an upper limit of 64 MiB for control files to both protect
against the Size overflowing (we allocate Size + 2 bytes), and
protect a bit against control files consisting only of zeroes.
|
|
Tarballs have long names and long link targets structured by a
special tar header with a GNU extension followed by the actual
content (padded to 512 bytes). Essentially, think of a name as
a special kind of file.
The limit of a file size in a header is 12 bytes, aka 10**12
or 1 TB. While this works OK-ish for file content that we stream
to extractors, we need to copy file names into memory, and this
opens us up to an OOM DoS attack.
Limit the file name size to 1 MiB, as libarchive does, to make
things safer.
|
|
GHSL-2020-169: This first hunk adds a check that we have more files
left to read in the file than the size of the member, ensuring that
(a) the number is not negative, which caused the crash here and (b)
ensures that we similarly avoid other issues with trying to read too
much data.
GHSL-2020-168: Long file names are encoded by a special marker in
the filename and then the real filename is part of what is normally
the data. We did not check that the length of the file name is within
the length of the member, which means that we got a overflow later
when subtracting the length from the member size to get the remaining
member size.
The file createdeb-lp1899193.cc was provided by GitHub Security Lab
and reformatted using apt coding style for inclusion in the test
case, both of these issues have an automated test case in
test/integration/test-ubuntu-bug-1899193-security-issues.
LP: #1899193
|
|
This allows patterns like ~nalpha~nbeta and ~nalpha!~nbeta to
work like they do in APT.
Also add a comment to remind readers that everything in START
should be in short too.
Cc: stable >= 2.0
|
|
The compiler does not know that the size is small and thinks we might
be doing a stack buffer overflow of the vla:
Add APT_ASSUME macro and silence -Wstringop-overflow in HexDigest()
The compiler does not know that the size of a hash is at most 512 bit,
so tell it that it is.
../apt-pkg/contrib/hashes.cc: In function ‘std::string HexDigest(gcry_md_hd_t, int)’:
../apt-pkg/contrib/hashes.cc:415:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
415 | Result[(Size)*2] = 0;
| ~~~~~~~~~~~~~~~~~^~~
../apt-pkg/contrib/hashes.cc:414:9: note: at offset [-9223372036854775808, 9223372036854775807] to an object with size at most 4294967295 declared here
414 | char Result[((Size)*2) + 1];
| ^~~~~~
Fix this by adding a simple assertion. This generates an extra two
instructions in the normal code path, so it's not exactly super costly.
|
|
Fixes lookup in as-installed testing
Gbp-Dch: ignore
|
|
|
|
This is more accurate
|
|
Apply hints suggested by the multi-arch hinter
See merge request apt-team/apt!137
|
|
Fix typo in Catalan translation.
See merge request apt-team/apt!132
|
|
Enhance rred for possible external usage
See merge request apt-team/apt!136
|
|
+ apt-doc, libapt-pkg-doc: Add Multi-Arch: foreign.
Changes-By: apply-multiarch-hints
|
|
|
|
|
|
The acquire system mode does this for a long time already and as it is
easy to implement and handy for manual testing as well we can support
it in the other modes, too.
|
|
Merging patches is a bit of non-trivial code we have for client-side
work, but as we support also server-side merging we can export this
functionality so that server software can reuse it.
Note that this just cleans up and makes rred behave a bit more like all
our other binaries by supporting setting configuration at runtime and
supporting --help and --version. If you can make due without this, the
now advertised functionality is provided already in earlier versions.
|
|
In LP#835625, it was reported that apt did not unpack multi-arch
packages in the correct order, and dpkg did not like that. The fix
also made apt configure packages together, which is not strictly
necessary.
This turned out to cause issues now, because of dependencies on
libc6:i386 that caused immediate configuration of that to not
work.
Work around the issue by not configuring multi-arch: same packages
in lockstep if they have the immediate flag set. This will be the
pseudo-essential set, and given how essential works, we mostly need
the native arch to work correctly anyway.
LP: #1871268
Regression-Of: 30426f4822516bdd26528aa2e6d8d69c1291c8d3
|
|
|
|
That mostly means deleting symbols which went private or have
disappeared and were previously compiler artefacts.
|
|
The versions "needing" these fixes are at least five years old, so in an
effort to save massive amounts of runtime and disk space (on aggregate at
least) we can drop these lines.
Reported-By: lintian maintainer-script-supports-ancient-package-version
|
|
Reported-By: dh_missing
|
|
| CMake Warning (dev) at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:273 (message):
| The package name passed to `find_package_handle_standard_args` (Berkeley)
| does not match the name of the calling package (BerkeleyDB). This can lead
| to problems in calling code that expects `find_package` result variables
| (e.g., `_FOUND`) to follow a certain pattern.
| Call Stack (most recent call first):
| CMake/FindBerkeleyDB.cmake:57 (find_package_handle_standard_args)
| CMakeLists.txt:83 (find_package)
| This warning is for project developers. Use -Wno-dev to suppress it.
And indeed, we checked for BERKLEY_DB_FOUND which was not defined so our
HAVE_BDB was not set – just that it is never used, so it wasn't noticed.
|
|
Closes: #968414
|
|
mirror.fail points to porn now apparently.
Cc: stable
|
|
We accidentally excluded virtual packages by excluding every
group that had a package, but where the package had no versions.
Rewrite the code so the lookup consistently uses VersionList()
instead of FirstVersion and FindPkg("any") - those are all the
same, and this is easier to read.
|
|
We passed "false" instead of false, and that apparently got
cast to bool, because it's a non-null pointer.
LP: #1876495
|
|
|
|
We are seeing more and more installations fail due to immediate
configuration issues related to libc6. Immediate configuration is
supposed to ensure that an essential package is configured immediately,
just in case some other packages use a part of the essential package
that only works if that package is configured.
This used to be a warning, it was turned into an error in some commit I
can't remember right now, but importantly, the error missed a return,
which means that ordering completed succesfully and packages were being
installed anyway; and after all that happened successfully, we'd print
an error at the end and exit with an error code, which is not super
useful.
Revert the error back to a warning such that the behavior stays the same
but we do not fail (unless we mess up ordering which then gets caught by
a consistency check later on.
Closes: #953260
Closes: #972552
LP: #1871268
|
|
Closes: #970037
[jak: Fix typo extended_status -> extended_states]
|
|
|
|
|
|
Closes: #969086
|
|
|
|
Add better acquire debugging support
See merge request apt-team/apt!130
|
