summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2021-01-08kernels: remove spurious || falseJulian Andres Klode
Gbp-Dch: ignore
2021-01-08Fix getMachineID copy-paste errorJulian Andres Klode
Gbp-Dch: ignore
2021-01-08Merge branch 'pu/phased-updates' into 'master'Julian Andres Klode
Add support for Phased-Update-Percentage See merge request apt-team/apt!129
2021-01-08Phase using source version to be binNMU-correctJulian Andres Klode
If we have different binNMU versions on different architectures, we don't want madness to ensue. This is a change from how update-manager does things, as Ubuntu does not have binNMUs, but I believe it's the right thing to do for a generic solution.
2021-01-08Add support for Phased-Update-PercentageJulian Andres Klode
This adds support for Phased-Update-Percentage by pinning upgrades that are not to be installed down to 1. The output of policy has been changed to add the level of phasing, and documentation has been improved to document how phased updates work. The patch detects if it is running in a chroot, and if so, always includes phased updates, restoring classic apt behavior to avoid behavioral changes on buildd chroots. Various options are added to control this all: * APT::Get::{Always,Never}-Include-Phased-Updates and their legacy update-manager equivalents to always or never include phased updates * APT::Machine-ID can be set to a UUID string to have all machines in a fleet phase the same * Dir::Etc::Machine-ID is weird in that it's default is sort of like ../machine-id, but not really, as ../machine-id would look up $PWD/../machine-id and not relative to Dir::Etc; but it allows you to override the path to machine-id (as opposed to the value) * Dir::Bin::ischroot is the path to the ischroot(1) binary which is used to detect whether we are running in a chroot.
2021-01-08Merge branch 'pu/autoremove-kernels-in-apt-only' into 'master'Julian Andres Klode
Only autoremove kernels in apt(8); respect --no-auto-remove See merge request apt-team/apt!149
2021-01-08Only autoremove kernels in apt(8); respect --no-auto-removeJulian Andres Klode
Automatically removing kernels in apt-get could be unexpected, so limit it to apt for now. To handle --no-auto-remove correctly, rewrite the hack that makes apt ignore APT::Get::AutomaticRemove options from config files such that it unsets the option. This then means we can do FindB("APT::Get::AutomaticRemove", true) as the default for APT::Get::AutomaticRemove::Kernels and get the behavior we want: If you set --no-auto-remove, it is respected as that FindB returns false; if you don't set it, it will be true.
2021-01-08Merge branch 'pu/optional-immediate' into 'master'Julian Andres Klode
Make immediate configuration optional See merge request apt-team/apt!148
2021-01-08Make immediate configuration optionalJulian Andres Klode
The benefits of immediate configuration are that Essential packages will be configured immediately, so if they are wrongly not working without being configured they won't fail later packages. However, we've reached the point where dependencies on the essential set are too complex for immediate configuration to always work, causing installations to error out at the end, despite having succeeded, because we did not correctly return the error here and did not check for pending errors before running dpkg. Given that we check and configure any packages at the end that have not been configured yet, or fail if we can't configure them; making immediate configuration optional is the best way forward - it orders as it does now, but then does not spuriously fail after having successfully installed everything. Closes: #973305, #188161, #211075, #649588 LP: #1871268
2021-01-07Merge branch 'pu/bump-codenames' into 'master'Julian Andres Klode
Bump codenames to bullseye/hirsute and adjust -security codename See merge request apt-team/apt!147
2021-01-07Bump codenames to bullseye/hirsute and adjust -security codenameJulian Andres Klode
Closes: #969932
2021-01-07Merge branch 'pu/depends' into 'master'Julian Andres Klode
?depends patterns and friends See merge request apt-team/apt!146
2021-01-05Merge branch 'bash-compat' into 'master'Julian Andres Klode
Be compatible with Bash See merge request apt-team/apt!142
2021-01-04Merge branch 'pu/kernel-autoremove' into 'master'Julian Andres Klode
Determine autoremovable kernels at run-time See merge request apt-team/apt!138
2021-01-04Only keep up to 3 (not 4) kernelsJulian Andres Klode
This fixes a problem on Ubuntu systems where the /boot partition has been sized to manage 3 kernels, but does not really work with 4 kernels which was causing problems all over the place.
2021-01-04Automatically remove unused kernels on dist-upgradeJulian Andres Klode
Kernels clutter /boot and /boot is small size, so we need to take extra care to remove kernels when possible.
2021-01-04Determine autoremovable kernels at run-timeJulian Andres Klode
Our kernel autoremoval helper script protects the currently booted kernel, but it only runs whenever we install or remove a kernel, causing it to protect the kernel that was booted at that point in time, which is not necessarily the same kernel as the one that is running right now. Reimplement the logic in C++ such that we can calculate it at run-time: Provide a function to produce a regular expression that matches all kernels that need protecting, and by changing the default root set function in the DepCache to make use of that expression. Note that the code groups the kernels by versions as before, and then marks all kernel packages with the same version. This optimized version inserts a virtual package $kernel into the cache when building it to avoid having to iterate over all packages in the cache to find the installed ones, significantly improving performance at a minor cost when building the cache. LP: #1615381
2021-01-04depcache: Cache our InRootSetFuncJulian Andres Klode
This avoids the cost of setting up the function every time we mark and sweep.
2021-01-01Merge branch 'http-to-https' into 'master'Julian Andres Klode
aptmethod: fix HTTP->HTTPS request sequences See merge request apt-team/apt!140
2020-12-31Greek program translation updateVangelis Skarmoutsos
See merge request apt-team/apt!144
2020-12-28Be compatible with BashDemi M. Obenour
On many distributions, /bin/sh is Bash. Bash’s `echo` builtin doesn’t interpret escape sequences, so most tests fail. Fix this by removing the escape sequence.
2020-12-27Implement ?reverse-depends/~R and friendsJulian Andres Klode
This was easy.
2020-12-27woofJulian Andres Klode
2020-12-27patterns: Add dependency patterns ?depends, ?conflicts, etc.Julian Andres Klode
These match the target package, not target versions which is slightly unfortunate but might make sense. Maybe we should add a version that matches Versions instead.
2020-12-27Release 2.1.15Julian Andres Klode
2020-12-23German program translation updateHelge Kreutzmann
Closes: #977938
2020-12-23connect: use ServiceNameOrPort, not Port, as the cache keyFaidon Liambotis
The "last connection" cache is currently being stored and looked up on the combination of (LastHost, LastPort). However, these are not what the arguments to getaddrinfo() were on the first try: the call is to getaddrinfo(Host, ServiceNameOrPort, ...), i.e. with the port *or if 0, the service name* (e.g. http). Effectively this means that the connection cache lookup for: https://example.org/... i.e. Host = example.org, Port = 0, Service = http would end up matching the "last" connection of (if existed): https://example.org/... i.e. Host = example.org, Port = 0, Service = https ...and thus performing a TLS request over an (unrelated) port 80 connection. Therefore, an HTTP request, followed up by an (unrelated) HTTPS request to the same server, would always fail. Address this by using as the cache key the ServiceNameOrPort, rather than Port.
2020-12-23connect: convert a C-style string to std::stringFaidon Liambotis
Convert the fixed-size (300) char array "ServStr" to a std::string, and simplify the code by removing snprintfs in the process. While at it, rename to the more aptly named "ServiceNameOrPort" and update the comment to reflect what this variable is meant to be.
2020-12-23basehttp: also consider Access when a Server's URIFaidon Liambotis
ServerState->Comp() is used by the HTTP methods main loop to check whether a connection can be reused, or whether a new one is needed. Unfortunately, the currently implementation only compares the Host and Port between the ServerState's internal URI, with a new URI. However these are URIs, and therefore Port is 0 when a URI port is not specificied, i.e. in the most common configurations. As a result, a ServerState for http://example.org/... will be reused for URIs of the form https://example.org/..., as both Host (example.org) and Port (0) match. In turn this means that GET requests will happen over port 80, in cleartext, even for those https URLs(!). URI Acquires for an http URI and subsequently for an https one, in the same aptmethod session, do not typically happen with apt as the frontend, as apt opens a new pipe with the "https" aptmethod binary (nowadays a symlink to http), which is why this hasn't been a problem in practice and has eluded detection so far. It does happen in the wild with other frontends (e.g. reprepro), plus is legitimately an odd and surprising behavior on apt's end. Therefore add a comparison for the URI's "Access" (= the scheme) in addition to Host and Port, to ensure that we're not reusing the same state for multiple different schemes.
2020-12-18Merge branch 'pu/uriencode' into 'master'Julian Andres Klode
Use encoded URIs in the acquire system See merge request apt-team/apt!139
2020-12-18Don't re-encode encoded URIs in pkgAcqFileDavid Kalnischkies
This commit potentially breaks code feeding apt an encoded URI using a method which does not get URIs send encoded. The webserverconfig requests in our tests are an example for this – but they only worked before if the server was expecting a double encoding as that was what was happening to an encoded URI: so unlikely to work as expected in practice. Now with the new methods we can drop this double encoding and rely on the URI being passed properly (and without modification) between the layers so that passing in encoded URIs should now work correctly.
2020-12-18Implement encoded URI handling in all methodsDavid Kalnischkies
Every method opts in to getting the encoded URI passed along while keeping compat in case we are operated by an older acquire system. Effectively this is just a change for the http-based methods as the others just decode the URI as they work with files directly.
2020-12-18Keep URIs encoded in the acquire systemDavid Kalnischkies
We do not deal a lot with URIs which need encoding, but then we do it is a pain that we store it decoded in the acquire system as it means we have to decode and reencode URIs eventually which is potentially giving us slightly different URIs. We see that in our own testing framework while setting up redirects as the config options are effectively double-encoded and decoded to pass them around successfully as otherwise %2f and / in an URI are treated the same. This commit adds the infrastructure for methods to opt into getting URIs send in encoded form (and returning them to us in encoded form, too) so that we eventually do not have to touch the URIs which is how it should be. This means though that we have to deal with methods who do not support this yet (aka: all at the moment) for which we decode and encode while communicating with them.
2020-12-18Proper URI encoding for config requests to our test webserverDavid Kalnischkies
Our http method encodes the URI again which results in the double encoding we have unwrap in the webserver (we did already, but we skip the filename handling now which does the first decode).
2020-12-17Do not require libxxhash-dev for including pkgcachegen.hJulian Andres Klode
2020-12-15Unroll pkgCache::sHash 8 time, break up dependencyJulian Andres Klode
Unroll pkgCache::sHash 8 times and break up the dependency between the iterations by expanding the calculation H(n) = 33 * H(n-1) + c 8 times rather than performing it 8 times. This seems to yield about a 0.4% performance improvement. I tried unrolling 4 and 2 bytes as well, those only having 3 ifs at the end rather than 1 small loop; but that was actually slower - potentially the code got to large and the cache went bonkers. I also tried unrolling 4 times instead of 8, thinking that smaller code might yield better results overall then, but that was slower as well.
2020-12-15Release 2.1.14Julian Andres Klode
2020-12-15Use XXH3 for cache, hash table hashingJulian Andres Klode
XXH3 is faster than both our CRC32c implementation as well as DJB hash for hash table hashing, so meh, let's switch to it.
2020-12-15test: fixup for hash table size increase (changed output order)Julian Andres Klode
2020-12-10Release 2.1.13Julian Andres Klode
2020-12-10Raise APT::Cache-HashtableSize to 196613Julian Andres Klode
We now have over 100k package names, my Ubuntu system has 125k arleady, so increase the hash table size to match, this will cost us about a MB in cache size, but give a very nice speed up somewhere around 3%-4% or so.
2020-12-09Merge branch 'pu/cve-2020-27350'Julian Andres Klode
2020-12-09CVE-2020-27350: tarfile: integer overflow: Limit tar items to 128 GiBJulian Andres Klode
The integer overflow was detected by DonKult who added a check like this: (std::numeric_limits<decltype(Itm.Size)>::max() - (2 * sizeof(Block))) Which deals with the code as is, but also still is a fairly big limit, and could become fragile if we change the code. Let's limit our file sizes to 128 GiB, which should be sufficient for everyone. Original comment by DonKult: The code assumes that it can add sizeof(Block)-1 to the size of the item later on, but if we are close to a 64bit overflow this is not possible. Fixing this seems too complex compared to just ensuring there is enough room left given that we will have a lot more problems the moment we will be acting on files that large as if the item is that large, the (valid) tar including it probably doesn't fit in 64bit either.
2020-12-09CVE-2020-27350: debfile: integer overflow: Limit control size to 64 MiBJulian Andres Klode
Like the code in arfile.cc, MemControlExtract also has buffer overflows, in code allocating memory for parsing control files. Specify an upper limit of 64 MiB for control files to both protect against the Size overflowing (we allocate Size + 2 bytes), and protect a bit against control files consisting only of zeroes.
2020-12-09tarfile: OOM hardening: Limit size of long names/links to 1 MiBJulian Andres Klode
Tarballs have long names and long link targets structured by a special tar header with a GNU extension followed by the actual content (padded to 512 bytes). Essentially, think of a name as a special kind of file. The limit of a file size in a header is 12 bytes, aka 10**12 or 1 TB. While this works OK-ish for file content that we stream to extractors, we need to copy file names into memory, and this opens us up to an OOM DoS attack. Limit the file name size to 1 MiB, as libarchive does, to make things safer.
2020-12-09CVE-2020-27350: arfile: Integer overflow in parsingJulian Andres Klode
GHSL-2020-169: This first hunk adds a check that we have more files left to read in the file than the size of the member, ensuring that (a) the number is not negative, which caused the crash here and (b) ensures that we similarly avoid other issues with trying to read too much data. GHSL-2020-168: Long file names are encoded by a special marker in the filename and then the real filename is part of what is normally the data. We did not check that the length of the file name is within the length of the member, which means that we got a overflow later when subtracting the length from the member size to get the remaining member size. The file createdeb-lp1899193.cc was provided by GitHub Security Lab and reformatted using apt coding style for inclusion in the test case, both of these issues have an automated test case in test/integration/test-ubuntu-bug-1899193-security-issues. LP: #1899193
2020-12-07patterns: Terminate short pattern by ~ and !Julian Andres Klode
This allows patterns like ~nalpha~nbeta and ~nalpha!~nbeta to work like they do in APT. Also add a comment to remind readers that everything in START should be in short too. Cc: stable >= 2.0
2020-12-04HexDigest: Silence -Wstringop-overflowJulian Andres Klode
The compiler does not know that the size is small and thinks we might be doing a stack buffer overflow of the vla: Add APT_ASSUME macro and silence -Wstringop-overflow in HexDigest() The compiler does not know that the size of a hash is at most 512 bit, so tell it that it is. ../apt-pkg/contrib/hashes.cc: In function ‘std::string HexDigest(gcry_md_hd_t, int)’: ../apt-pkg/contrib/hashes.cc:415:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=] 415 | Result[(Size)*2] = 0; | ~~~~~~~~~~~~~~~~~^~~ ../apt-pkg/contrib/hashes.cc:414:9: note: at offset [-9223372036854775808, 9223372036854775807] to an object with size at most 4294967295 declared here 414 | char Result[((Size)*2) + 1]; | ^~~~~~ Fix this by adding a simple assertion. This generates an extra two instructions in the normal code path, so it's not exactly super costly.
2020-12-02test-method-rred: Use apthelper instead of apt-helperJulian Andres Klode
Fixes lookup in as-installed testing Gbp-Dch: ignore
2020-12-02gitignore: Add .*.swp filesJulian Andres Klode