| Age | Commit message (Collapse) | Author |
|---|
|
Curl requires URLs to be urlencoded. We are however giving it
undecoded URLs. This causes it go completely nuts if there is
a space in the URI, producing requests like:
GET /a file HTTP/1.1
which the servers then interpret as a GET request for "/a" with
HTTP version "file" or some other non-sense.
This works around the issue by encoding the path component of
the URL. I'm not sure if we should encode other parts of the URL
as well, this one seems to do the trick for the actual issue at
hand.
A more correct fix is to avoid the dequoting and (re-)quoting
of URLs when a redirect occurs / a new request is sent. That's
been on the radar for probably a year or two now, but nobody
bothered implementing that yet.
LP: #1651923
|
|
Our implementation of wildcards was rudimentary. It worked for some
common ones, but it was also broken: For example, armel matched any-armel,
but should match any-arm.
With this commit, we load the correct tables from dpkg. Supported are
both triplets and quadruplet tables (the latter introduced in dpkg 1.18.11).
There are some odd things we have to deal with in the cache filter for
historical and API reasons:
* The character "*" must be accepted as an alternative to any - in fact
it may appear anywhere in the wildcard as we also allow fnmatch() style
wildcard matching on the commandline.
* The code might get passed an arch with a minus at the end, for example
the cmdline "install apt:any-arm-" will first try to check if any-arm-
is a valid architecture. We deal with this by rejecting any wildcard
ending in a minus.
* Triplets are actually implemented by extending them to faux quadruplets
- by prepending a "base" component for the architecture tuple, and "any"
if there is a wildcard component.
Once we have constructed a wildcard, it is transformed into an fnmatch()
expression for historical reasons. In the future, we should really get a
tuple class and implement matching in a better, more explicit way.
This does for now though - it passes all the test cases and accepts all
things it should accept.
Closes: #748936
Thanks: James Clarke <jrtc27@jrtc27.com> for the initial patch
|
|
The idea is simple: Each¹ Find*( call starts with a call check if the
given option (with the requested type) exists in the whitelist. The
whitelist is specified via our configure-index file so that we have
a better chance at keeping it current. the whitelist is loaded via a
special (undocumented for now) configuration stanza and if none is
loaded the empty whitelist will make it so that no warnings are shown.
Much needs to be done still, but that is as good a time as any to take a
snapshot of the current state and release it into the wild given that it
found some bugs already and has no practical effect on users.
¹ not all in this iteration, but many
|
|
Users end up believing that this is a --force mode as -f is common for
that, but apt doesn't have such a mode and --fix-broken is really not
about forcing something but actually trying to fix the breakage which
tends to be the result of a user forcing something on its system via
low-level forced dpkg calls.
Example: The "common" pattern of "dpkg -i ./foo.deb; apt install -f" is
nowadays far better dealt with via "apt install ./foo.deb".
And while at it the two places handing out this suggestion are changed
to use the same strings to avoid needless translation work in the future
and the suggestion uses 'apt' instead of 'apt-get' as this will be run
interactively by a user, so its a good opportunity to showcase what we
can do and will allow us to be more helpful to the user.
Closes: #709092
Thanks: Kristian Glass for initial patch!
|
|
The config list APT::Build-Essential gets a similar treatment to other
lists now by having the value of the option itself be an override for
the list allowing to disable build-essentials entirely as well as
adding/overriding as usual by now in other lists.
Reported-By: Johannes 'josch' Schauer on IRC
|
|
The implementation is quite different compared to --arch-only due to ABI
reasons but functionality wise they are similar and usually both
available for symmetry at least.
Closes: #845775
|
|
The crude way of preparing a message to be a multiline value failed at
generation valid deb822 in case the error message ended with a new line
like the resolving errors from apt do. apt itself can parse these, but
other tools like grep-dctrl choke on it, so be nice and print valid.
Reported-By: Johannes 'josch' Schauer on IRC
|
|
Clearsigned files like InRelease, .dsc, .changes and co can potentially
include unsigned or additional messages blocks ignored by gpg in
verification, but a potential source of trouble in our own parsing
attempts – and an unneeded risk as the usecases for the clearsigned
files we deal with do not reasonably include unsigned parts (like emails
or some such).
This commit changes the silent ignoring to warnings for now to get an
impression on how widespread unintended unsigned parts are, but
eventually we want to turn these into hard errors.
|
|
Importing a new secret key into gpg(2) can be increadibly slow which
prolongs the test runs significantly – by caching the homedir we gain a
significant speedbonus as reimporting already present keys seems like a
far less costly operation.
Git-Dch: Ignore
|
|
That was the case already for tar-only and diff-only, but in a more
confusing way and without a message while dsc "worked" before resulting
in a dpkg-source error shortly after as tar/diff files aren't available…
|
|
The test test-handle-redirect-as-used-mirror-change serves multiple
clients at the same time, so the order of the output is undefined and
once in a while the two clients will intermix their lines causing the
grep we perform on it later to fail making our tests fail.
Solved by introducing client-specific logfiles which we all grep and
sort the result to have the results more stable.
Git-Dch: Ignore
|
|
Having binary files in /etc is kinda annoying – not that the armored
files are much better – but it is hard to keep tabs on which format the
file has ("simple" or "keybox") and different gnupg versions have
different default binary formats which can be confusing for users to
work with (beside that it is binary).
Adding support for this now will enable us in some distant future to
move to armored later on, much like we added trusted.gpg.d years before
the world picked it up.
|
|
dpkg stumbles over these (#844300) and we haven't dropped 'easier'
removes to be implicit and to be scheduled by dpkg by default so far
so we shouldn't push the decision in such cases to dpkg either.
|
|
Our old idea was to look for the first package which would be "touched"
and take this as the package dpkg is talking about, but that is
incorrect in complicated situations like a package upgraded to/from
multiple M-A:same siblings installed.
As we us the progress report to decide what is still needed we have to
be reasonabily right about the package dpkg is talking about, so we jump
to quite a few loops to get it.
|
|
Given that we use the progress information to skip over actions dpkg has
already done like not purging a package which was already removed and
had no config files or not acting on disappeared packages and such it is
important that apt and dpkg agree on which states the package has to
pass through.
To ensure that we keep tabs on this in the future a warning is added at
the end if apt hasn't seen all the action it was supposed to see. I
can't wait for the first bugreporters to wonder about this…
|
|
This also changes Acquire-By-Hash to be "yes" rather than "true", so it
is consistent with dak's output.
Closes: #272557
|
|
We have the last Release file around for other checks, so its trivial to
look if the new Release file contains a new codename (e.g. the user has
"testing" in the sources and it flipped from stretch to buster).
Such a change can be okay and expected, but also be a hint of problems,
so a warning if we see it happen seems okay. We can only print it once
anyhow and frontends and co are likely to ignore/hide it.
|
|
A suite or codename entry in the Release file is checked against the
distribution field in the sources.list entry that lead to the download of that
Release file. This distribution entry can contain slashes in the distribution
field:
deb http://security.debian.org/debian wheezy/updates main
However, the Release file may only contain "wheezy" in the Codename field and
not "wheezy/updates". So a transformation needs to take place that removes the
last / and everything that comes after (e.g. "/updates"). This fails, however,
for valid cases like a reprepro snapshot where the given Codename contains
slashes but is perfectly fine and doesn't need to be transformed. Since that
transformation is essentially just a workaround for special cases like the
security repository, it should be checked if the literal Codename without any
transformations happened is valid and only if isn't the dist should be checked
against the transformated one.
This way special cases like security.debian.org are handled and reprepro
snapshots work too.
The initial patch was taken as insperationto move whole transformation
to CheckDist() which makes this method more accepting & easier to use
(but according to codesearch.d.n we are the only users anyhow).
Thanks: Lukas Anzinger for initial patch
Closes: 644610
|
|
You can pretty much achieve the same with a local dummy package if you
want to, but libapt has an inbuilt setting for essential: "apt" which
can be overridden with this option as well – it could be helpful in
quick tests and what not so adding this alternative shouldn't really
hurt much.
We aren't going to document them much through as care must be taken in
regards to the binary caches as they aren't invalidated by config
options alone, so the effects of old settings could still be in them,
similar to the other already existing pkgCacheGen option(s).
Closes: 767891
Thanks: Anthony Towns for initial patch
|
|
On Travis CI running tests with code coverage enabled sometimes
generates lines like:
profiling:/path/to/file.gcda:Merge mismatch for function 257
It would be nice if we could resolve this somehow as it garbles the
statistics, but until then it is far more annoying that this causes
test failures for no good reason.
Gbp-Dch: Ignore
|
|
Reported-By: Christoph Berg (Myon) on IRC
|
|
The 0.0.0.0:0 tor reports is pretty useless by itself, but even if an IP
would be reported it is better to show the user the hostname we wanted
the proxy to connect to in the same error message. We improve upon it
further by looking for this bind address in particular and remap error
messages slightly to give users a better chance of figuring out what
went wrong. Upstream Tor can't do that as it is technically wrong.
|
|
Some people do not recognize the field value with such an arcane name
and/or expect it to refer to something different (e.g. #839257).
We can't just rename it internally as its an avoidance strategy as such
fieldname existed previously with less clear semantics, but we can spare
the general public from this implementation detail.
|
|
Sometimes you should really act upon your todos.
Especially if you have placed them directly in the code.
Closes: 841874
|
|
We can't cleanup the environment like e.g. sudo would do as you usually
want the environment to "leak" into these helpers, but some variables
like HOME should really not have still the value of the root user – it
could confuse the helpers (USER) and HOME isn't accessible anyhow.
Closes: 842877
|
|
See also c5f22e483cc0f31f2938874370ac776e40792069
Gbp-Dch: Ignore
|
|
dpkg 1.18.11 includes:
* Do not log nor print duplicate dpkg removal action. We print
“Removing <package> (<version>)” lines and log remove action twice
when purging a package from frontends, because they usually first call
--remove and then --purge sequentially. When purging a package which is
already in config-files (i.e. it has been removed before), do not print
nor log the remove action.
|
|
A user relying on the deprecated behaviour of apt-get to accept a source
with an unknown pubkey to install a package containing the key expects
that the following 'apt-get update' causes the source to be considered
as trusted, but in case the source hadn't changed in the meantime this
wasn't happening: The source kept being untrusted until the Release file
was changed.
This only effects sources not using InRelease and only apt-get, the apt
binary downright refuses this course of actions, but it is a common way
of adding external sources.
Closes: 838779
|
|
In effect this is an extension of the 6 years old commit
a8dfff90aa740889eb99d00fde5d70908d9fd88a which uses the autoremover to
remove packages again from the solution which are no longer needed to be
there. Commonly these are dependencies of packages we end up not
installed due to problem resolver decisions. Slightly less common is
the situation we deal with here: a package which we wanted to upgrade
sporting a new dependency, but ended up holding back.
The problem is that all versions of an installed reverse dependencies can
bring back a "garbage" package – we need to do this as there is
nothing inherently wrong in having garbage packages installed or upgrade
them, which itself would have garbage dependencies, so just blindly
killing all new garbage packages would prevent the upgrade (and actually
generate errors). What we should be doing is looking only at the version
we will have on the system, disregarding all old/new reverse dependencies.
Reported-By: Stuart Prescott (themill) on IRC
|
|
If the dependency line does not contain spaces in the repository
but does in the dpkg status file (because dpkg normalized the
dependency list), the dpkg line might be longer than the line
in the repository. If it now happens to be longer than 1024
characters, it would be skipped, causing the hashes to be
out of date.
Note that we have to bump the minor cache version again as
this changes the format slightly, and we might get mismatches
with an older src cache otherwise.
Fixes Debian/apt#23
|
|
Even if we only configure a single architecture, install dpkg, so
dpkg can assert multi arch correctly. This also has the nice side
effect of making single architecture and multiple architecture
test cases more uniform.
This fixes a regression from f878d3a862128bc1385616751ae1d78246b1bd01
("test: Assert multi-arch in the chroot").
|
|
If we copied one of the existing status files, we might not have
a trailing newline, so let's add one.
Gbp-Dch: ignore
|
|
Commit b60c8a89c281f2bb945d426d2215cbf8f5760738 improved the situation,
but due to inconsistency mostly for planners, not for solvers. As the
idea of hiding errors if we show another error is a bit scary (as the
extern error might be a followup of our intern error, rather than the
reason for our intern error as it is at the moment) we don't discard the
errors, but if we got an extern error we show them directly removing
them from the error list at the end of the run – that list will contain
the extern error which hopefully gives us the best of both worlds.
The problem itself is the same as before: The externals exiting before
apt is done talking to them.
Reported-By: Johannes 'josch' Schauer on IRC
|
|
Employ a priority queue instead of a normal queue to hold
the items; and only add items to the running pipeline if
their priority is the same or higher than the priority
of items in the queue.
The priorities are designed for a 3 stage pipeline system:
In stage 1, all Release files and .diff/Index files are fetched. This
allows us to determine what files remain to be fetched, and thus
ensures a usable progress reporting.
In stage 2, all Pdiff patches are fetched, so we can apply them
in parallel with fetching other files in stage 3.
In stage 3, all other files are fetched (complete index files
such as Contents, Packages).
Performance improvements, mainly from fetching the pdiff patches
before complete files, so they can be applied in parallel:
For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update
of Debian unstable and testing with Contents and appstream for
amd64 and i386, update time reduced from 37 seconds to 24-28
seconds.
Previously, apt would first download new DEP11 icon tarballs
and metadata files, causing the CPU to be idle. By fetching
the diffs in stage 2, we can now patch our contents and Packages
files while we are downloading the DEP11 stuff.
|
|
In gpgv1 GOODSIG (and the other messages of status-fd) are documented as
sending the long keyid. In gpgv2 it is documented to be either long
keyid or the fingerprint. At the moment it is still the long keyid, but
the documentation hints at the possibility of changing this.
We care about this for Signed-By support as we detect this way if the
right fingerprint has signed this file (or not). The check itself is
done via VALIDSIG which always is a fingerprint, but there must also be
a GOODSIG (as expired sigs are valid, too) found to be accepted which
wouldn't be found in the fingerprint-case and the signature hence
refused.
|
|
gpg annoyingly changed its output and broke our test suite
again by adding two extra lines about key type and issuer.
Really annoying.
Those lines also have more than one space after the colon,
so let's use \s* there - and also change the other lines to
support variable length whitespace in case gpg decides to
break things there too.
|
|
I actually tried to amend the previous commit, but apparently
I forgot to add the file mode change.
Gbp-Dch: ignore
|
|
If a Binary field contains one or more spaces before a comma, the
code produced a segmentation fault, as it accidentally set a pointer
to 0 instead of the value of the pointer.
If the comma is at the beginning of the field, the code would
create a binStartNext that points one element before the start
of the string, which is undefined behavior.
We also need to check that we do not exit the string during the
replacement of spaces before commas: A string of the form " ,"
would normally exit the boundary of the Buffer:
binStartNext = offset 1 ','
binEnd = offset 0 ' '
isspace_ascii(*binEnd) = true => --binEnd
=> binEnd = - 1
We get rid of the problem by only allowing spaces to be eliminated
if they are not the first character of the buffer:
binStartNext = offset 1 ','
binEnd = offset 0 ' '
binEnd > buffer = false, isspace_ascii(*binEnd) = true
=> exit loop
=> binEnd remains 0
|
|
This works around an issue on Fedora where dpkg complains about
missing build-essential:
dpkg-checkbuilddeps: Unmet build dependencies: build-essential:native
Gbp-Dch: ignore
|
|
Introduce a new -qq mode for our integration test framework,
and make travis use it.
The new -qq mode sets MSGLEVEL to 1. In MSGLEVEL=1, no messages
are generated for passed tests, and all testcase filenames are
printed in the same line.
Also install first in travis, do not ls the installed output
and run the install with chronic, so we only get output if it
failed.
Gbp-Dch: ignore
|
|
An absolute filename for a *.deb file starts with a /. A package with
the name of the file is inserted in the cache which is provided by the
"real" package for internal reasons. The pinning code detects a regex
based wildcard by having the regex start with /. That is no problem
as a / can not be included in a package name… expect that our virtual
filename package can and does.
We fix this two ways actually: First, a regex is only being considered a
regex if it also ends with / (we don't support flags). That stops our
problem with the virtual filename packages already, but to be sure we
also do not enter the loop if matcher and package name are equal.
It has to be noted that the creation of pins for virtual packages like
the here effected filename packages is pointless as only versions can be
pinned, but checking that a package is really purely virtual is too
costly compared to just creating an unused pin.
Closes: 835818
|
|
Without randomizing the order in which we download the index files we
leak needlessly information to the mirrors of which architecture is
native or foreign on this system. More importantly, we leak the order in
which description translations will be used which in most cases will e.g.
have the native tongue first.
Note that the leak effect in practice is limited as apt detects if a file
it wants to download is already available in the latest version from a
previous download and does not query the server in such cases. Combined
with the fact that Translation files are usually updated infrequently
and not all at the same time, so a mirror can never be sure if it got asked
about all files the user wants.
|
|
|
|
On BSD systems, the root group is wheel, not root, so let's
just use the default group here.
Gbp-Dch: ignore
|
|
The host system might not have a dpkg installed, which makes
dpkg fail with:
dpkg not recorded as installed, cannot check for multi-arch support!
That's entirely useless of course. We want to know if dpkg could
support multi-arch in our chroot, so we pseudo-install dpkg into
the chroot and pretend it's version is one version higher than
the minimum dpkg version, so dpkg --assert-multi-arch works on
recent dpkgs.
Gbp-Dch: ignore
|
|
This fixes issues with chroots, but the goal here was to get
the test suite working on systems without dpkg.
|
|
This is needed on FreeBSD which has versions like 11.0-RC1,
otherwise the tests would fail.
|
|
This check should work regardless if dpkg was installed by dpkg
or by a native package manager like RPM or pkg.
Gbp-Dch: ignore
|
|
This is more safe against sticky bits. For example, in FreeBSD
all files created in /tmp have the group set to wheel.
Gbp-Dch: ignore
|
|
Gbp-Dch: ignore
|
