| Age | Commit message (Collapse) | Author |
|---|
|
Testing the current implementation can benefit from being able to be
feed an EIPP request and produce a fully compliant response. It is also
a great test for EIPP in general.
|
|
We can trim generation time and size of the EIPP scenario considerable
if we we avoid telling the planers about "uninteresting" packages.
This is one of the simpler but already very effective reductions:
Do not tell planers about versions which are neither installed nor are
to be installed as they have no effect on the plan we don't need to tell
the planer about them. EDSP solvers need to know about all versions for
better choice and error messages, but planers really don't.
Git-Dch: Ignore
|
|
The very first step in introducing the "external installation planer
protocol" (short: EIPP) as part of my GSoC2016 project.
The description reads: APT-based tools like apt-get, aptitude, synaptic,
… work with the user to figure out how their system should look like
after they are done installing/removing packages and their dependencies.
The actual installation/removal of packages is done by dpkg with the
constrain that dependencies must be fulfilled at any point in time (e.g.
to run maintainer scripts).
Historically APT has a super micro-management approach to this task
which hasn't aged that well over the years mostly ignoring changes in
dpkg and growing into an unmaintainable mess hardly anyone can debug and
everyone fears to touch – especially as more and more requirements are
tacked onto it like handling cycles and triggers, dealing with
"important" packages first, package sources on removable media, touch
minimal groups to be able to interrupt the process if needed (e.g.
unattended-upgrades) which not only sky-rocket complexity but also can
be mutually exclusive as you e.g. can't have minimal groups and minimal
trigger executions at the same time.
|
|
In 3bdff17c894d0c3d0f813d358fc45d7a263f3552 we did it for the datetime
parsing, but we use the same style in the parsing for pdiff (where the
size of the file is in the middle of the three fields) so imbueing here
as well is a good idea.
|
|
Rewritten in 9febc2b238e1e322dce1f94ecbed46d595893b52 for c++ locales
usage and rewritten again in 1d742e01470bba27715a8191c50adde4b39c2f19 to
avoid a currently present stdlibc++6 bug in the std::get_time
implementation. The later implementation uses still stringstreams for
parsing, but forgot to explicitly reset the locale to something sane
(for parsing english dates that is), so date and especially the parsing
of a number is depending on the locale. Turns out, the French (among
others) format their numbers with space as thousand separator so for
some reason the stdlibc++6 thinks its a good idea to interpret the
entire datetime string as a single number instead of realizing that in
"25 Jun …" the later parts can't reasonably be part of that number even
through there are spaces there…
Workaround is hence: LC_NUMERIC=C.UTF-8
Closes: 828011
|
|
Weak had no dedicated option before and Insecure and Downgrade were both
global options, which given the effect they all have on security is
rather bad. Setting them for individual repositories only isn't great
but at least slightly better and also more consistent with other
settings for repositories.
|
|
Filesize is a silly hash all by itself, but in combination with others
it can be a strong opponent, so ensuring that it is in the list of
hashes and hence checked by the normal course of action the acquire
process takes is a good thing.
|
|
For "Hash Sum mismatch" that info doesn't make a whole lot of
difference, but for the new insufficient info message an indicator that
while this hashes are there and even match, they aren't enough from a
security standpoint.
|
|
Downloading and saying "Hash Sum mismatch" isn't very friendly from a
user POV, so with this change we try to detect such cases early on and
report it, preferably before download even started.
Closes: 827758
|
|
With this commit all APT-based clients default to refusing to work with
unsigned or otherwise insufficently secured repositories. In terms of
apt and apt-get this changes nothing, but it effects all tools using
libapt like aptitude, synaptic or packagekit.
The exception remains apt-get for stretch for now as this might break
too many scripts/usecases too quickly.
The documentation is updated and extended to reflect how to opt out or
in on this behaviour change.
Closes: 808367
|
|
Handling the extra check (and force requirement) for downgrades in
security in our AllowInsecureRepositories checker helps in having this
check everywhere instead of just in the most common place and requiring
a little extra force in such cases is always good.
|
|
APT can be forced to deal with repositories which have no security
features whatsoever, so just giving up on repositories which "just" fail
our current criteria of good security features is the wrong incentive.
Of course, repositories are better of fixing their setup to provide the
minimum of security features, but sometimes this isn't possible:
Historic repositories for example which do not change (anymore).
That also fixes problem with repositories which are marked as trusted,
but are providing only weak security features which would fail the
parsing of the Release file.
Closes: 827364
|
|
Unsecure repositories result in error messages by default which causes
the acquire run to fail hard, but non-failing repositories are still
updated just like in the slightly less hard-failures which got this
behaviour in 35664152e47a1d4d712fd52e0f0a2dc8ed359d32.
|
|
There is a subtile difference between an empty setting and "DIRECT" in
the configuration as the later overrides the generic settings while the
earlier does not. Also, non-zero exitcodes should really be reported as
an error rather than silently discarded.
|
|
Regression introduced in 8f858d560e3b7b475c623c4e242d1edce246025a.
Commands are probably better of always having output through as the
fall through to the generic proxy settings is likely not intended. As
documenting and implementing this more consistently is kind of a
regression through, it is split off into the next commit.
Closes: 827713
|
|
As reported upstream in
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71556
the implementation of std::get_time is currently not as accepting as
strptime is, especially in how hours should be formatted.
Just reverting 9febc2b238e1e322dce1f94ecbed46d595893b52 would be
possible, but then we would reopen the problems fixed by it, so instead
I opted here for a rewrite of the parsing logic which makes this method
a lot longer, but at least it provides the same benefits as the rewrite
in std::get_time was intended to give us and decouples us from the fix
of the issue in the standard library implementation of GCC.
LP: 1593583
|
|
Merging by URI means that having sources lines with different URI
methods results in 'strange' warning and error messages, which aren't
very friendly from a user point of view as not encoding the method in
the filename is effectivly an implementation detail.
Merging by filename removes these messages and makes everything "work"
even if it isn't working the way it is configured as the indexes aren't
acquired over the method given, but over the first method for this
release file (which argueably is an implementation detail stemming from
the filename encoding, too).
So either direction isn't perfectly "right", but personally I prefer
"magic" over strange error messages (and doing a full-circle detection
of this with its own messages which would need to be translated feels
like way too much effort for dubious gain).
Closes: 826944
|
|
We usually use absolute paths to specific the location of dpkg, apt-key
and the like, but there is nothing wrong with using just the command
name and instead let exec(3) make the lookup in PATH.
We had a wild mixture before, so opting for the more accepting option
out of the two seems about right especially as it makes no difference in
the default case as apt uses absolute paths.
|
|
Just closing the fd would be enough, but while we are at it we can also
use the Popen interface to have an easier time with this.
|
|
Not a big deal to leak fds in debugging mode, but for completeness.
Git-Dch: Ignore
|
|
|
|
Seen first in #826783, but as this buglog also shows leaked uncompressed
files as well we don't close it just yet.
|
|
This effects only compressors configured on the fly (rather then the
inbuilt ones as they use a library).
|
|
The comment says it should have been removed with Lenny+1 which is a
small while ago already, so it seems like a good time now…
And as this is a cleanup commit it also gets right of spurious
whitespace at the end of lines, adds missing fold markers and similar
busy work.
Git-Dch: Ignore
|
|
We had an old FIXME saying that it is probably pointless to do this if
we limit by length of the commandline already and I completely agree.
The splitting is bad enough if it must be done, so we should only do it
if we have to (as in absolute length of commandline) and, but that is
just a remark, it is unlikely that we ever have/had a call triggering
this as the default value was ~32000 items…
|
|
We end our operation by calling "dpkg --configure -a", so instead of
running a (big) configure run with all packages mentioned explicitly
before this, we simply skip them and let them be handled by this call
implicitly.
There isn't really an observeable gain to be had here from a speed
point, but it helps in avoiding an (uncommon) problem of having a too
long commandline passed to dpkg, which we would split up (probably
incorrectly).
|
|
Most (if not all) solvers should be able to run perfectly fine without
root privileges as they get the entire state they are supposed to work
on via stdin and do not perform any action directly, but just pass
suggestions on via stdout.
The new default is to run them all as _apt hence, but each solver can
configure another user if it chooses/must. The security benefits are
minimal at best, but it helps preventing silly mistakes (see
35f3ed061f10a25a3fb28bc988fddbb976344c4d) and that is always good.
Note that our 'apt' and 'dump' solver already dropped privileges if they
had them.
|
|
Create and log the EDSP(like) request even if we use the internal
resolver.
|
|
It wasn't noticeable before, but now with the (optional) logging it can
be observed that we decide in the internal path two times if an internal
or external solver should be used (and hence with logging, it is
attempted twice), so if we are in the internal path call the internal
resolver directly, which means those internal methods need to be public
– but we can hide them based on the symbol at least.
|
|
For bugreports and co it could be handy to have the scenario and all the
settings used in it around later for inspection for EDSP like protocols.
EDSP might not be the most interesting as the user can still interrupt
the process before the solution is applied and users tend to have an
opinion on the "rightness" of a solution, so it is disabled by default.
|
|
EDSP(-like) protocols are one-shot processes working on data which
exists only as long as they run (as they get feed via a pipe), so trying
to write a cache for it is pretty pointless, especially as it will
usually fail as the cache files tend to be owned by root, but the
process is run as a unpriviledged user (either _apt if called by root,
otherwise the user of the caller).
So this was in fact only observeable with our testcases which run as
non-root and the worst which happens is that a valid cache is overridden
with an invalid one which the next run will detect and not use.
Git-Dch: Ignore
|
|
With have better ways to compare, manipulate and work with strings, so
use it instead of counting string length by hand with is a wonder it
hasn't failed yet. Ignoreable from a changelog perspective as there is
no behaviour change.
Git-Dch: Ignore
|
|
The classes are all marked as hidden, so changing them is no problem ABI
wise and will help with introducing protocols similar to EDSP.
The change has no observeable behavior difference, its just code
juggling.
Git-Dch: Ignore
|
|
EDSP had a WriteSolution method to write out the entire solution based
on the inspection of a given pkgDepCache, but that is rather inflexible
both for EDSP itself and for other EDSP like-protocols. It seems better
to use a smaller scope in printing just a single stanza based on a given
version as there is more reuse potential.
|
|
Currently an EDSP solver gets send basically all versions which means
the absolute count is the same, but that might not be true forever (and
with the skipping of rc-only versions it kinda is already) and even if
it were true, segfaulting on bad input seems wrong.
|
|
|
|
HTTP/1.1 hardcodes GMT (RFC 7231 §7.1.1.1) and what is good enough for the
internet must be good enough for us™ as we reuse the implementation
internally to parse (most) dates we encounter in various places like the
Release files with their Date and Valid-Until header fields.
Implementing a fully timezone aware parser just feels too hard for no
effective benefit as it would take 5+ years (= until LTS's are out of
fashion) until a repository could use non-UTC dates and expect it to
work. Not counting non-apt implementations which might or might not
only want to encounter UTC here as well.
As a bonus, this eliminates the use of an instance of setlocale in
libapt.
Closes: 819697
|
|
Setting the C++ locale via std::locale::global(std::locale("")); which
would otherwise default to the default C locale (aka: unaffected by
setlocale) effects the formatting of numeric types in IO streams, which
for output for humans is perfectly sensible, but breaks our many text
interfaces used and parsed by us and others without expecting the
numbers to be formatted.
Closes: #825396
|
|
libapt allows to configure compressors to be used by its system via
configuration implemented in 03bef78461c6f443187b60799402624326843396,
but that was never really documented and also only partly working, which
also explains why the tests weren't using it…
|
|
Reported-By: lintian: spelling-error-in-binary
Git-Dch: Ignore
|
|
No real code change, just moving code around heavily to decouple the
EDSP specific parts from those we can reuse for EDSP-like protocols.
Git-Dch: Ignore
|
|
Its more space and runtime efficient to use a boolean set instead of a
CacheSet-based implementation.
Git-Dch: Ignore
|
|
This allows to differentiate properly between 'apt-get upgrade', 'apt
upgrade' and 'apt full-upgrade'.
|
|
apt/apt-pkg/edsp.cc: In function ‘bool EDSP::WriteLimitedScenario(pkgDepCache&, FILE*, const PackageSet&, OpProgress*)’:
apt/apt-pkg/edsp.cc:245:56: warning: cannot optimize loop, the loop counter may overflow [-Wunsafe-loop-optimizations]
std::string dependencies[pkgCache::Dep::Enhances + 1];
^
Using a std::array to silence gcc as well as as a code improvement feels right here.
Git-Dch: Ignore
|
|
The &= introduced in the EDSP-FileFd conversion isn't working to full
satisfaction for multiple && clauses as the && has a higher binding than
&= has, so that the methods were called even through they shouldn't
have because of previous errors. Using variadic functions we can solve
this in a slightly cleaner way bringing down the amount of 'broken pipe'
errors for the error case of the dump resolver substantially.
Git-Dch: Ignore
|
|
I doubt there is any non-src:apt usage of these interfaces.
|
|
The report mentions "apt list --upgradable", but there are others which
have inconsistent behavior ranging from segfaulting to doing something
with the partial (and hence incomplete) data. We had a recent report
about sources.list (#818628), this one mentions prefences, the obvious
next step is conf files… so the testcase is adapted to check for all
three in file and directory versions and run a bunch of commands each
time which should all have more or less the same behavior in such a case
(aka error out).
Closes: 824503
|
|
This fixes comparisons where either the stored or the input string
have a trailing comma.
|
|
This hopefully makes debugging things easier.
|
|
Unexpected are for examples removal requests for versions which aren't
installed, installations of already installed versions & requests to
install and remove a package at the same time.
|
