| Age | Commit message (Collapse) | Author |
|---|
|
Commit b60c8a89c281f2bb945d426d2215cbf8f5760738 improved the situation,
but due to inconsistency mostly for planners, not for solvers. As the
idea of hiding errors if we show another error is a bit scary (as the
extern error might be a followup of our intern error, rather than the
reason for our intern error as it is at the moment) we don't discard the
errors, but if we got an extern error we show them directly removing
them from the error list at the end of the run – that list will contain
the extern error which hopefully gives us the best of both worlds.
The problem itself is the same as before: The externals exiting before
apt is done talking to them.
Reported-By: Johannes 'josch' Schauer on IRC
|
|
If a solver/planner exits before apt is done writing we will generate
write errors. Solvers like 'dump' can be pretty quick in failing but
produce a valid EDSP error report apt should read, parse and display
instead of just discarding even through we had write errors.
|
|
Reported-By: cppcheck
Gbp-Dch: Ignore
|
|
Simulations are frequently run by unprivileged users which naturally
don't have the permissions to write to the default location for the eipp
file. Either way is bad as running in simulation mode doesn't mean we
don't want to run the logging (as EIPP runs the same regardless of
simulation or 'real' run), but showing the warnings is relatively
pointless in the default setup, so, in case we would produce errors and
perform a simulation we will discard the warnings and carry on.
Running apt with an external planner wouldn't have generated these
messages btw.
Closes: 832614
|
|
If other logs can't be written this is a warning to,
so for consistency sake translate the errors to warnings.
|
|
Unlikely to happen in practice and I wonder more how I could miss these
in earlier reviews, but okay, lets fix it for consistency now.
|
|
All apt versions support numeric as well as 3-character timezones just
fine and its actually hard to write code which doesn't "accidently"
accepts it. So why change? Documenting the Date/Valid-Until fields in
the Release file is easy to do in terms of referencing the
datetime format used e.g. in the Debian changelogs (policy §4.4). This
format specifies only the numeric timezones through, not the nowadays
obsolete 3-character ones, so in the interest of least surprise we should
use the same format even through it carries a small risk of regression
in other clients (which encounter repositories created with
apt-ftparchive).
In case it is really regressing in practice, the hidden option
-o APT::FTPArchive::Release::NumericTimezone=0
can be used to go back to good old UTC as timezone.
The EDSP and EIPP protocols use this 'new' format, the text interface
used to communicate with the acquire methods does not for compatibility
reasons even if none of our methods would be effected and I doubt any
other would (in these instances the timezone is 'GMT' as that is what
HTTP/1.1 requires). Note that this is only true for apt talking to
methods, (libapt-based) methods talking to apt will respond with the
'new' format. It is therefore strongly adviced to support both also in
method input.
|
|
Julian noticed on IRC that I fall victim to a lovely false friend by
calling referring to a 'planer' all the time even through these are
machines to e.g. remove splinters from woodwork ("make stuff plane").
The term I meant is written in german in this way (= with a single n)
but in english there are two, aka: 'planner'.
As that is unreleased code switching all instances without any
transitional provisions. Also the reason why its skipped in changelog.
Thanks: Julian Andres Klode
Gbp-Dch: Ignore
|
|
In 385d9f2f23057bc5808b5e013e77ba16d1c94da4 I implemented the storage of
scenario files based on enabling this by default for EIPP, but I
implemented it first optionally for EDSP to have it independent.
The reasons mentioned in the earlier commit (debugging and bugreports)
obviously apply here, especially as EIPP solutions aren't user approved,
nearly impossible to verify before starting the execution and at the
time of error the scenario has changed already, so that reproducing the
issue becomes hard(er).
|
|
Freeing 'Install' for future use as an interface for "dpkg --install",
which is currently not used by any existent planer, so the
implementation of it itself will be delayed until then.
|
|
A rather special need option, but the internal planer supports this and
we have a testcase for it & sometimes it is hit (as a bug through). The
option itself mostly serves as a reminder for implementors that they
should be careful with removes and especially temporary removes if they
perform any.
|
|
APT has 3 modes: no immediate configuration, all packages are configured
immediately and its default mode of configuring essentials and
pseudo-essentials immediately only. While this seems like a job of
different planers at first, it might be handy to have it as an option,
too, in case a planer (like apts internal one) supports different modes
where the introduction of individual planers would be counter intuitive.
|
|
The generation of the EIPP request was a bit to strict not generation
what would actually be needed to be part of the scenario.
|
|
For the order there is no inherent difference between delete or purge,
so we don't tell the planer about this and instead decide in apt if a
package should be purged or not which also allows us to not tell the
planer about rc-only purges as we can trivially do this on our on as
there is no need to plan such purges.
|
|
Testing the current implementation can benefit from being able to be
feed an EIPP request and produce a fully compliant response. It is also
a great test for EIPP in general.
|
|
We can trim generation time and size of the EIPP scenario considerable
if we we avoid telling the planers about "uninteresting" packages.
This is one of the simpler but already very effective reductions:
Do not tell planers about versions which are neither installed nor are
to be installed as they have no effect on the plan we don't need to tell
the planer about them. EDSP solvers need to know about all versions for
better choice and error messages, but planers really don't.
Git-Dch: Ignore
|
|
The very first step in introducing the "external installation planer
protocol" (short: EIPP) as part of my GSoC2016 project.
The description reads: APT-based tools like apt-get, aptitude, synaptic,
… work with the user to figure out how their system should look like
after they are done installing/removing packages and their dependencies.
The actual installation/removal of packages is done by dpkg with the
constrain that dependencies must be fulfilled at any point in time (e.g.
to run maintainer scripts).
Historically APT has a super micro-management approach to this task
which hasn't aged that well over the years mostly ignoring changes in
dpkg and growing into an unmaintainable mess hardly anyone can debug and
everyone fears to touch – especially as more and more requirements are
tacked onto it like handling cycles and triggers, dealing with
"important" packages first, package sources on removable media, touch
minimal groups to be able to interrupt the process if needed (e.g.
unattended-upgrades) which not only sky-rocket complexity but also can
be mutually exclusive as you e.g. can't have minimal groups and minimal
trigger executions at the same time.
|
|
|
|
Most (if not all) solvers should be able to run perfectly fine without
root privileges as they get the entire state they are supposed to work
on via stdin and do not perform any action directly, but just pass
suggestions on via stdout.
The new default is to run them all as _apt hence, but each solver can
configure another user if it chooses/must. The security benefits are
minimal at best, but it helps preventing silly mistakes (see
35f3ed061f10a25a3fb28bc988fddbb976344c4d) and that is always good.
Note that our 'apt' and 'dump' solver already dropped privileges if they
had them.
|
|
Create and log the EDSP(like) request even if we use the internal
resolver.
|
|
For bugreports and co it could be handy to have the scenario and all the
settings used in it around later for inspection for EDSP like protocols.
EDSP might not be the most interesting as the user can still interrupt
the process before the solution is applied and users tend to have an
opinion on the "rightness" of a solution, so it is disabled by default.
|
|
With have better ways to compare, manipulate and work with strings, so
use it instead of counting string length by hand with is a wonder it
hasn't failed yet. Ignoreable from a changelog perspective as there is
no behaviour change.
Git-Dch: Ignore
|
|
EDSP had a WriteSolution method to write out the entire solution based
on the inspection of a given pkgDepCache, but that is rather inflexible
both for EDSP itself and for other EDSP like-protocols. It seems better
to use a smaller scope in printing just a single stanza based on a given
version as there is more reuse potential.
|
|
Currently an EDSP solver gets send basically all versions which means
the absolute count is the same, but that might not be true forever (and
with the skipping of rc-only versions it kinda is already) and even if
it were true, segfaulting on bad input seems wrong.
|
|
No real code change, just moving code around heavily to decouple the
EDSP specific parts from those we can reuse for EDSP-like protocols.
Git-Dch: Ignore
|
|
Its more space and runtime efficient to use a boolean set instead of a
CacheSet-based implementation.
Git-Dch: Ignore
|
|
This allows to differentiate properly between 'apt-get upgrade', 'apt
upgrade' and 'apt full-upgrade'.
|
|
apt/apt-pkg/edsp.cc: In function ‘bool EDSP::WriteLimitedScenario(pkgDepCache&, FILE*, const PackageSet&, OpProgress*)’:
apt/apt-pkg/edsp.cc:245:56: warning: cannot optimize loop, the loop counter may overflow [-Wunsafe-loop-optimizations]
std::string dependencies[pkgCache::Dep::Enhances + 1];
^
Using a std::array to silence gcc as well as as a code improvement feels right here.
Git-Dch: Ignore
|
|
The &= introduced in the EDSP-FileFd conversion isn't working to full
satisfaction for multiple && clauses as the && has a higher binding than
&= has, so that the methods were called even through they shouldn't
have because of previous errors. Using variadic functions we can solve
this in a slightly cleaner way bringing down the amount of 'broken pipe'
errors for the error case of the dump resolver substantially.
Git-Dch: Ignore
|
|
I doubt there is any non-src:apt usage of these interfaces.
|
|
Unexpected are for examples removal requests for versions which aren't
installed, installations of already installed versions & requests to
install and remove a package at the same time.
|
|
The spec was slightly inconsistent if the preferences setting is
available only as generic or specific setting & the code only supported
the specific one, while for the strict-pinning was only generic…
As the usual pattern for apt is to have both options we adapt the spec
and code to support both as well.
This also adds a purely informal "Solver" field so in case the request
is saved in a file, we know to which solver the sent preferences apply.
Closes: 823918
|
|
Versions which are only available in dpkg/status aren't installable and
apt doesn't pick them as candidate for this reason – for the same reason
such packages shouldn't be sent to an external solver via EDSP. The
packages are pinned to -1, but if the solver has strict pinning disabled
it could end up picking this version anyhow – which is a request apt can
not satisfy.
Reported-By: Maximiliano Curia <maxy@debian.org> on IRC
|
|
|
|
The EDSP output generated by apt didn't include the versioned provides
information so that every provides looked like an unversioned one in the
eyes of an external resolver.
|
|
This was discussed a while ago on #debian-apt and now that I see myself
making this mistake lets bite the bullet and fix it in the easy way out
version: Using a new name which fits with a similar named setter and
deprecate the old method instead of 'hostily' changing API.
Closes: #803471
|
|
In private-install.cc we call MarkInstall with FromUser=true, which sets
the bit accordingly, but while applying the EDSP solution we call mark
install on all packages with FromUser=false, so MarkInstall believes
this install is an automatic one and sets it to auto – so that a new package
which is explicitely installed via an external solver is marked as auto
and is hence also up for garbage collection in a following call.
Ideally MarkInstall wouldn't reset it, but the detection is hard to do
without regressing in other cases – and ideally ideally MarkInstall
wouldn't deal with the autobit at all – so we work around this on the
calling side for now.
|
|
The syntax of "Source" is different in EDSP compared to the the field of
the same name in 'the rest' of Debian, so documented this accordingly
and send the version as a new field.
|
|
How the Multi-Arch field and pkg:<arch> dependencies interact was
discussed at DebConf15 in the "MultiArch BoF". dpkg and apt (among other
tools like dose) had a different interpretation in certain scenarios
which we resolved by agreeing on dpkg view – and this commit realizes
this agreement in code.
As was the case so far libapt sticks to the idea of trying to hide
MultiArch as much as possible from individual frontends and instead
translates it to good old SingleArch. There are certainly situations
which can be improved in frontends if they know that MultiArch is upon
them, but these are improvements – not necessary changes needed
to unbreak a frontend.
The implementation idea is simple: If we parse a dependency on foo:amd64
the dependency is formed on a package 'foo:amd64' of arch 'any'. This
package is provided by package 'foo' of arch 'amd64', but not by 'foo'
of arch 'i386'. Both of those foo packages provide each other through
(assuming foo is M-A:foreign) to allow a dependency on 'foo' to be
satisfied by either foo of amd64 or i386. Packages can also declare to
provide 'foo:amd64' which is translated to providing 'foo:amd64:any' as
well.
This indirection over provides was chosen as the alternative would be to
teach dependency resolvers how to deal with architecture specific
dependencies – which violates the design idea of avoiding resolver
changes, especially as architecture-specific dependencies are a
cornercase with quite a few subtil rules. Handling it all over versioned
provides as we already did for M-A in general seems much simpler as it
just works for them.
This switch to :any has actually a "surprising" benefit as well: Even
frontends showing a package name via .Name() [which doesn't show the
architecture] will display the "architecture" for dependencies in which
it was explicitely requested, while we will not show the 'strange' :any
arch in FullName(true) [= pretty-print] either. Before you had to
specialcase these and by default you wouldn't get these details shown.
The only identifiable disadvantage is that this complicates error
reporting and handling. apt-get's ShowBroken has existing problems with
virtual packages [it just shows the name without any reason], so that
has to be worked on eventually. The other case is that detecting if a
package is completely unknown or if it was at least referenced somewhere
needs to acount for this "split" – not that it makes a practical
difference which error is shown… but its one of the improvements
possible.
|
|
Git-Dch: ignore
|
|
|
|
Before MultiArch implicits weren't a thing, so they were hidden by
default by definition. Adding them for MultiArch solved many problems,
but having no reliable way of detecting which dependency (and provides)
is implicit or not causes problems everytime we want to output
dependencies without confusing our observers with unneeded
implementation details.
The really notworthy point here is actually that we keep now a better
record of how a dependency came to be so that we can later reason about
it more easily, but that is hidden so deep down in the library internals
that change is more the problems it solves than the change itself.
|
|
We aren't and we will not be really compatible again with the previous
stable abi, so lets drop these markers (which never made it into a
released version) for good as they have outlived their intend already.
Git-Dch: Ignore
|
|
To have a chance to keep the ABI for a while we need all three to team
up. One of them missing and we might loose, so ensuring that they are
available is a very tedious but needed task once in a while.
Git-Dch: Ignore
|
|
We used to read the Release file for each Packages file and store the
data in the PackageFile struct even through potentially many Packages
(and Translation-*) files could use the same data. The point of the
exercise isn't the duplicated data through. Having the Release files as
first-class citizens in the Cache allows us to properly track their
state as well as allows us to use the information also for files which
aren't in the cache, but where we know to which Release file they
belong (Sources are an example for this).
This modifies the pkgCache structs, especially the PackagesFile struct
which depending on how libapt users access the data in these structs can
mean huge breakage or no visible change. As a single data point:
aptitude seems to be fine with this. Even if there is breakage it is
trivial to fix in a backportable way while avoiding breakage for
everyone would be a huge pain for us.
Note that not all PackageFile structs have a corresponding ReleaseFile.
In particular the dpkg/status file as well as *.deb files have not. As
these have only a Archive property need, the Component property takes
over this duty and the ReleaseFile remains zero. This is also the reason
why it isn't needed nor particularily recommended to change from
PackagesFile to ReleaseFile blindly. Sticking with the earlier is
usually the better option.
|
|
Git-Dch: Ignore
|
|
Accessing the package records to acquire this information is pretty
costly, so that information wasn't used so far in many places. The most
noticeable user by far is EDSP at the moment, but there are ideas to
change that which this commit tries to enable.
|
|
APT's cache can include packages from architectures dpkg has no
knowledge about and can therefore not be installed for e.g. to allow
easy lookups. There is no point in telling external solvers about them
though and some of them might even be really talkative about ignoring
them if we do.
|
|
Adds also a small testcase for EDSP
Git-Dch: Ignore
|
|
Solvers are supposed to exit successfully even if they haven't found a
solution, but a solver which fails drastically (like e.g. segfaults)
should be detected and dealt with accordingly instead of ignored.
|
