summaryrefslogtreecommitdiff
path: root/apt-pkg/edsp.cc
AgeCommit message (Collapse)Author
2016-12-31ensure generation of valid EDSP error stanzasDavid Kalnischkies
The crude way of preparing a message to be a multiline value failed at generation valid deb822 in case the error message ended with a new line like the resolving errors from apt do. apt itself can parse these, but other tools like grep-dctrl choke on it, so be nice and print valid. Reported-By: Johannes 'josch' Schauer on IRC
2016-09-07edsp: try 2 to read responses even if writing failedDavid Kalnischkies
Commit b60c8a89c281f2bb945d426d2215cbf8f5760738 improved the situation, but due to inconsistency mostly for planners, not for solvers. As the idea of hiding errors if we show another error is a bit scary (as the extern error might be a followup of our intern error, rather than the reason for our intern error as it is at the moment) we don't discard the errors, but if we got an extern error we show them directly removing them from the error list at the end of the run – that list will contain the extern error which hopefully gives us the best of both worlds. The problem itself is the same as before: The externals exiting before apt is done talking to them. Reported-By: Johannes 'josch' Schauer on IRC
2016-07-29edsp: try to read responses even if writing failedDavid Kalnischkies
If a solver/planner exits before apt is done writing we will generate write errors. Solvers like 'dump' can be pretty quick in failing but produce a valid EDSP error report apt should read, parse and display instead of just discarding even through we had write errors.
2016-07-27(error) va_list 'args' was opened but not closed by va_end()David Kalnischkies
Reported-By: cppcheck Gbp-Dch: Ignore
2016-07-27eipp: avoid producing file warnings in simulationDavid Kalnischkies
Simulations are frequently run by unprivileged users which naturally don't have the permissions to write to the default location for the eipp file. Either way is bad as running in simulation mode doesn't mean we don't want to run the logging (as EIPP runs the same regardless of simulation or 'real' run), but showing the warnings is relatively pointless in the default setup, so, in case we would produce errors and perform a simulation we will discard the warnings and carry on. Running apt with an external planner wouldn't have generated these messages btw. Closes: 832614
2016-07-05EIPP/EDSP log can't be written is a warning, not an errorDavid Kalnischkies
If other logs can't be written this is a warning to, so for consistency sake translate the errors to warnings.
2016-07-05report write errors in EDSP/EIPP properly back to callerDavid Kalnischkies
Unlikely to happen in practice and I wonder more how I could miss these in earlier reviews, but okay, lets fix it for consistency now.
2016-07-02use +0000 instead of UTC by default as timezone in outputDavid Kalnischkies
All apt versions support numeric as well as 3-character timezones just fine and its actually hard to write code which doesn't "accidently" accepts it. So why change? Documenting the Date/Valid-Until fields in the Release file is easy to do in terms of referencing the datetime format used e.g. in the Debian changelogs (policy §4.4). This format specifies only the numeric timezones through, not the nowadays obsolete 3-character ones, so in the interest of least surprise we should use the same format even through it carries a small risk of regression in other clients (which encounter repositories created with apt-ftparchive). In case it is really regressing in practice, the hidden option -o APT::FTPArchive::Release::NumericTimezone=0 can be used to go back to good old UTC as timezone. The EDSP and EIPP protocols use this 'new' format, the text interface used to communicate with the acquire methods does not for compatibility reasons even if none of our methods would be effected and I doubt any other would (in these instances the timezone is 'GMT' as that is what HTTP/1.1 requires). Note that this is only true for apt talking to methods, (libapt-based) methods talking to apt will respond with the 'new' format. It is therefore strongly adviced to support both also in method input.
2016-06-29eipp: let apt make a plan, not make stuff planeDavid Kalnischkies
Julian noticed on IRC that I fall victim to a lovely false friend by calling referring to a 'planer' all the time even through these are machines to e.g. remove splinters from woodwork ("make stuff plane"). The term I meant is written in german in this way (= with a single n) but in english there are two, aka: 'planner'. As that is unreleased code switching all instances without any transitional provisions. Also the reason why its skipped in changelog. Thanks: Julian Andres Klode Gbp-Dch: Ignore
2016-06-27eipp: enable xz-compressed scenario loggingDavid Kalnischkies
In 385d9f2f23057bc5808b5e013e77ba16d1c94da4 I implemented the storage of scenario files based on enabling this by default for EIPP, but I implemented it first optionally for EDSP to have it independent. The reasons mentioned in the earlier commit (debugging and bugreports) obviously apply here, especially as EIPP solutions aren't user approved, nearly impossible to verify before starting the execution and at the time of error the scenario has changed already, so that reproducing the issue becomes hard(er).
2016-06-27eipp: rename stanza 'Install' to 'Unpack'David Kalnischkies
Freeing 'Install' for future use as an interface for "dpkg --install", which is currently not used by any existent planer, so the implementation of it itself will be delayed until then.
2016-06-27eipp: add Allow-Temporary-Remove-of-EssentialsDavid Kalnischkies
A rather special need option, but the internal planer supports this and we have a testcase for it & sometimes it is hit (as a bug through). The option itself mostly serves as a reminder for implementors that they should be careful with removes and especially temporary removes if they perform any.
2016-06-27eipp: implement Immediate-Configuration flagDavid Kalnischkies
APT has 3 modes: no immediate configuration, all packages are configured immediately and its default mode of configuring essentials and pseudo-essentials immediately only. While this seems like a job of different planers at first, it might be handy to have it as an option, too, in case a planer (like apts internal one) supports different modes where the introduction of individual planers would be counter intuitive.
2016-06-27eipp: properly handle arch-specific providesDavid Kalnischkies
The generation of the EIPP request was a bit to strict not generation what would actually be needed to be part of the scenario.
2016-06-27eipp: make no difference between remove & purgeDavid Kalnischkies
For the order there is no inherent difference between delete or purge, so we don't tell the planer about this and instead decide in apt if a package should be purged or not which also allows us to not tell the planer about rc-only purges as we can trivially do this on our on as there is no need to plan such purges.
2016-06-27eipp: provide the internal planer as an external oneDavid Kalnischkies
Testing the current implementation can benefit from being able to be feed an EIPP request and produce a fully compliant response. It is also a great test for EIPP in general.
2016-06-27eipp: output at most two versions per packageDavid Kalnischkies
We can trim generation time and size of the EIPP scenario considerable if we we avoid telling the planers about "uninteresting" packages. This is one of the simpler but already very effective reductions: Do not tell planers about versions which are neither installed nor are to be installed as they have no effect on the plan we don't need to tell the planer about them. EDSP solvers need to know about all versions for better choice and error messages, but planers really don't. Git-Dch: Ignore
2016-06-27eipp: implement version 0.1 of the protocolDavid Kalnischkies
The very first step in introducing the "external installation planer protocol" (short: EIPP) as part of my GSoC2016 project. The description reads: APT-based tools like apt-get, aptitude, synaptic, … work with the user to figure out how their system should look like after they are done installing/removing packages and their dependencies. The actual installation/removal of packages is done by dpkg with the constrain that dependencies must be fulfilled at any point in time (e.g. to run maintainer scripts). Historically APT has a super micro-management approach to this task which hasn't aged that well over the years mostly ignoring changes in dpkg and growing into an unmaintainable mess hardly anyone can debug and everyone fears to touch – especially as more and more requirements are tacked onto it like handling cycles and triggers, dealing with "important" packages first, package sources on removable media, touch minimal groups to be able to interrupt the process if needed (e.g. unattended-upgrades) which not only sky-rocket complexity but also can be mutually exclusive as you e.g. can't have minimal groups and minimal trigger executions at the same time.
2016-06-10don't leak EDSP solver output fdDavid Kalnischkies
2016-06-08edsp: drop privileges before executing solversDavid Kalnischkies
Most (if not all) solvers should be able to run perfectly fine without root privileges as they get the entire state they are supposed to work on via stdin and do not perform any action directly, but just pass suggestions on via stdout. The new default is to run them all as _apt hence, but each solver can configure another user if it chooses/must. The security benefits are minimal at best, but it helps preventing silly mistakes (see 35f3ed061f10a25a3fb28bc988fddbb976344c4d) and that is always good. Note that our 'apt' and 'dump' solver already dropped privileges if they had them.
2016-06-08edsp: if logging is requested, do it for internal, tooDavid Kalnischkies
Create and log the EDSP(like) request even if we use the internal resolver.
2016-06-08edsp: optionally store a compressed copy of the last scenarioDavid Kalnischkies
For bugreports and co it could be handy to have the scenario and all the settings used in it around later for inspection for EDSP like protocols. EDSP might not be the most interesting as the user can still interrupt the process before the solution is applied and users tend to have an opinion on the "rightness" of a solution, so it is disabled by default.
2016-06-06get right of the hardcoded string-length comparesDavid Kalnischkies
With have better ways to compare, manipulate and work with strings, so use it instead of counting string length by hand with is a wonder it hasn't failed yet. Ignoreable from a changelog perspective as there is no behaviour change. Git-Dch: Ignore
2016-06-04edsp: use a stanza based interface for solution writingDavid Kalnischkies
EDSP had a WriteSolution method to write out the entire solution based on the inspection of a given pkgDepCache, but that is rather inflexible both for EDSP itself and for other EDSP like-protocols. It seems better to use a smaller scope in printing just a single stanza based on a given version as there is more reuse potential.
2016-06-04edsp: use an ID mapping for the internal solverDavid Kalnischkies
Currently an EDSP solver gets send basically all versions which means the absolute count is the same, but that might not be true forever (and with the skipping of rc-only versions it kinda is already) and even if it were true, segfaulting on bad input seems wrong.
2016-05-20refactor EDSP code into EDSP and EDSP-like partsDavid Kalnischkies
No real code change, just moving code around heavily to decouple the EDSP specific parts from those we can reuse for EDSP-like protocols. Git-Dch: Ignore
2016-05-20edsp: use a limited scenario based on bool-arrayDavid Kalnischkies
Its more space and runtime efficient to use a boolean set instead of a CacheSet-based implementation. Git-Dch: Ignore
2016-05-20edsp: add Forbid-{New-Install,Remove} and Upgrade-AllDavid Kalnischkies
This allows to differentiate properly between 'apt-get upgrade', 'apt upgrade' and 'apt full-upgrade'.
2016-05-20avoid triggering gcc's -Wunsafe-loop-optimizations in EDSPDavid Kalnischkies
apt/apt-pkg/edsp.cc: In function ‘bool EDSP::WriteLimitedScenario(pkgDepCache&, FILE*, const PackageSet&, OpProgress*)’: apt/apt-pkg/edsp.cc:245:56: warning: cannot optimize loop, the loop counter may overflow [-Wunsafe-loop-optimizations] std::string dependencies[pkgCache::Dep::Enhances + 1]; ^ Using a std::array to silence gcc as well as as a code improvement feels right here. Git-Dch: Ignore
2016-05-20edsp: try harder to not generate unneeded error messagesDavid Kalnischkies
The &= introduced in the EDSP-FileFd conversion isn't working to full satisfaction for multiple && clauses as the && has a higher binding than &= has, so that the methods were called even through they shouldn't have because of previous errors. Using variadic functions we can solve this in a slightly cleaner way bringing down the amount of 'broken pipe' errors for the error case of the dump resolver substantially. Git-Dch: Ignore
2016-05-20convert EDSP to be based on FileFd instead of FILE*David Kalnischkies
I doubt there is any non-src:apt usage of these interfaces.
2016-05-12edsp: warn if unexpected stanzas appear in the solutionDavid Kalnischkies
Unexpected are for examples removal requests for versions which aren't installed, installations of already installed versions & requests to install and remove a package at the same time.
2016-05-10edsp: support generic and solver-specific configsDavid Kalnischkies
The spec was slightly inconsistent if the preferences setting is available only as generic or specific setting & the code only supported the specific one, while for the strict-pinning was only generic… As the usual pattern for apt is to have both options we adapt the spec and code to support both as well. This also adds a purely informal "Solver" field so in case the request is saved in a file, we know to which solver the sent preferences apply. Closes: 823918
2016-05-10don't sent uninstallable rc-only versions via EDSPDavid Kalnischkies
Versions which are only available in dpkg/status aren't installable and apt doesn't pick them as candidate for this reason – for the same reason such packages shouldn't be sent to an external solver via EDSP. The packages are pinned to -1, but if the solver has strict pinning disabled it could end up picking this version anyhow – which is a request apt can not satisfy. Reported-By: Maximiliano Curia <maxy@debian.org> on IRC
2016-04-25edsp: ask policy engine for the pin of the version directlyDavid Kalnischkies
2016-02-16pass versioned provides to external solvers in EDSPDavid Kalnischkies
The EDSP output generated by apt didn't include the versioned provides information so that every provides looked like an unversioned one in the eyes of an external resolver.
2015-11-04sanify API to get 'the' candidate versionDavid Kalnischkies
This was discussed a while ago on #debian-apt and now that I see myself making this mistake lets bite the bullet and fix it in the easy way out version: Using a new name which fits with a similar named setter and deprecate the old method instead of 'hostily' changing API. Closes: #803471
2015-09-14do not discard new manual-bits while applying EDSP solutionsDavid Kalnischkies
In private-install.cc we call MarkInstall with FromUser=true, which sets the bit accordingly, but while applying the EDSP solution we call mark install on all packages with FromUser=false, so MarkInstall believes this install is an automatic one and sets it to auto – so that a new package which is explicitely installed via an external solver is marked as auto and is hence also up for garbage collection in a following call. Ideally MarkInstall wouldn't reset it, but the detection is hard to do without regressing in other cases – and ideally ideally MarkInstall wouldn't deal with the autobit at all – so we work around this on the calling side for now.
2015-09-14add Source-Version field for EDSPDavid Kalnischkies
The syntax of "Source" is different in EDSP compared to the the field of the same name in 'the rest' of Debian, so documented this accordingly and send the version as a new field.
2015-09-14implement dpkgs vision of interpreting pkg:<arch> dependenciesDavid Kalnischkies
How the Multi-Arch field and pkg:<arch> dependencies interact was discussed at DebConf15 in the "MultiArch BoF". dpkg and apt (among other tools like dose) had a different interpretation in certain scenarios which we resolved by agreeing on dpkg view – and this commit realizes this agreement in code. As was the case so far libapt sticks to the idea of trying to hide MultiArch as much as possible from individual frontends and instead translates it to good old SingleArch. There are certainly situations which can be improved in frontends if they know that MultiArch is upon them, but these are improvements – not necessary changes needed to unbreak a frontend. The implementation idea is simple: If we parse a dependency on foo:amd64 the dependency is formed on a package 'foo:amd64' of arch 'any'. This package is provided by package 'foo' of arch 'amd64', but not by 'foo' of arch 'i386'. Both of those foo packages provide each other through (assuming foo is M-A:foreign) to allow a dependency on 'foo' to be satisfied by either foo of amd64 or i386. Packages can also declare to provide 'foo:amd64' which is translated to providing 'foo:amd64:any' as well. This indirection over provides was chosen as the alternative would be to teach dependency resolvers how to deal with architecture specific dependencies – which violates the design idea of avoiding resolver changes, especially as architecture-specific dependencies are a cornercase with quite a few subtil rules. Handling it all over versioned provides as we already did for M-A in general seems much simpler as it just works for them. This switch to :any has actually a "surprising" benefit as well: Even frontends showing a package name via .Name() [which doesn't show the architecture] will display the "architecture" for dependencies in which it was explicitely requested, while we will not show the 'strange' :any arch in FullName(true) [= pretty-print] either. Before you had to specialcase these and by default you wouldn't get these details shown. The only identifiable disadvantage is that this complicates error reporting and handling. apt-get's ShowBroken has existing problems with virtual packages [it just shows the name without any reason], so that has to be worked on eventually. The other case is that detecting if a package is completely unknown or if it was at least referenced somewhere needs to acount for this "split" – not that it makes a practical difference which error is shown… but its one of the improvements possible.
2015-08-17Fix all the wrong removals of includes that iwyu got wrongMichael Vogt
Git-Dch: ignore
2015-08-17Cleanup includes after running iwyuMichael Vogt
2015-08-10hide implicit deps in apt-cache again by defaultDavid Kalnischkies
Before MultiArch implicits weren't a thing, so they were hidden by default by definition. Adding them for MultiArch solved many problems, but having no reliable way of detecting which dependency (and provides) is implicit or not causes problems everytime we want to output dependencies without confusing our observers with unneeded implementation details. The really notworthy point here is actually that we keep now a better record of how a dependency came to be so that we can later reason about it more easily, but that is hidden so deep down in the library internals that change is more the problems it solves than the change itself.
2015-08-10remove the compatibility markers for 4.13 abiDavid Kalnischkies
We aren't and we will not be really compatible again with the previous stable abi, so lets drop these markers (which never made it into a released version) for good as they have outlived their intend already. Git-Dch: Ignore
2015-06-16add d-pointer, virtual destructors and de-inline de/constructorsDavid Kalnischkies
To have a chance to keep the ABI for a while we need all three to team up. One of them missing and we might loose, so ensuring that they are available is a very tedious but needed task once in a while. Git-Dch: Ignore
2015-06-12store Release files data in the CacheDavid Kalnischkies
We used to read the Release file for each Packages file and store the data in the PackageFile struct even through potentially many Packages (and Translation-*) files could use the same data. The point of the exercise isn't the duplicated data through. Having the Release files as first-class citizens in the Cache allows us to properly track their state as well as allows us to use the information also for files which aren't in the cache, but where we know to which Release file they belong (Sources are an example for this). This modifies the pkgCache structs, especially the PackagesFile struct which depending on how libapt users access the data in these structs can mean huge breakage or no visible change. As a single data point: aptitude seems to be fine with this. Even if there is breakage it is trivial to fix in a backportable way while avoiding breakage for everyone would be a huge pain for us. Note that not all PackageFile structs have a corresponding ReleaseFile. In particular the dpkg/status file as well as *.deb files have not. As these have only a Archive property need, the Component property takes over this duty and the ReleaseFile remains zero. This is also the reason why it isn't needed nor particularily recommended to change from PackagesFile to ReleaseFile blindly. Sticking with the earlier is usually the better option.
2014-11-08guard ABI changes for SourcePkg/Ver in pkgCacheDavid Kalnischkies
Git-Dch: Ignore
2014-09-27store source name and version in binary cacheDavid Kalnischkies
Accessing the package records to acquire this information is pretty costly, so that information wasn't used so far in many places. The most noticeable user by far is EDSP at the moment, but there are ideas to change that which this commit tries to enable.
2014-06-18don't send pkg from an unknown architecture via EDSPDavid Kalnischkies
APT's cache can include packages from architectures dpkg has no knowledge about and can therefore not be installed for e.g. to allow easy lookups. There is no point in telling external solvers about them though and some of them might even be really talkative about ignoring them if we do.
2014-05-30support parsing EDSP requests Architecture{,s} stanzaDavid Kalnischkies
Adds also a small testcase for EDSP Git-Dch: Ignore