From e85987ab2281e765a2b3e19aefe7f99f656428d8 Mon Sep 17 00:00:00 2001 From: David Kalnischkies Date: Sat, 5 May 2018 01:37:47 +0200 Subject: move special READMEs into doc/ and format as md The formatting is very basic and more should probably be done, but we at least get the files out of the root directory which in case of the various READMEs was confusing salsa which one display as (central) README. --- README.SrvRecords | 22 -- README.json-hooks.md | 159 ----------- README.md | 14 + README.progress-reporting | 69 ----- debian/apt-doc.docs | 10 +- doc/acquire-additional-files.md | 325 +++++++++++++++++++++ doc/acquire-additional-files.txt | 319 --------------------- doc/apt-verbatim.ent | 2 +- doc/external-dependency-solver-protocol.md | 377 +++++++++++++++++++++++++ doc/external-dependency-solver-protocol.txt | 377 ------------------------- doc/external-installation-planner-protocol.md | 301 ++++++++++++++++++++ doc/external-installation-planner-protocol.txt | 301 -------------------- doc/json-hooks-protocol.md | 159 +++++++++++ doc/progress-reporting.md | 76 +++++ doc/srv-records-support.md | 22 ++ 15 files changed, 1281 insertions(+), 1252 deletions(-) delete mode 100644 README.SrvRecords delete mode 100644 README.json-hooks.md delete mode 100644 README.progress-reporting create mode 100644 doc/acquire-additional-files.md delete mode 100644 doc/acquire-additional-files.txt create mode 100644 doc/external-dependency-solver-protocol.md delete mode 100644 doc/external-dependency-solver-protocol.txt create mode 100644 doc/external-installation-planner-protocol.md delete mode 100644 doc/external-installation-planner-protocol.txt create mode 100644 doc/json-hooks-protocol.md create mode 100644 doc/progress-reporting.md create mode 100644 doc/srv-records-support.md diff --git a/README.SrvRecords b/README.SrvRecords deleted file mode 100644 index a3da64519..000000000 --- a/README.SrvRecords +++ /dev/null @@ -1,22 +0,0 @@ -DNS SRV record support in apt -============================= - -Apt supports a subset of the DNS SRV server records protocol as -described in RFC 2782 for service discovery. - -Before connecting to the requested server APT will send a SRV -record request of the form "_$protocol._tcp._$host", e.g. -"_http._tcp.ftp.debian.org" or "_http._tcp.security.debian.org". - -If the server sends SRV records -as a reply APT will use those to connect to the server(s) in -this reply. It will honor the "priority" field in the reply. - -However it does not implement the "weight" algorithm as described -in RFC 2782. It will use an equal weight for each server of the -same priority. - -If connecting to a server fails APT will retry with the next one -and remove the server from the list of valid servers for this -session. - diff --git a/README.json-hooks.md b/README.json-hooks.md deleted file mode 100644 index f516c1afe..000000000 --- a/README.json-hooks.md +++ /dev/null @@ -1,159 +0,0 @@ -## JSON Hooks - -APT 1.6 introduces support for hooks that talk JSON-RPC 2.0. Hooks act -as a server, and APT as a client. - -## Wire protocol - -APT communicates with hooks via a UNIX domain socket in file descriptor -`$APT_HOOK_SOCKET`. The transport is a byte stream (SOCK_STREAM). - -The byte stream contains multiple JSON objects, each representing a -JSON-RPC request or response, and each terminated by an empty line -(`\n\n`). Therefore, JSON objects containing empty lines may not be -used. - -For protocol version `0.1`, each JSON object must be encoded on a single -line. - -## Lifecycle - -The general life of a hook is as following. - -1. Hook is started -2. Hello handshake is exchanged -3. One or more calls or notifications are sent from apt to the hook -4. Bye notification is send - -It is unspecified whether a hook is sent one or more messages. For -example, a hook may be started only once for the lifetime of the apt -process and receive multiple notificatgions, but a hook may also be -started multiple times. Hooks should thus be stateless. - -## JSON messages - -### Hello handshake - -APT performs a call to the method `org.debian.apt.hooks.hello` with -the named parameter `versions` containing a list of supported protocol -versions. The hook picks the version it supports. The current version -is `"0.1"`, and support for that version is mandatory. - -*Example*: - -1. APT: - ```{"jsonrpc":"2.0","method":"org.debian.apt.hooks.hello","id":0,"params":{"versions":["0.1"]}}``` - - -2. Hook: - ```{"jsonrpc":"2.0","id":0,"result":{"version":"0.1"}}``` - -### Bye notification - -Before closing the connection, APT sends a notification for the -method `org.debian.apt.hooks.bye`. - -### Hook notification - -The following methods are supported: - -1. `org.debian.apt.hooks.install.pre-prompt` - Run before the y/n prompt -1. `org.debian.apt.hooks.install.post` - Run after success -1. `org.debian.apt.hooks.install.fail` - Run after failed install -1. `org.debian.apt.hooks.search.pre` - Run before search -1. `org.debian.apt.hooks.search.post` - Run after successful search -1. `org.debian.apt.hooks.search.fail` - Run after search without results - -They can be registered by adding them to the list: - -```AptCli::Hooks::``` - -where `` is the name of the hook. It is recommended that these -option names are prefixed with `Binary::apt`, so that they only take -effect for the `apt` binary. Otherwise, there may be compatibility issues -with scripts and alike. - -#### Parameters - -*command*: The command used on the command-line. For example, `"purge"`. - -*search-terms*: Any non-option arguments given to the command. - -*unknown-packages*: For non-search hooks, a subset of *search-terms* -that APT could not find packages in the cache for. - -*packages*: An array of modified packages. This is mostly useful for -install. Each package has the following attributes: - -- *id*: An unsigned integer describing the package -- *name*: The name of the package -- *architecture*: The architecture of the package. For `"all"` packages, this will be the native architecture; - use per-version architecture fields to see `"all"`. - -- *mode*: One of `install`, `deinstall`, `purge`, or `keep`. `keep` - is not exposed in 0.1. To determine an upgrade, check - that a current version is installed. -- *automatic*: Whether the package is/will be automatically installed -- *versions*: An array with up to 3 fields: - - - *candidate*: The candidate version - - *install*: The version to be installed - - *current*: The version currently installed - - Each version is represented as an object with the following fields: - - - *id*: An unsigned integer - - *version*: The version as a string - - *architecture*: Architecture of the version - - *pin*: The pin priority - -#### Example - -```json -{ - "jsonrpc": "2.0", - "method": "org.debian.apt.hooks.install.pre", - "params": { - "command": "purge", - "search-terms": [ - "petname-", - "lxd+" - ], - "packages": [ - { - "id": 1500, - "name": "ebtables", - "architecture": "amd64", - "mode": "install", - "automatic": 1, - "versions": { - "candidate": { - "id": 376, - "version": "2.0.10.4-3.5ubuntu2", - "architecture": "amd64", - "pin": 990 - }, - "install": { - "id": 376, - "version": "2.0.10.4-3.5ubuntu2", - "architecture": "amd64", - "pin": 990 - } - } - } - ] - } -} -``` - -#### Compatibility note -Future versions of APT might make these calls instead of notifications. - -## Evolution of this protocol -New incompatible versions may be introduced with each new feature -release of apt (1.7, 1.8, etc). No backward compatibility is promised -until protocol 1.0: New stable feature releases may support a newer -protocol version only (for example, 1.7 may only support 0.2). - -Additional fields may be added to objects without bumping the protocol -version. diff --git a/README.md b/README.md index fb1c620f8..e17f1f441 100644 --- a/README.md +++ b/README.md @@ -198,3 +198,17 @@ block the correct installation of very important packages. Which option to use d if you are interested in the topology sorting (`Debug::pkgOrderList`), the dependency-aware cycle and unconfigured prevention (`Debug::pkgPackageManager`) or the actual calls to dpkg (`Debug::pkgDpkgPm`). + + +Additional documentation +------------------------ + +Many more things could and should be said about APT and its usage but are more +targeted at developers of related programs or only of special interest. + +* [Protocol specification of APTs communication with external dependency solvers (EDSP)](./doc/external-dependency-solver-protocol.md) +* [Protocol specification of APTs communication with external installation planners (EIPP)](./doc/external-installation-planner-protocol.md) +* [Howto use and configure APT to acquire additional files in 'update' operations](./doc/acquire-additional-files.md) +* [Download and package installation progress reporting details](./doc/progress-reporting.md) +* [Remarks on DNS SRV record support in APT](./doc/srv-records-support.md) +* [Protocol specification of APT interfacing with external hooks via JSON](./doc/json-hooks-protocol.md) diff --git a/README.progress-reporting b/README.progress-reporting deleted file mode 100644 index 134ed5334..000000000 --- a/README.progress-reporting +++ /dev/null @@ -1,69 +0,0 @@ -Install-progress reporting --------------------------- - -If the apt options: "APT::Status-Fd" is set, apt will send status -reports to that fd. The status information is separated with a ':', -there are the following status conditions: - -status = {"pmstatus", "dlstatus", "conffile-prompt", "error", "media-change" } - -The reason for using a fd instead of a OpProgress class is that many -apt front-end fork a (vte) terminal for the actual installation. - -The reason to do the mapping and l10n of the dpkg states to human -readable (and translatable) strings is that this way the translation -needs to be done only once for all front-ends. - - -pmstatus --------- -Status of the package manager (dpkg). This is send when packages -are installed/removed. -pmstatus:pkgname:TotalPercentage:action-description - -PkgName = the name of the package -TotalPercentage = the total progress between [0..100] -description = a i18ned human readable description of the current action - - -Example: -# ./apt-get install -o APT::Status-Fd=2 3dchess >/dev/null -pmstatus:3dchess:20:Preparing 3dchess -pmstatus:3dchess:40:Unpacking 3dchess -pmstatus:3dchess:60:Preparing to configure 3dchess -pmstatus:3dchess:80:Configuring 3dchess -pmstatus:3dchess:100:Installed 3dchess - -pmerror -------- -pmerror:deb:TotalPercentage:error string - -Example: -pmerror: /var/cache/apt/archives/krecipes_0.8.1-0ubuntu1_i386.deb : 75% : trying to overwrite `/usr/share/doc/kde/HTML/en/krecipes/krectip.png', which is also in package krecipes-data - - -pmconffile ----------- -pmconffile:conffile:percent:'current-conffile' 'new-conffile' useredited distedited - - -media-change ------------- -media-change:medium:drive:human-readable string - -example: -media-change: Ubuntu 5.10 _Breezy Badger_ - Alpha i386 (20050830):/cdrom/:Please insert the disc labeled: 'Ubuntu 5.10 _Breezy Badger_ - Alpha i386 (20050830)' in the drive '/cdrom/' and press enter. - - -dlstatus --------- -dlstatus:AlreadDownloaded:TotalPercentage:action-description - -AlreadyDownloaded = the number of already downloaded packages -TotalPercentage = the total progress between [0..100] -description = a i18ned human readable description of the current action - -Example: -dlstatus:1:9.05654:Downloading file 1 of 3 (4m40s remaining) -dlstatus:1:9.46357:Downloading file 1 of 3 (4m39s remaining) -dlstatus:1:9.61022:Downloading file 1 of 3 (4m38s remaining) diff --git a/debian/apt-doc.docs b/debian/apt-doc.docs index bd494002c..2134da07f 100644 --- a/debian/apt-doc.docs +++ b/debian/apt-doc.docs @@ -1,4 +1,6 @@ -README.progress-reporting -doc/acquire-additional-files.txt -doc/external-dependency-solver-protocol.txt -doc/external-installation-planner-protocol.txt +doc/acquire-additional-files.md +doc/external-dependency-solver-protocol.md +doc/external-installation-planner-protocol.md +doc/json-hooks-protocol.md +doc/progress-reporting.md +doc/srv-records-support.md diff --git a/doc/acquire-additional-files.md b/doc/acquire-additional-files.md new file mode 100644 index 000000000..30c83ef25 --- /dev/null +++ b/doc/acquire-additional-files.md @@ -0,0 +1,325 @@ +# Acquire additional files in 'update' operations + +The download and verification of data from multiple sources in different +compression formats, with partial downloads and patches is an involved +process which is hard to implement correctly and securely. + +APT front-ends share the code and binaries to make this happen in libapt +with the Acquire system, supported by helpers shipped in the apt package +itself and additional transports in individual packages like +`apt-transport-https`. + +For its own operation libapt needs or can make use of *Packages*, *Sources* +and *Translation-* files, which it will acquire by default, but +a repository might contain more data files (e.g. `Contents`) a front-end +(e.g. `apt-file`) might want to use and would therefore need to be +downloaded as well. + +This file describes the configuration scheme such a front-end can use to +instruct the Acquire system to download those additional files. + +# The Configuration Stanza + +The Acquire system uses the same configuration settings to implement the +files it downloads by default. These settings are the default, but if +they would be written in a configuration file the configuration +instructing the Acquire system to download the *Packages* files would look +like this (see also `apt.conf(5)` manpage for configuration file syntax): + + Acquire::IndexTargets::deb::Packages { + MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages"; + ShortDescription "Packages"; + Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages"; + + flatMetaKey "Packages"; + flatDescription "$(RELEASE) Packages"; + + Optional "no"; + }; + +All files which should be downloaded (nicknamed *Targets*) are mentioned +below the `Acquire::IndexTargets` scope. `deb` is here the type of the +`sources.list` entry the file should be acquired for. The only other +supported value is hence `deb-src`. Beware: You can't specify multiple +types here and you can't download the same (evaluated) `MetaKey` from +multiple types! + +After the type you can pick any valid and unique string which preferable +refers to the file it downloads (In the example we picked *Packages*). +This string is used as identifier (if not explicitly set otherwise) for +the target class and accessible as `Identifier` and `Created-By` e.g. +in the `apt-get indextargets` output as detailed below. The identifier +is also used to allow user to enable/disable targets per sources.list +entry. + +All targets have three main properties you can define: + +* `MetaKey`: The identifier of the file to be downloaded as used in the + Release file. It is also the relative location of the file from the + Release file. You can neither download from a different server + entirely (absolute URI) nor access directories above the Release file + (e.g. "../../"). +* `ShortDescription`: Very short string intended to be displayed to the + user e.g. while reporting progress. apt will e.g. use this string in + the last line to indicate progress of e.g. the download of a specific + item. +* `Description`: A preferable human understandable and readable identifier + of which file is acquired exactly. Mainly used for progress reporting + and error messages. apt will e.g. use this string in the Get/Hit/Err + progress lines. + An identifier of the site accessed as seen in the sources.list (e.g. + `http://example.org/debian` or `file:/path/to/a/repository`) is + automatically prefixed for this property. + + +Additional optional properties: + +* `Identifier`: The default value is the unique string identifying this + file (in the example above it was *Packages*) also accessible as + `Created-By`. The difference is that using this property multiple files + can be subsumed under one identifier e.g. if you configure multiple + possible locations for the files (with `Fallback-Of`), but the front-end + doesn't need to handle files from the different locations differently. +* `DefaultEnabled`: The default value is `yes` which means that apt will + try to acquire this target from all sources. If set to `no` the user + has to explicitly enable this target in the sources.list file with the + `Targets` option(s) – or override this value in a config file. +* `Optional`: The default value is `yes` and should be kept at this value. + If enabled the acquire system will skip the download if the file isn't + mentioned in the `Release` file. Otherwise this is treated as a hard + error and the update process fails. Note that failures while + downloading (e.g. 404 or hash verification errors) are failures, + regardless of this setting. +* `KeepCompressed`: The default is the value of `Acquire::GzipIndexes`, + which defaults to `false`. If `true`, the acquire system will keep the + file compressed on disk rather than extract it. If your front-end can't + deal with compressed files transparently you have to explicitly set + this option to `false` to avoid problems with users setting the option + globally. On the other hand, if you set it to `true` or don't set it you + have to ensure your front-end can deal with all compressed fileformats + supported by apt (libapt users can e.g. use `FileFd`, others can use + the `cat-file` command of `/usr/lib/apt/apt-helper`). +* `Fallback-Of`: Is by default not set. If it is set and specifies another + target name (see `Created-By`) which was found in the *Release* file the + download of this target will be skipped. This can be used to implement + fallback(chain)s to allow transitions like the rename of target files. + The behavior if cycles are formed with Fallback-Of is undefined! +* `flatMetaKey`, `flatDescription`: APT supports two types of repositories: + dists-style repositories which are the default and by far the most + common which are named after the fact that the files are in an + elaborated directory structure. In contrast a flat-style repository + lumps all files together in one directory. Support for these flat + repositories exists mainly for legacy purposes only. It is therefore + recommend to not set these values. + + +The acquire system will automatically choose to download a compressed +file if it is available and uncompress it for you, just as it will also +use PDiff patching if provided by the repository and enabled by the +user. You only have to ensure that the Release file contains the +information about the compressed files/PDiffs to make this happen. +**NO** properties have to be set to enable this! + + +More properties exist, but these should **NOT** be set by front-ends +requesting files. They exist for internal and end-user usage only. +Some of these are – which are documented here only to ensure that they +aren't accidentally used by front-ends: + +* `PDiffs`: controls if apt will try to use PDiffs for this target. + Defaults to the value of `Acquire::PDiffs` which is *true* by default. + Can be overridden per-source by the sources.list option of the same + name. See the documentation for both of these for details. +* `By-Hash`: controls if apt will try to use an URI constructed from + a hashsum of the file to download. See the documentation for config + option `Acquire::By-Hash` and sources.list option `By-Hash` for details. +* `CompressionTypes`: The default value is a space separated list of + compression types supported by apt (see `Acquire::CompressionTypes`). + You can set this option to prevent apt from downloading a compression + type a front-end can't open transparently. This should always be + a temporary workaround through and a bug should be reported against + the front-end in question. +* `KeepCompressedAs`: The default value is a space separated list of + compression types supported by apt (see previous option) which is + sorted by the cost-value of the compression in ascending order, + except that cost=0 "compressions" (like uncompressed) are listed last. + + +# More examples + +The stanzas for `Translation-*` files as well as for `Sources` files would +look like this: + +Acquire::IndexTargets { + deb::Translations { + MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)"; + ShortDescription "Translation-$(LANGUAGE)"; + Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)"; + + flatMetaKey "$(LANGUAGE)"; + flatDescription "$(RELEASE) Translation-$(LANGUAGE)"; + }; + + deb-src::Sources { + MetaKey "$(COMPONENT)/source/Sources"; + ShortDescription "Sources"; + Description "$(RELEASE)/$(COMPONENT) Sources"; + + flatMetaKey "Sources"; + flatDescription "$(RELEASE) Sources"; + + Optional "no"; + }; +}; + +# Substitution variables + +As seen in the examples, properties can contain placeholders filled in +by the acquire system. The following variables are known; note that +unknown variables have no default value nor are they touched: They are +printed as-is. + +* `$(RELEASE)`: This is usually an archive- or codename, e.g. *stable* or + *stretch*. Note that flat-style repositories do not have an archive- + or codename per-se, so the value might very well be just "/" or so. +* `$(COMPONENT)`: as given in the sources.list, e.g. *main*, *non-free* or + *universe*. Note that flat-style repositories again do not really + have a meaningful value here. +* `$(LANGUAGE)`: Values are all entries (expect *none*) of configuration + option `Acquire::Languages`, e.g. *en*, *de* or *de_AT*. +* `$(ARCHITECTURE)`: Values are all entries of configuration option + `APT::Architectures` (potentially modified by sources.list options), + e.g. *amd64*, *i386* or *armel* for the *deb* type. In type *deb-src* + this variable has the value *source*. +* `$(NATIVE_ARCHITECTURE)`: The architecture apt treats as the native + architecture for this system configured as `APT::Architecture` + defaulting to the architecture apt itself was built for. + +Note that while more variables might exist in the implementation, these +are to be considered undefined and their usage strongly discouraged. If +you have a need for other variables contact us. + +# Accessing files + +Do **NOT** hardcode specific file locations, names or compression types in +your application! You will notice that the configuration options give +you no choice over where the downloaded files will be stored. This is by +design so multiple applications can download and use the same file +rather than each and every one of them potentially downloads and uses +its own copy somewhere on disk. + +`apt-get indextargets` can be used to get the location as well as other +information about all files downloaded (aka: you will see *Packages*, +*Sources* and *Translation-* files here as well). Provide a line of the +default output format as parameter to filter out all entries which do +not have such a line. With `--format`, you can further more define your +own output style. The variables are what you see in the output, just all +uppercase and wrapped in `$()`, as in the configuration file. + +To get all the filenames of all *Translation-en* files you can e.g. call: + + apt-get indextargets --format '$(FILENAME)' "Identifier: Translations" "Language: en" + +The line-based filtering and the formatting is rather crude and feature- +less by design: The default format is Debian's standard format `deb822` +(in particular: Field names are case-insensitive and the order of fields +in the stanza is undefined), so instead of apt reimplementing powerful +filters and formatting for this command, it is recommend to use piping +and dedicated tools like `grep-dctrl` if you need more than the basics +provided. + +Accessing this information via libapt is done by reading the +sources.lists (`pkgSourceList`), iterating over the metaIndex objects this +creates and calling `GetIndexTargets()` on them. See the source code of +`apt-get indextargets` for a complete example. + +Note that by default targets are not listed if they weren't downloaded. +If you want to see all targets, you can use the `--no-release-info`, which +also removes the *Codename*, *Suite*, *Version*, *Origin*, *Label* and *Trusted* +fields from the output as these also display data which needs to be +downloaded first and could hence be inaccurate [on the pro-side: This +mode is faster as it doesn't require a valid binary cache to operate]. +The most notable difference perhaps is in the *Filename* field through: By +default it indicates an existing file, potentially compressed (Hint: +libapt users can use `FileFd` to open compressed files transparently). In +the `--no-release-info` mode the indicated file doesn't need to exist and +it will always refer to an uncompressed file, even if the index would be +(or is) stored compressed. + +Remarks on fields only available in (default) `--release-info mode`: + +* `Trusted`: Denotes with a *yes* or *no* if the data in this file is + authenticated by a trust chain rooted in a trusted gpg key. You should + be careful with untrusted data and warn the user if you use it. +* `Codename`, `Suite`, `Version`, `Origin` and `Label` are fields from the + *Release* file, are only present if they are present in the *Release* file + and contain the same data. + +Remarks on other available fields: + +* `MetaKey`, `ShortDesc`, `Description`, `Site`, `Release`: as defined + by the configuration and described further above. +* `Identifier`: Defaults to the value of `Created-By`, but can be set + explicitly in the configuration (see above). Prefer this field over + `Created-By` to subsume multiple file(location)s (see `Fallback-Of`). +* `Created-By`: configuration entity responsible for this target +* `Target-Of`: type of the sources.list entry +* `URI`, `Repo-URI`: avoid using. Contains potentially username/password. + Prefer `Site`, especially for display. +* `Optional`, `DefaultEnabled`, `KeepCompressed`: Decode the options of the + same name from the configuration. +* `Language`, `Architecture`, `Component`: as defined further above, but with + the catch that they might be missing if they don't effect the target + (aka: They weren't used while evaluating the `MetaKey` template). + +Again, additional fields might be visible in certain implementations, +but you should avoid using them and instead talk to us about a portable +implementation. + +# Multiple applications requiring the same files + +It is highly encouraged that applications talk to each other and to us +about which files they require. It is usually best to have a common +package ship the configuration needed to get the files, but specific +needs might require specific solutions. Again: **talk to us**. + +Bad things will happen if multiple front-ends request the same file(s) +via different targets, which is another reason why coordination is very +important! + +# Acquiring files not mentioned in the Release file + +You can't. This is by design as these files couldn't be verified to not +be modified in transit, corrupted by the download process or simple if +they are present at all on the server, which would require apt to probe +for them. APT did this in the past for legacy reasons, we do not intend +to go back to these dark times. + +This is also why you can't request files from a different server. It +would have the additional problem that this server might not even be +accessible (e.g. proxy settings) or that local sources (file:/, cdrom:/) +start requesting online files… + +In other words: We would be opening Pandora's box. + +# Acquiring files to a specific location on disk + +You can't by design to avoid multiple front-ends requesting the same file +to be downloaded to multiple different places on (different) disks +(among other reasons). See the next point for a solution if you really +have to force a specific location by creating symlinks. + +# Post processing the acquired files + +You can't modify the files apt has downloaded as apt keeps state with +e.g. the modification times of the files and advanced features like +PDiffs break. + +You can however install an `APT::Update::Post-Invoke{-Success,}` hook +script and use them to copy (modified) files to a different location. +Use `apt-get indextargets` (or similar) to get the filenames – do not +look into `/var/lib/apt/lists` directly! + +Please avoid time consuming calculations in the scripts and instead just +trigger a background task as there is little to no feedback for the user +while hook scripts run. diff --git a/doc/acquire-additional-files.txt b/doc/acquire-additional-files.txt deleted file mode 100644 index c55d5fe19..000000000 --- a/doc/acquire-additional-files.txt +++ /dev/null @@ -1,319 +0,0 @@ -# Acquire additional files in 'update' operations - -The download and verification of data from multiple sources in different -compression formats, with partial downloads and patches is an involved -process which is hard to implement correctly and securely. - -APT front-ends share the code and binaries to make this happen in libapt -with the Acquire system, supported by helpers shipped in the apt package -itself and additional transports in individual packages like -apt-transport-https. - -For its own operation libapt needs or can make use of Packages, Sources -and Translation-* files, which it will acquire by default, but -a repository might contain more data files (e.g. Contents) a front-end -(e.g. apt-file) might want to use and would therefore need to be -downloaded as well. - -This file describes the configuration scheme such a front-end can use to -instruct the Acquire system to download those additional files. - -# The Configuration Stanza - -The Acquire system uses the same configuration settings to implement the -files it downloads by default. These settings are the default, but if -they would be written in a configuration file the configuration -instructing the Acquire system to download the Packages files would look -like this (see also apt.conf(5) manpage for configuration file syntax): - - Acquire::IndexTargets::deb::Packages { - MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages"; - ShortDescription "Packages"; - Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages"; - - flatMetaKey "Packages"; - flatDescription "$(RELEASE) Packages"; - - Optional "no"; - }; - -All files which should be downloaded (nicknamed 'Targets') are mentioned -below the Acquire::IndexTargets scope. 'deb' is here the type of the -sources.list entry the file should be acquired for. The only other -supported value is hence 'deb-src'. Beware: You can't specify multiple -types here and you can't download the same (evaluated) MetaKey from -multiple types! - -After the type you can pick any valid and unique string which preferable -refers to the file it downloads (In the example we picked 'Packages'). -This string is used as identifier (if not explicitly set otherwise) for -the target class and accessible as 'Identifier' and 'Created-By' e.g. -in the "apt-get indextargets" output as detailed below. The identifier -is also used to allow user to enable/disable targets per sources.list -entry. - -All targets have three main properties you can define: -* MetaKey: The identifier of the file to be downloaded as used in the - Release file. It is also the relative location of the file from the - Release file. You can neither download from a different server - entirely (absolute URI) nor access directories above the Release file - (e.g. "../../"). -* ShortDescription: Very short string intended to be displayed to the - user e.g. while reporting progress. apt will e.g. use this string in - the last line to indicate progress of e.g. the download of a specific - item. -* Description: A preferable human understandable and readable identifier - of which file is acquired exactly. Mainly used for progress reporting - and error messages. apt will e.g. use this string in the Get/Hit/Err - progress lines. - An identifier of the site accessed as seen in the sources.list (e.g. - "http://example.org/debian" or "file:/path/to/a/repository") is - automatically prefixed for this property. - - -Additional optional properties: -* Identifier: The default value is the unique string identifying this - file (in the example above it was 'Packages') also accessible as - Created-By. The difference is that using this property multiple files - can be subsumed under one identifier e.g. if you configure multiple - possible locations for the files (with Fallback-Of), but the front-end - doesn't need to handle files from the different locations differently. -* DefaultEnabled: The default value is 'yes' which means that apt will - try to acquire this target from all sources. If set to 'no' the user - has to explicitly enable this target in the sources.list file with the - Targets option(s) – or override this value in a config file. -* Optional: The default value is 'yes' and should be kept at this value. - If enabled the acquire system will skip the download if the file isn't - mentioned in the Release file. Otherwise this is treated as a hard - error and the update process fails. Note that failures while - downloading (e.g. 404 or hash verification errors) are failures, - regardless of this setting. -* KeepCompressed: The default is the value of Acquire::GzipIndexes, - which defaults to false. If true, the acquire system will keep the - file compressed on disk rather than extract it. If your front-end can't - deal with compressed files transparently you have to explicitly set - this option to false to avoid problems with users setting the option - globally. On the other hand, if you set it to true or don't set it you - have to ensure your front-end can deal with all compressed fileformats - supported by apt (libapt users can e.g. use FileFd, others can use - the cat-file command of /usr/lib/apt/apt-helper). -* Fallback-Of: Is by default not set. If it is set and specifies another - target name (see Created-By) which was found in the Release file the - download of this target will be skipped. This can be used to implement - fallback(chain)s to allow transitions like the rename of target files. - The behavior if cycles are formed with Fallback-Of is undefined! -* flat{MetaKey,Description}: APT supports two types of repositories: - dists-style repositories which are the default and by far the most - common which are named after the fact that the files are in an - elaborated directory structure. In contrast a flat-style repository - lumps all files together in one directory. Support for these flat - repositories exists mainly for legacy purposes only. It is therefore - recommend to not set these values. - - -The acquire system will automatically choose to download a compressed -file if it is available and uncompress it for you, just as it will also -use PDiff patching if provided by the repository and enabled by the -user. You only have to ensure that the Release file contains the -information about the compressed files/PDiffs to make this happen. -*NO* properties have to be set to enable this! - - -More properties exist, but these should *NOT* be set by front-ends -requesting files. They exist for internal and end-user usage only. -Some of these are – which are documented here only to ensure that they -aren't accidentally used by front-ends: -* PDiffs: controls if apt will try to use PDiffs for this target. - Defaults to the value of Acquire::PDiffs which is true by default. - Can be overridden per-source by the sources.list option of the same - name. See the documentation for both of these for details. -* By-Hash: controls if apt will try to use an URI constructed from - a hashsum of the file to download. See the documentation for config - option Acquire::By-Hash and sources.list option By-Hash for details. -* CompressionTypes: The default value is a space separated list of - compression types supported by apt (see Acquire::CompressionTypes). - You can set this option to prevent apt from downloading a compression - type a front-end can't open transparently. This should always be - a temporary workaround through and a bug should be reported against - the front-end in question. -* KeepCompressedAs: The default value is a space separated list of - compression types supported by apt (see previous option) which is - sorted by the cost-value of the compression in ascending order, - except that cost=0 "compressions" (like uncompressed) are listed last. - - -# More examples - -The stanzas for Translation-* files as well as for Sources files would -look like this: - -Acquire::IndexTargets { - deb::Translations { - MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)"; - ShortDescription "Translation-$(LANGUAGE)"; - Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)"; - - flatMetaKey "$(LANGUAGE)"; - flatDescription "$(RELEASE) Translation-$(LANGUAGE)"; - }; - - deb-src::Sources { - MetaKey "$(COMPONENT)/source/Sources"; - ShortDescription "Sources"; - Description "$(RELEASE)/$(COMPONENT) Sources"; - - flatMetaKey "Sources"; - flatDescription "$(RELEASE) Sources"; - - Optional "no"; - }; -}; - -# Substitution variables - -As seen in the examples, properties can contain placeholders filled in -by the acquire system. The following variables are known; note that -unknown variables have no default value nor are they touched: They are -printed as-is. - -* $(RELEASE): This is usually an archive- or codename, e.g. "stable" or - "stretch". Note that flat-style repositories do not have an archive- - or codename per-se, so the value might very well be just "/" or so. -* $(COMPONENT): as given in the sources.list, e.g. "main", "non-free" or - "universe". Note that flat-style repositories again do not really - have a meaningful value here. -* $(LANGUAGE): Values are all entries (expect "none") of configuration - option Acquire::Languages, e.g. "en", "de" or "de_AT". -* $(ARCHITECTURE): Values are all entries of configuration option - APT::Architectures (potentially modified by sources.list options), - e.g. "amd64", "i386" or "armel" for the 'deb' type. In type 'deb-src' - this variable has the value "source". -* $(NATIVE_ARCHITECTURE): The architecture apt treats as the native - architecture for this system configured as APT::Architecture - defaulting to the architecture apt itself was built for. - -Note that while more variables might exist in the implementation, these -are to be considered undefined and their usage strongly discouraged. If -you have a need for other variables contact us. - -# Accessing files - -Do NOT hardcode specific file locations, names or compression types in -your application! You will notice that the configuration options give -you no choice over where the downloaded files will be stored. This is by -design so multiple applications can download and use the same file -rather than each and every one of them potentially downloads and uses -its own copy somewhere on disk. - -"apt-get indextargets" can be used to get the location as well as other -information about all files downloaded (aka: you will see Packages, -Sources and Translation-* files here as well). Provide a line of the -default output format as parameter to filter out all entries which do -not have such a line. With --format, you can further more define your -own output style. The variables are what you see in the output, just all -uppercase and wrapped in $(), as in the configuration file. - -To get all the filenames of all Translation-en files you can e.g. call: - apt-get indextargets --format '$(FILENAME)' "Identifier: Translations" "Language: en" - -The line-based filtering and the formatting is rather crude and feature- -less by design: The default format is Debian's standard format deb822 -(in particular: Field names are case-insensitive and the order of fields -in the stanza is undefined), so instead of apt reimplementing powerful -filters and formatting for this command, it is recommend to use piping -and dedicated tools like 'grep-dctrl' if you need more than the basics -provided. - -Accessing this information via libapt is done by reading the -sources.lists (pkgSourceList), iterating over the metaIndex objects this -creates and calling GetIndexTargets() on them. See the source code of -"apt-get indextargets" for a complete example. - -Note that by default targets are not listed if they weren't downloaded. -If you want to see all targets, you can use the --no-release-info, which -also removes the Codename, Suite, Version, Origin, Label and Trusted -fields from the output as these also display data which needs to be -downloaded first and could hence be inaccurate [on the pro-side: This -mode is faster as it doesn't require a valid binary cache to operate]. -The most notable difference perhaps is in the Filename field through: By -default it indicates an existing file, potentially compressed (Hint: -libapt users can use FileFd to open compressed files transparently). In -the --no-release-info mode the indicated file doesn't need to exist and -it will always refer to an uncompressed file, even if the index would be -(or is) stored compressed. - -Remarks on fields only available in (default) --release-info mode: -* Trusted: Denotes with a 'yes' or 'no' if the data in this file is - authenticated by a trust chain rooted in a trusted gpg key. You should - be careful with untrusted data and warn the user if you use it. -* Codename, Suite, Version, Origin and Label are fields from the Release - file, are only present if they are present in the Release file and - contain the same data. - -Remarks on other available fields: -* MetaKey, ShortDesc, Description, Site, Release: as defined - by the configuration and described further above. -* Identifier: Defaults to the value of Created-By, but can be set - explicitly in the configuration (see above). Prefer this field over - Created-By to subsume multiple file(location)s (see Fallback-Of). -* Created-By: configuration entity responsible for this target -* Target-Of: type of the sources.list entry -* URI, Repo-URI: avoid using. Contains potentially username/password. - Prefer 'Site', especially for display. -* Optional, DefaultEnabled, KeepCompressed: Decode the options of the - same name from the configuration. -* Language, Architecture, Component: as defined further above, but with - the catch that they might be missing if they don't effect the target - (aka: They weren't used while evaluating the MetaKey template). - -Again, additional fields might be visible in certain implementations, -but you should avoid using them and instead talk to us about a portable -implementation. - -# Multiple applications requiring the same files - -It is highly encouraged that applications talk to each other and to us -about which files they require. It is usually best to have a common -package ship the configuration needed to get the files, but specific -needs might require specific solutions. Again: talk to us. - -Bad things will happen if multiple front-ends request the same file(s) -via different targets, which is another reason why coordination is very -important! - -# Acquiring files not mentioned in the Release file - -You can't. This is by design as these files couldn't be verified to not -be modified in transit, corrupted by the download process or simple if -they are present at all on the server, which would require apt to probe -for them. APT did this in the past for legacy reasons, we do not intend -to go back to these dark times. - -This is also why you can't request files from a different server. It -would have the additional problem that this server might not even be -accessible (e.g. proxy settings) or that local sources (file:/, cdrom:/) -start requesting online files… - -In other words: We would be opening Pandora's box. - -# Acquiring files to a specific location on disk - -You can't by design to avoid multiple front-ends requesting the same file -to be downloaded to multiple different places on (different) disks -(among other reasons). See the next point for a solution if you really -have to force a specific location by creating symlinks. - -# Post processing the acquired files - -You can't modify the files apt has downloaded as apt keeps state with -e.g. the modification times of the files and advanced features like -PDiffs break. - -You can however install an APT::Update::Post-Invoke{-Success,} hook -script and use them to copy (modified) files to a different location. -Use 'apt-get indextargets' (or similar) to get the filenames – do not -look into /var/lib/apt/lists directly! - -Please avoid time consuming calculations in the scripts and instead just -trigger a background task as there is little to no feedback for the user -while hook scripts run. diff --git a/doc/apt-verbatim.ent b/doc/apt-verbatim.ent index 53e8e54d4..168a39012 100644 --- a/doc/apt-verbatim.ent +++ b/doc/apt-verbatim.ent @@ -225,7 +225,7 @@ " > -/usr/share/doc/apt-doc/acquire-additional-files.txt"> +/usr/share/doc/apt-doc/acquire-additional-files.md.gz"> ``` + +where `` is the name of the hook. It is recommended that these +option names are prefixed with `Binary::apt`, so that they only take +effect for the `apt` binary. Otherwise, there may be compatibility issues +with scripts and alike. + +#### Parameters + +*command*: The command used on the command-line. For example, `"purge"`. + +*search-terms*: Any non-option arguments given to the command. + +*unknown-packages*: For non-search hooks, a subset of *search-terms* +that APT could not find packages in the cache for. + +*packages*: An array of modified packages. This is mostly useful for +install. Each package has the following attributes: + +- *id*: An unsigned integer describing the package +- *name*: The name of the package +- *architecture*: The architecture of the package. For `"all"` packages, this will be the native architecture; + use per-version architecture fields to see `"all"`. + +- *mode*: One of `install`, `deinstall`, `purge`, or `keep`. `keep` + is not exposed in 0.1. To determine an upgrade, check + that a current version is installed. +- *automatic*: Whether the package is/will be automatically installed +- *versions*: An array with up to 3 fields: + + - *candidate*: The candidate version + - *install*: The version to be installed + - *current*: The version currently installed + + Each version is represented as an object with the following fields: + + - *id*: An unsigned integer + - *version*: The version as a string + - *architecture*: Architecture of the version + - *pin*: The pin priority + +#### Example + +```json +{ + "jsonrpc": "2.0", + "method": "org.debian.apt.hooks.install.pre", + "params": { + "command": "purge", + "search-terms": [ + "petname-", + "lxd+" + ], + "packages": [ + { + "id": 1500, + "name": "ebtables", + "architecture": "amd64", + "mode": "install", + "automatic": 1, + "versions": { + "candidate": { + "id": 376, + "version": "2.0.10.4-3.5ubuntu2", + "architecture": "amd64", + "pin": 990 + }, + "install": { + "id": 376, + "version": "2.0.10.4-3.5ubuntu2", + "architecture": "amd64", + "pin": 990 + } + } + } + ] + } +} +``` + +#### Compatibility note +Future versions of APT might make these calls instead of notifications. + +## Evolution of this protocol +New incompatible versions may be introduced with each new feature +release of apt (1.7, 1.8, etc). No backward compatibility is promised +until protocol 1.0: New stable feature releases may support a newer +protocol version only (for example, 1.7 may only support 0.2). + +Additional fields may be added to objects without bumping the protocol +version. diff --git a/doc/progress-reporting.md b/doc/progress-reporting.md new file mode 100644 index 000000000..c4963af68 --- /dev/null +++ b/doc/progress-reporting.md @@ -0,0 +1,76 @@ +Install-progress reporting +-------------------------- + +If the apt options `APT::Status-Fd` is set, apt will send status +reports to that fd. The status information is separated with a '`:`', +there are the following status conditions: + +* pmstatus +* dlstatus +* conffile-prompt +* error +* media-change + +The reason for using a fd instead of an OpProgress class is that many +apt front-end fork a (vte) terminal for the actual installation. + +The reason to do the mapping and l10n of the dpkg states to human +readable (and translatable) strings is that this way the translation +needs to be done only once for all front-ends. + + +pmstatus +-------- +Status of the package manager (dpkg). This is send when packages +are installed/removed. + pmstatus:pkgname:TotalPercentage:action-description + +* PkgName = the name of the package +* TotalPercentage = the total progress between [0..100] +* description = a i18ned human readable description of the current action + +Example: + + # ./apt-get install -o APT::Status-Fd=2 3dchess >/dev/null + pmstatus:3dchess:20:Preparing 3dchess + pmstatus:3dchess:40:Unpacking 3dchess + pmstatus:3dchess:60:Preparing to configure 3dchess + pmstatus:3dchess:80:Configuring 3dchess + pmstatus:3dchess:100:Installed 3dchess + +pmerror +------- + pmerror:deb:TotalPercentage:error string + +Example: + + pmerror: /var/cache/apt/archives/krecipes_0.8.1-0ubuntu1_i386.deb : 75% : trying to overwrite `/usr/share/doc/kde/HTML/en/krecipes/krectip.png', which is also in package krecipes-data + + +pmconffile +---------- + pmconffile:conffile:percent:'current-conffile' 'new-conffile' useredited distedited + + +media-change +------------ + media-change:medium:drive:human-readable string + +Example: + + media-change: Ubuntu 5.10 _Breezy Badger_ - Alpha i386 (20050830):/cdrom/:Please insert the disc labeled: 'Ubuntu 5.10 _Breezy Badger_ - Alpha i386 (20050830)' in the drive '/cdrom/' and press enter. + + +dlstatus +-------- + dlstatus:AlreadDownloaded:TotalPercentage:action-description + +* AlreadyDownloaded = the number of already downloaded packages +* TotalPercentage = the total progress between [0..100] +* description = a i18ned human readable description of the current action + +Example: + + dlstatus:1:9.05654:Downloading file 1 of 3 (4m40s remaining) + dlstatus:1:9.46357:Downloading file 1 of 3 (4m39s remaining) + dlstatus:1:9.61022:Downloading file 1 of 3 (4m38s remaining) diff --git a/doc/srv-records-support.md b/doc/srv-records-support.md new file mode 100644 index 000000000..004811cf2 --- /dev/null +++ b/doc/srv-records-support.md @@ -0,0 +1,22 @@ +DNS SRV record support in apt +============================= + +Apt supports a subset of the DNS SRV server records protocol as +described in [RFC 2782](https://tools.ietf.org/html/rfc2782) for service discovery. + +Before connecting to the requested server APT will send a SRV +record request of the form `_$protocol._tcp._$host`, e.g. +`_http._tcp.ftp.debian.org` or `_http._tcp.security.debian.org`. + +If the server sends SRV records +as a reply APT will use those to connect to the server(s) in +this reply. It will honor the `priority` field in the reply. + +However it does not implement the `weight` algorithm as described +in RFC 2782. It will use an equal weight for each server of the +same priority. + +If connecting to a server fails APT will retry with the next one +and remove the server from the list of valid servers for this +session. + -- cgit v1.2.3