Merge branch 'feature/amtshilfe' into 'master'

reimplement mirror method

See merge request apt-team/apt!1

12 files changed, 478 insertions, 293 deletions

diff --git a/doc/CMakeLists.txt b/doc/CMakeLists.txt
index 73d808c64..7cca4cf81 100644
--- a/doc/CMakeLists.txt
+++ b/doc/CMakeLists.txt
@@ -84,6 +84,9 @@ add_docbook(apt-man MANPAGE ALL
         apt_preferences.5.xml
         apt-secure.8.xml
         apt-sortpkgs.1.xml
+        apt-transport-http.1.xml
+        apt-transport-https.1.xml
+        apt-transport-mirror.1.xml
         sources.list.5.xml
     DEPENDS ${ENTITIES}
     TRANSLATED_ENTITIES ${TRANSLATED_ENTITIES}
diff --git a/doc/apt-transport-http.1.xml b/doc/apt-transport-http.1.xml
new file mode 100644
index 000000000..546e47761
--- /dev/null
+++ b/doc/apt-transport-http.1.xml
@@ -0,0 +1,138 @@
+<?xml version="1.0" encoding="utf-8" standalone="no"?>
+<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
+  "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
+<!ENTITY % aptent SYSTEM "apt.ent"> %aptent;
+<!ENTITY % aptverbatiment SYSTEM "apt-verbatim.ent"> %aptverbatiment;
+<!ENTITY % aptvendor SYSTEM "apt-vendor.ent"> %aptvendor;
+]>
+
+<refentry>
+
+ <refentryinfo>
+   &apt-author.team;
+   &apt-email;
+   &apt-product;
+   <!-- The last update date -->
+   <date>2017-11-22T00:00:00Z</date>
+ </refentryinfo>
+
+ <refmeta>
+   <refentrytitle>apt-transport-http</refentrytitle>
+   <manvolnum>1</manvolnum>
+   <refmiscinfo class="manual">APT</refmiscinfo>
+ </refmeta>
+
+ <!-- Man page title -->
+ <refnamediv>
+    <refname>apt-transport-http</refname>
+    <refpurpose>APT transport for downloading via the Hypertext Transfer Protocol (HTTP)</refpurpose>
+ </refnamediv>
+
+<refsect1><title>Description</title>
+<para>This APT transport allows the use of repositories accessed via the
+Hypertext Transfer Protocol (HTTP). It is available by default and probably the
+most used of all transports. Note that a transport is never called directly by
+a user but used by APT tools based on user configuration.</para>
+<para>HTTP is an unencrypted transport protocol meaning that the
+whole communication with the remote server (or proxy) can be observed by a
+sufficiently capable attacker referred to commonly as man in the middle (MITM).
+Such an attacker can <emphasis>not</emphasis> modify the communication to compromise
+the security of your system through as APTs data security model is independent of the
+chosen transport method. This is explained in detail in &apt-secure;. An overview over
+available transport methods is given in &sources-list;.</para>
+</refsect1>
+
+<refsect1><title>Options</title>
+<para>Various options are available to modify its behaviour which can be set in
+an &apt-conf; file ranging from proxy configuration to workaround for specific
+server insufficiencies.</para>
+
+<refsect2><title>Proxy Configuration</title>
+<para>The environment variable <envar>http_proxy</envar> is supported for system wide configuration.
+Proxies specific to apt can be configured via the option <literal>Acquire::http::Proxy</literal>.
+Proxies which should be used only for certain hosts can be specified via
+<literal>Acquire::http::Proxy::<replaceable>host</replaceable></literal>. Even more finegrained control
+can be achieved via proxy autodetection detailed further below.
+All these options use the URI format <literal><replaceable>scheme</replaceable>://[[<replaceable>user</replaceable>][:<replaceable>pass</replaceable>]@]<replaceable>host</replaceable>[:<replaceable>port</replaceable>]/</literal>.
+Supported URI schemes are <literal>socks5h</literal> (SOCKS5 with remote DNS resolution), <literal>http</literal> and <literal>https</literal>.
+Authentification details can be supplied via &apt-authconf; instead of including it in the URI directly.</para>
+<para>The various APT configuration options support the special value <literal>DIRECT</literal> meaning that
+no proxy should be used. The environment variable <envar>no_proxy</envar> with the same propose is also supported.</para>
+<para>Further more there are three settings provided for cache control with HTTP/1.1 compliant proxy caches:
+<literal>Acquire::http::No-Cache</literal> tells the proxy not to use its
+cached response under any circumstances.
+<literal>Acquire::http::Max-Age</literal> sets the allowed maximum age (in
+seconds) of an index file in the cache of the proxy.
+<literal>Acquire::http::No-Store</literal> specifies that the proxy should not
+store the requested archive files in its cache, which can be used to prevent
+the proxy from polluting its cache with (big) .deb files.</para>
+</refsect2>
+
+<refsect2><title>Automatic Proxy Configuration</title>
+<para><literal>Acquire::http::Proxy-Auto-Detect</literal> can be used to
+specify an external command to discover the http proxy to use. The first
+and only parameter is an URI denoting the host to be contacted to allow
+for host-specific configuration. APT expects the command to output the
+proxy on stdout as a single line in the previously specified URI format
+or the word <literal>DIRECT</literal> if no proxy should be used. No output
+indicates that the generic proxy settings should be used.</para>
+<para>Note that auto-detection will not be used for a host if a host-specific proxy
+configuration is already set via <literal>Acquire::http::Proxy::<replaceable>host</replaceable></literal>.</para>
+<para>See the &squid-deb-proxy-client; and &auto-apt-proxy; packages for example implementations.</para>
+<para>This option takes precedence over the legacy option name <literal>Acquire::http::ProxyAutoDetect</literal>.</para>
+</refsect2>
+
+<refsect2><title>Connection Configuration</title>
+<para>The option <literal>Acquire::http::Timeout</literal> sets the timeout timer used by the method;
+this value applies to the connection as well as the data timeout.</para>
+<para>The used bandwidth can be limited with
+<literal>Acquire::http::Dl-Limit</literal> which accepts integer values in
+kilobytes per second. The default value is 0 which deactivates the limit and
+tries to use all available bandwidth. Note that this option implicitly
+disables downloading from multiple servers at the same time.</para>
+<para>The setting <literal>Acquire::http::Pipeline-Depth</literal> can be used to
+enable HTTP pipelining (RFC 2616 section 8.1.2.2) which can be beneficial e.g. on
+high-latency connections. It specifies how many requests are sent in a pipeline.
+APT tries to detect and workaround misbehaving webservers and proxies at runtime, but
+if you know that yours does not conform to the HTTP/1.1 specification pipelining can
+be disabled by setting the value to 0. It is enabled by default with the value 10.</para>
+<para><literal>Acquire::http::AllowRedirect</literal> controls whether APT will follow
+redirects, which is enabled by default.</para>
+<para><literal>Acquire::http::User-Agent</literal> can be used to set a different
+User-Agent for the http download method as some proxies allow access for clients
+only if the client uses a known identifier.</para>
+<para><literal>Acquire::http::SendAccept</literal> is enabled by default and
+sends a <literal>Accept: text/*</literal> header field to the server for
+requests without file extensions to prevent the server from attempting content
+negotiation.</para>
+</refsect2>
+</refsect1>
+
+<refsect1><title>Examples</title>
+<literallayout>
+Acquire::http {
+	Proxy::example.org "DIRECT";
+	Proxy "socks5h://apt:pass@localhost:9050";
+	Proxy-Auto-Detect "/usr/local/bin/apt-http-proxy-auto-detect";
+	No-Cache "true";
+	Max-Age "3600";
+	No-Store "true";
+	Timeout "10";
+	Dl-Limit "42";
+	Pipeline-Depth "0";
+	AllowRedirect "false";
+	User-Agent "My APT-HTTP";
+	SendAccept "false";
+};
+</literallayout>
+</refsect1>
+
+<refsect1>
+<title>See Also</title>
+<para>&apt-conf; &apt-authconf; &sources-list;
+</para>
+</refsect1>
+
+ &manbugs;
+
+</refentry>
diff --git a/doc/apt-transport-https.1.xml b/doc/apt-transport-https.1.xml
new file mode 100644
index 000000000..97137fc2c
--- /dev/null
+++ b/doc/apt-transport-https.1.xml
@@ -0,0 +1,133 @@
+<?xml version="1.0" encoding="utf-8" standalone="no"?>
+<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
+  "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
+<!ENTITY % aptent SYSTEM "apt.ent"> %aptent;
+<!ENTITY % aptverbatiment SYSTEM "apt-verbatim.ent"> %aptverbatiment;
+<!ENTITY % aptvendor SYSTEM "apt-vendor.ent"> %aptvendor;
+]>
+
+<refentry>
+
+ <refentryinfo>
+   &apt-author.team;
+   &apt-email;
+   &apt-product;
+   <!-- The last update date -->
+   <date>2017-11-22T00:00:00Z</date>
+ </refentryinfo>
+
+ <refmeta>
+   <refentrytitle>apt-transport-https</refentrytitle>
+   <manvolnum>1</manvolnum>
+   <refmiscinfo class="manual">APT</refmiscinfo>
+ </refmeta>
+
+ <!-- Man page title -->
+ <refnamediv>
+    <refname>apt-transport-https</refname>
+    <refpurpose>APT transport for downloading via the HTTP Secure protocol (HTTPS)</refpurpose>
+ </refnamediv>
+
+<refsect1><title>Description</title>
+<para>This APT transport allows the use of repositories accessed via the
+HTTP Secure protocol (HTTPS) also referred to as HTTP over TLS. It is available
+by default since apt 1.5 and was before that available in a <package>apt-transport-https</package>
+package. Note that a transport is never called directly by
+a user but used by APT tools based on user configuration.</para>
+<para>HTTP is by itself an unencrypted transport protocol (compare &apt-transport-http;),
+which, as indicated by the appended S is wrapped in an encrypted layer known as
+Transport Layer Security (TLS) which provides end-to-end encryption.
+A sufficiently capable attacker can still observe the communication partners
+and deeper analyse of the encrypted communcation might still reveal important details.
+An overview over available alternative transport methods is given in &sources-list;.</para>
+</refsect1>
+
+<refsect1><title>Options</title>
+<para>The HTTPS protocol is based on the HTTP protocol and as such this implementation
+has the same relation meaning that all options supported by &apt-transport-http; are also
+available via <literal>Acquire::https</literal> and will default to the same values specified
+for <literal>Acquire::http</literal>. This manpage will only document the options
+<emphasis>unique to https</emphasis>.</para>
+
+<refsect2><title>Server credentials</title>
+<para>By default all certificates trusted by the system (see
+<package>ca-certificates</package> package) are used for the verification of
+the server certificate. An alternative certificate authority (CA) can be
+configured with the <literal>Acquire::https::CAInfo</literal> option and its
+host-specific option <literal>Acquire::https::CAInfo::<replaceable>host</replaceable></literal>.
+The option specifies a file is made of the concatenation of the CA certificates
+(in PEM format) creating the chain used for the verification of the path
+from the root (self signed one). If the remote server provides the
+whole chain during the exchange, the file need only contain the root
+certificate. Otherwise, the whole chain is required. If you need to support
+multiple authorities, the only way is to concatenate everything.</para>
+<para>A custom certificate revocation list (CRL) can be configured with the options
+<literal>Acquire::https::CRLFile</literal> and
+<literal>Acquire::https::CRLFile::<replaceable>host</replaceable></literal> respectively.
+Like the previous option a file in PEM format needs to be specified.</para>
+</refsect2>
+
+<refsect2><title>Disabling security</title>
+<para>When authenticating the server, if the certificate verification fails
+for some reason (expired, revoked, man in the middle, …), the connection fails.
+This is obviously what you want in all cases and what the default value (true)
+of the option <literal>Acquire::https::Verify-Peer</literal> and its host-specific
+variant provides. If you know <emphasis>exactly</emphasis> what you are doing,
+setting this option to "false" allows you to skip peer certificate verification and
+make the exchange succeed. Again, this option is for debugging or testing purpose
+only as it removes all security provided by the use of HTTPS.</para>
+<para>Similarly the option <literal>Acquire::https::Verify-Host</literal> and its
+host-specific variant can be used to deactivate a security feature: The certificate
+provided by the server includes the identity of the server which should match the
+DNS name used to access it. By default, as requested by RFC 2818, the name of the
+mirror is checked against the identity found in the certificate. This default behavior
+is safe and should not be changed, but if you know that the server you are using has a
+DNS name which does not match the identity in its certificate, you can set the option to
+"false", which will prevent the comparison to be done.</para>
+</refsect2>
+
+<refsect2><title>Client authentication</title>
+<para>Beside the password based authentication available (see &apt-authconf;) HTTPS supports
+authentication based on client certificates as well via <literal>Acquire::https::SSLCert</literal>
+and <literal>Acquire::https::SSLKey</literal>. They respectively should be set to the filename of
+the X.509 client certificate and the associated (unencrypted) private key, both in PEM format.
+In practice the use of the host-specific variants of both options is highly recommended.</para>
+</refsect2>
+
+</refsect1>
+
+<refsect1><title>Examples</title>
+<literallayout>
+Acquire::https {
+	Proxy::example.org "DIRECT";
+	Proxy "socks5h://apt:pass@localhost:9050";
+	Proxy-Auto-Detect "/usr/local/bin/apt-https-proxy-auto-detect";
+	No-Cache "true";
+	Max-Age "3600";
+	No-Store "true";
+	Timeout "10";
+	Dl-Limit "42";
+	Pipeline-Depth "0";
+	AllowRedirect "false";
+	User-Agent "My APT-HTTPS";
+	SendAccept "false";
+
+	CAInfo "/path/to/ca/certs.pem";
+	CRLFile "/path/to/all/crl.pem";
+	Verify-Peer "true";
+	Verify-Host::broken.example.org "false";
+	SSLCert::example.org "/path/to/client/cert.pem";
+	SSLKey::example.org "/path/to/client/key.pem"
+};
+</literallayout>
+</refsect1>
+
+<refsect1>
+<title>See Also</title>
+<para>&apt-transport-http; &apt-conf; &apt-authconf; &sources-list;
+</para>
+</refsect1>
+
+ &manbugs;
+
+</refentry>
diff --git a/doc/apt-transport-mirror.1.xml b/doc/apt-transport-mirror.1.xml
new file mode 100644
index 000000000..72001fad5
--- /dev/null
+++ b/doc/apt-transport-mirror.1.xml
@@ -0,0 +1,150 @@
+<?xml version="1.0" encoding="utf-8" standalone="no"?>
+<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
+  "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
+<!ENTITY % aptent SYSTEM "apt.ent"> %aptent;
+<!ENTITY % aptverbatiment SYSTEM "apt-verbatim.ent"> %aptverbatiment;
+<!ENTITY % aptvendor SYSTEM "apt-vendor.ent"> %aptvendor;
+]>
+
+<refentry>
+
+ <refentryinfo>
+   &apt-author.team;
+   &apt-email;
+   &apt-product;
+   <!-- The last update date -->
+   <date>2017-12-09T00:00:00Z</date>
+ </refentryinfo>
+
+ <refmeta>
+   <refentrytitle>apt-transport-mirror</refentrytitle>
+   <manvolnum>1</manvolnum>
+   <refmiscinfo class="manual">APT</refmiscinfo>
+ </refmeta>
+
+ <!-- Man page title -->
+ <refnamediv>
+    <refname>apt-transport-mirror</refname>
+    <refpurpose>APT transport for more automated mirror selection</refpurpose>
+ </refnamediv>
+
+<refsect1><title>Description</title>
+<para>This APT transport isn't implementing a protocol to access local or remote repositories
+on its own, but acquires a mirrorlist and redirects all requests to the mirror(s) picked from
+this list access via other transports like &apt-transport-http;. The basic functionality is
+available since apt 0.7.24, but was undocumented until apt 1.6 which contained a complete
+rework of the transport and its supported features. Note that a transport is never called directly
+by a user but used by APT tools based on user configuration.</para>
+<para>If the acquisition of a file via a mirror fails the method ensures that automatically
+another possible mirror of the list is tried until either the file is retrieved or no mirror is
+left in the list handling transparently server downtimes and similar problems.</para>
+<para>The security implications of the transport are based on the security considerations
+associated with the transport used to acquire the mirrorlist and the transports involved in
+accessing the chosen mirror(s) by the transport.</para>
+</refsect1>
+
+<refsect1><title>Options</title>
+<para>This transport has no configuration options at present. The mirror selection is
+based entirely on the mirrors offered in the mirrorlist and the files apt needs to
+acquire.</para>
+
+<refsect2><title>Mirrorlist format</title>
+<para>A mirrorlist contains at least one line each specifying an URI for a mirror.
+Empty lines and those starting with a hash key (<literal>#</literal>) are ignored.
+An URI always starts with a URI scheme which defines the transport used for this
+mirror. If the URI e.g. starts with <literal>http:</literal> the responsible transport
+is &apt-transport-http; which might have specific requirements for the format of
+the remaining part of the URI.</para>
+<para>An URI can optionally be separated from metadata about the mirror by a tab.
+Multiple datapoints in the provided metadata can itself be separated by spaces for tabs.
+(This is an advanced feature only available with apt >= 1.6. Earlier apt versions will
+fail to parse mirrorlists using this feature)</para>
+<para>Since apt 1.6 the usage of compressed mirrorlists is also supported.
+Note that the filename of the mirrorlist must specify the compression algorithm used,
+there is no auto-detection based on file content performed.</para>
+</refsect2>
+
+<refsect2><title>Mirror selection by metadata</title>
+<para>As specified in the format a mirror can have additional metadata attached to
+prevent a mirror from being selected for acquiring a file not matching this metadata.
+This way the mirrorlist can e.g. contain partial mirrors serving only certain
+architectures and apt will automatically choose a different mirror for files requiring
+an unlisted architecture. Supported are limits for the architecture (<literal>arch</literal>),
+codename of the release (<literal>codename</literal>), component of the repository
+the file is in (<literal>component</literal>), language the file applies to (<literal>lang</literal>),
+suite name of the release (<literal>suite</literal>) and type of the file (<literal>type</literal>).</para>
+</refsect2>
+
+<refsect2><title>Fallback order for mirrors</title>
+<para>If no priority is given via the metadata key <literal>priority</literal> for a
+mirror the order in which mirrors are contacted is random. If a certain set of mirrors
+should be tried first before any of another set is tried a priority can be explicitly
+set. The mirrors with the lowest number are tried first. Mirrors which have no explicit
+priority set default to the highest possible number and are therefore tried last. The
+choice between mirrors with the same priority is again random.</para>
+</refsect2>
+
+<refsect2><title>Allowed transports in a mirrorlist</title>
+<para>The availability and choice of transports in a mirrorlist is limited by how the apt
+client is accessing the mirrorlist. If a local transport like <literal>file</literal>
+or <literal>copy</literal> is used the mirrorlist can also include local sources while a
+mirrorlist accessed via <literal>http</literal> can not. Additionally, a mirrorlist can
+not contain a mirrorlist or other wrapping transports (like <package>apt-transport-tor</package>).
+See the documentation of these transports on how to use them with the mirror method.</para>
+<para>Note that apt versions before 1.6 do not support any other transport than <literal>http</literal>.</para>
+</refsect2>
+</refsect1>
+
+<refsect1><title>Examples</title>
+<refsect2><title>Basic example</title>
+<para>A basic mirrorlist example supported by all apt versions with a mirror method
+(>= 0.7.24) in which the client will pick any of the three mirrors:</para>
+<literallayout>
+http://ftp.de.debian.org/debian/
+http://ftp.us.debian.org/debian/
+http://deb.debian.org/debian/
+</literallayout>
+<para>Assuming a file with this content is stored as <filename>/etc/apt/mirrorlist.txt</filename>
+on your machine it can be used like this in &sources-list; (since apt 1.6):</para>
+<literallayout>
+deb mirror+file:/etc/apt/mirrorlist.txt &debian-stable-codename; main
+</literallayout>
+<para>All versions of the mirror method support a mirrorlist accessible via http, so assuming
+it is available at <literal>http://apt.example.org/mirror.lst</literal> the sources.list entry
+from above could be written instead as:</para>
+<literallayout>
+deb mirror://apt.example.org/mirror.lst &debian-stable-codename; main
+</literallayout>
+<para>Note that since apt 1.6 the use of <literal>mirror+http</literal> should
+be preferred over <literal>mirror</literal> for uniformity. The functionality is the same.</para>
+</refsect2>
+<refsect2><title>Example with metadata-enhanced mirror selection</title>
+<para>As explained in the format definition apt versions before 1.6 do not support this and
+will fail parsing the mirrorlist. The example mirrorlist is proposefully complicated to show some
+aspects of the selection. The following setup is assumed: The first mirror is local mirror
+accessible via the file method, but potentially incomplete. The second mirror has a great
+connection, but is a partial mirror in sofar as it only contains files related
+to the architectures <literal>amd64</literal> and <literal>all</literal>. The remaining mirrors
+are average mirrors which should be contacted only if the earlier ones didn't work.
+</para>
+<literallayout>
+file:/srv/local/debian/mirror/	priority:1 type:index
+http://partial.example.org/mirror/	priority:2 arch:amd64 arch:all type:deb
+http://ftp.us.debian.org/debian/	type:deb
+http://ftp.de.debian.org/debian/	type:deb
+https://deb.debian.org/debian/
+</literallayout>
+<para>In this setup with this mirrorlist the first mirror will be used to download all
+index files assuming the mirrorlist itself is accessed via a local transport like
+<literal>file</literal>. If it isn't, the mirror is otherwise inaccessible or does not
+contain the requested file another mirror will be used to acquire the file, which one
+depending on the type of the file: An index file will be served by the last
+mirror in the list, while a package of architecture <literal>amd64</literal> is served by
+the second and those of e.g. architecture <literal>i386</literal> by one of the last three.</para>
+
+</refsect2>
+</refsect1>
+
+ &manbugs;
+
+</refentry>
diff --git a/doc/apt-verbatim.ent b/doc/apt-verbatim.ent
index 946c7cc7d..8d0a56b6c 100644
--- a/doc/apt-verbatim.ent
+++ b/doc/apt-verbatim.ent
@@ -81,6 +81,17 @@
   </citerefentry>"
 >
 
+<!ENTITY apt-transport-http "<citerefentry>
+    <refentrytitle><command>apt-transport-http</command></refentrytitle>
+    <manvolnum>1</manvolnum>
+  </citerefentry>"
+>
+
+<!ENTITY apt-transport-https "<citerefentry>
+    <refentrytitle><command>apt-transport-https</command></refentrytitle>
+    <manvolnum>1</manvolnum>
+  </citerefentry>"
+>
 
 <!ENTITY sources-list "<citerefentry>
     <refentrytitle><filename>sources.list</filename></refentrytitle>
@@ -148,6 +159,12 @@
   </citerefentry>"
 >
 
+<!ENTITY auto-apt-proxy "<citerefentry>
+    <refentrytitle><command>auto-apt-proxy</command></refentrytitle>
+    <manvolnum>1</manvolnum>
+  </citerefentry>"
+>
+
 <!ENTITY debsign "<citerefentry>
     <refentrytitle><command>debsign</command></refentrytitle>
     <manvolnum>1</manvolnum>
diff --git a/doc/apt.conf.5.xml b/doc/apt.conf.5.xml
index 6f47bb029..e0be6a37d 100644
--- a/doc/apt.conf.5.xml
+++ b/doc/apt.conf.5.xml
@@ -409,99 +409,12 @@ APT::Compressor::rev {
      be symlinked when possible instead of copying. True is the default.</para></listitem>
      </varlistentry>
 
-     <varlistentry><term><option>http</option></term>
-     <listitem><para><literal>http::Proxy</literal> sets the default proxy to use for HTTP
-     URIs. It is in the standard form of <literal>http://[[user][:pass]@]host[:port]/</literal>.
-     Per host proxies can also be specified by using the form
-     <literal>http::Proxy::&lt;host&gt;</literal> with the special keyword <literal>DIRECT</literal> 
-	 meaning to use no proxies. If no one of the above settings is specified,
-	 <envar>http_proxy</envar> environment variable
-	 will be used.</para>
-
-     <para>Three settings are provided for cache control with HTTP/1.1 compliant
-     proxy caches.
-     <literal>No-Cache</literal> tells the proxy not to use its cached
-     response under any circumstances.
-     <literal>Max-Age</literal> sets the allowed maximum age (in seconds) of
-     an index file in the cache of the proxy.
-     <literal>No-Store</literal> specifies that the proxy should not store
-     the requested archive files in its cache, which can be used to prevent
-     the proxy from polluting its cache with (big) .deb files.</para>
-
-     <para>The option <literal>timeout</literal> sets the timeout timer used by the method;
-     this value applies to the connection as well as the data timeout.</para>
-
-     <para>The setting <literal>Acquire::http::Pipeline-Depth</literal> can be used to
-     enable HTTP pipelining (RFC 2616 section 8.1.2.2) which can be beneficial e.g. on
-     high-latency connections. It specifies how many requests are sent in a pipeline.
-     APT tries to detect and workaround misbehaving webservers and proxies at runtime, but
-     if you know that yours does not conform to the HTTP/1.1 specification pipelining can
-     be disabled by setting the value to 0. It is enabled by default with the value 10.</para>
-
-     <para><literal>Acquire::http::AllowRedirect</literal> controls whether APT will follow
-     redirects, which is enabled by default.</para>
-
-     <para>The used bandwidth can be limited with
-     <literal>Acquire::http::Dl-Limit</literal> which accepts integer
-     values in kilobytes per second. The default value is 0 which
-     deactivates the limit and tries to use all available bandwidth.
-     Note that this option implicitly disables downloading from
-     multiple servers at the same time.</para>
-
-     <para><literal>Acquire::http::User-Agent</literal> can be used to set a different
-     User-Agent for the http download method as some proxies allow access for clients
-     only if the client uses a known identifier.</para>
-
-     <para><literal>Acquire::http::Proxy-Auto-Detect</literal> can be used to
-     specify an external command to discover the http proxy to use. The first
-     and only parameter is an URI denoting the host to be contacted to allow
-     for host-specific configuration. APT expects the command to output the
-     proxy on stdout as a single line in the style <literal>http://proxy:port/</literal>
-     or the word <literal>DIRECT</literal> if no proxy should be used. No output
-     indicates that the generic proxy settings should be used.
-
-     Note that auto-detection will not be used for a host if a host-specific proxy
-     configuration is already set via <literal>Acquire::http::Proxy::<replaceable>HOST</replaceable></literal>.
-
-     See the &squid-deb-proxy-client; package for an example implementation that
-     uses avahi.
-
-     This option takes precedence over the legacy option name
-     <literal>ProxyAutoDetect</literal>.
-     </para>
-
-     </listitem>
+     <varlistentry><term><option>http</option> <option>https</option></term>
+     <listitem><para>The options in these scopes configure APTs acquire transports for the protocols
+     HTTP and HTTPS and are documented in the &apt-transport-http; and &apt-transport-https;
+     manpages respectively.</para></listitem>
      </varlistentry>
 
-     <varlistentry><term><option>https</option></term>
-	 <listitem><para>
-	 The <literal>Cache-control</literal>, <literal>Timeout</literal>,
-	 <literal>AllowRedirect</literal>, <literal>Dl-Limit</literal> and
-	 <literal>proxy</literal> options work for HTTPS URIs in the same way
-	 as for the <literal>http</literal> method, and default to the same
-	 values if they are not explicitly set. The
-	 <literal>Pipeline-Depth</literal> option is not yet supported.
-	 </para>
-
-	 <para><literal>CaInfo</literal> suboption specifies place of file that
-	 holds info about trusted certificates.
-	 <literal>&lt;host&gt;::CaInfo</literal> is the corresponding per-host option.
-	 <literal>Verify-Peer</literal> boolean suboption determines whether or not the
-	 server's host certificate should be verified against trusted certificates.
-	 <literal>&lt;host&gt;::Verify-Peer</literal> is the corresponding per-host option.
-	 <literal>Verify-Host</literal> boolean suboption determines whether or not the
-	 server's hostname should be verified.
-	 <literal>&lt;host&gt;::Verify-Host</literal> is the corresponding per-host option.
-	 <literal>SslCert</literal> determines what certificate to use for client
-	 authentication. <literal>&lt;host&gt;::SslCert</literal> is the corresponding per-host option.
-	 <literal>SslKey</literal> determines what private key to use for client
-	 authentication. <literal>&lt;host&gt;::SslKey</literal> is the corresponding per-host option.
-	 <literal>SslForceVersion</literal> overrides default SSL version to use.
-	 It can contain either of the strings '<literal>TLSv1</literal>' or
-	 '<literal>SSLv3</literal>'.
-	 <literal>&lt;host&gt;::SslForceVersion</literal> is the corresponding per-host option.
-	 </para></listitem></varlistentry>
-
      <varlistentry><term><option>ftp</option></term>
      <listitem><para>
      <literal>ftp::Proxy</literal> sets the default proxy to use for FTP URIs.
diff --git a/doc/examples/CMakeLists.txt b/doc/examples/CMakeLists.txt
index 1998867db..8d9ea068f 100644
--- a/doc/examples/CMakeLists.txt
+++ b/doc/examples/CMakeLists.txt
@@ -1,4 +1,4 @@
-install(FILES apt.conf apt-https-method-example.conf configure-index preferences
+install(FILES apt.conf configure-index preferences
         DESTINATION ${CMAKE_INSTALL_DOCDIR}/examples)
 install(FILES apt-ftparchive.conf ftp-archive.conf
         DESTINATION ${CMAKE_INSTALL_DOCDIR}/../apt-utils/examples)
diff --git a/doc/examples/apt-https-method-example.conf b/doc/examples/apt-https-method-example.conf
deleted file mode 100644
index 3152ef1e2..000000000
--- a/doc/examples/apt-https-method-example.conf
+++ /dev/null
@@ -1,186 +0,0 @@
-/* This file is a sample configuration for apt https method. Configuration
-   parameters found in this example file are expected to be used in main
-   apt.conf file, just like other configuration parameters for different
-   methods (ftp, file, ...).
-
-   This example file starts with a common setup that voluntarily exhibits
-   all available configurations knobs with simple comments. Extended
-   comments on the behavior of the option is provided at the end for
-   better readability. As a matter of fact, a common configuration file
-   will certainly contain far less elements and benefit of default values
-   for many parameters.
-
-   Because some configuration parameters for apt https method in following
-   examples apply to specific (fictional) repositories, the associated
-   sources.list file is provided here:
-
-   ...
-
-   deb     https://secure.dom1.tld/debian unstable main contrib non-free
-   deb-src https://secure.dom1.tld/debian unstable main contrib non-free
-
-   deb     https://secure.dom2.tld/debian unstable main contrib non-free
-   deb-src https://secure.dom2.tld/debian unstable main contrib non-free
-
-   ...
-
-
-   Some notes on the servers:
-
-    - secure.dom1.tld is freely accessible using https (no client
-      authentication is required).
-    - secure.dom1.tld certificate is part of a multi level PKI, and we
-      want to specifically check the issuer of its certificate. We do
-      not have the constraint for secure.dom2.tld
-    - secure.dom2.tld requires client authentication by certificate
-      to access its content.
-    - The certificate presented by both server have (as expected) a CN that
-      matches their respective DNS names.
-    - We have CRL available for both dom1.tld and dom2.tld PKI, and intend
-      to use them.
-    - It sometimes happens that we had other more generic https available
-      repository to our list. We want the checks to be performed against
-      a common list of anchors (like the one provided by ca-certificates
-      package for instance)
-
-   The sample configuration below basically covers those simple needs.
-*/
-
-
-// Verify peer certificate and also matching between certificate name
-// and server name as provided in sources.list (default values)
-Acquire::https::Verify-Peer "true";
-Acquire::https::Verify-Host "true";
-
-// Except otherwise specified, use that list of anchors
-Acquire::https::CaInfo     "/etc/ssl/certs/ca-certificates.pem";
-
-// Use a specific anchor and associated CRL. Enforce issuer of
-// server certificate using its cert.
-Acquire::https::secure.dom1.tld::CaInfo     "/etc/apt/certs/ca-dom1-crt.pem";
-Acquire::https::secure.dom1.tld::CrlFile    "/etc/apt/certs/ca-dom1-crl.pem";
-Acquire::https::secure.dom1.tld::IssuerCert "/etc/apt/certs/secure.dom1-issuer-crt.pem";
-
-// Like previous for anchor and CRL, but also provide our
-// certificate and keys for client authentication.
-Acquire::https::secure.dom2.tld::CaInfo  "/etc/apt/certs/ca-dom2-crt.pem";
-Acquire::https::secure.dom2.tld::CrlFile "/etc/apt/certs/ca-dom2-crl.pem";
-Acquire::https::secure.dom2.tld::SslCert "/etc/apt/certs/my-crt.pem";
-Acquire::https::secure.dom2.tld::SslKey  "/etc/apt/certs/my-key.pem";
-
-// No need to downgrade, TLS will be proposed by default. Uncomment
-// to have SSLv3 proposed.
-// Acquire::https::mirror.ipv6.ssi.corp::SslForceVersion "SSLv3";
-
-// No need for more debug if every is fine (default). Uncomment
-// me to get additional information.
-// Debug::Acquire::https "true";
-
-
-/*
-  Options with extended comments:
-
-  Acquire::https[::repo.domain.tld]::CaInfo  "/path/to/ca/certs.pem";
-
-    A string providing the path of a file containing the list of trusted
-    CA certificates used to verify the server certificate. The pointed
-    file is made of the concatenation of the CA certificates (in
-    PEM format) creating the chain used for the verification of the path
-    from the root (self signed one). If the remote server provides the
-    whole chain during the exchange, the file need only contain the root
-    certificate. Otherwise, the whole chain is required.
-
-    If you need to support multiple authorities, the only way is to
-    concatenate everything.
-
-    If None is provided, the default CA bundle used by GnuTLS (apt https
-    method is linked against libcurl-gnutls) is used. At the time of
-    writing, /etc/ssl/certs/ca-certificates.crt.
-
-    If no specific hostname is provided, the file is used by default
-    for all https targets. If a specific mirror is provided, it is
-    used for the https entries in the sources.list file that use that
-    repository (with the same name).
-
-  Acquire::https[::repo.domain.tld]::CrlFile  "/path/to/all/crl.pem";
-
-    Like previous knob but for passing the list of CRL files (in PEM
-    format) to be used to verify revocation status. Again, if the
-    option is defined with no specific mirror (probably makes little
-    sense), this CRL information is used for all defined https entries
-    in sources.list file. In a mirror specific context, it only applies
-    to that mirror.
-
-  Acquire::https[::repo.domain.tld]::IssuerCert "/path/to/issuer/cert.pem";
-
-    Allows one to constrain the issuer of the server certificate (for all
-    https mirrors or a specific one) to a specific issuer. If the
-    server certificate has not been issued by this certificate,
-    connection fails.
-
-  Acquire::https[::repo.domain.tld]::Verify-Peer "true";
-
-    When authenticating the server, if the certificate verification fails
-    for some reason (expired, revoked, man in the middle, lack of anchor,
-    ...), the connection fails. This is obviously what you want in all
-    cases and what the default value (true) of this option provides.
-
-    If you know EXACTLY what you are doing, setting this option to "false"
-    allow you to skip peer certificate verification and make the exchange
-    succeed. Again, this option is for debugging or testing purpose only.
-    It removes ALL the security provided by the use of SSL.TLS to secure
-    the HTTP exchanges.
-
-  Acquire::https[::repo.domain.tld]::Verify-Host "true";
-
-    The certificate provided by the server during the TLS/SSL exchange
-    provides the identity of the server which should match the DNS name
-    used to access it. By default, as requested by RFC 2818, the name
-    of the mirror is checked against the identity found in the
-    certificate. This default behavior is safe and should not be
-    changed. If you know that the server you are using has a DNS name
-    which does not match the identity in its certificate, you can
-    [report that issue to its administrator or] set the option to
-    "false", which will prevent the comparison to be done.
-
-    The options can be set globally or on a per-mirror basis. If set
-    globally, the DNS name used is the one found in the sources.list
-    file in the https URI.
-
-  Acquire::https[::repo.domain.tld]::SslCert "/path/to/client/cert.pem";
-  Acquire::https[::repo.domain.tld]::SslKey  "/path/to/client/key.pem";
-
-    These two options provides support for client authentication using
-    certificates. They respectively accept the X.509 client certificate
-    in PEM format and the associated client key in PEM format (non
-    encrypted form).
-
-    The options can be set globally (which rarely makes sense) or on a
-    per-mirror basis.
-
-  Acquire::https[::repo.domain.tld]::SslForceVersion "TLSv1";
-
-    This option can be use to select the version which will be proposed
-    to the server. "SSLv3" and "TLSv1" are supported. SSLv2, which is
-    considered insecure anyway is not supported (by gnutls, which is
-    used by libcurl against which apt https method is linked).
-
-    When the option is set to "SSLv3" to have apt propose SSLv3 (and
-    associated sets of ciphersuites) instead of TLSv1 (the default)
-    when performing the exchange. This prevents the server to select
-    TLSv1 and use associated ciphersuites. You should probably not use
-    this option except if you know exactly what you are doing.
-
-    Note that the default setting does not guarantee that the server
-    will not select SSLv3 (for ciphersuites and SSL/TLS version as
-    selection is always done by the server, in the end). It only means
-    that apt will not advertise TLS support.
-
-  Debug::Acquire::https "true";
-
-    This option can be used to show debug information. Because it is
-    quite verbose, it is mainly useful to debug problems in case of
-    failure to connect to a server for some reason. The default value
-    is "false".
-
-*/
diff --git a/doc/examples/configure-index b/doc/examples/configure-index
index 153637ebc..9088bd844 100644
--- a/doc/examples/configure-index
+++ b/doc/examples/configure-index
@@ -274,9 +274,6 @@ Acquire
   // - cache-control values
   // - Dl-Limit, Timeout, ... values
   // if not set explicit for https
-  //
-  // see /usr/share/doc/apt/examples/apt-https-method-example.conf.gz
-  // for more examples
   https
   {
 	Verify-Peer "false";
diff --git a/doc/method.dbk b/doc/method.dbk
index e5e035a2b..410d6898c 100644
--- a/doc/method.dbk
+++ b/doc/method.dbk
@@ -267,6 +267,11 @@ an informational string provided for visual debugging.
 </listitem>
 <listitem>
 <para>
+351 Aux Request - Method requests an auxiliary file
+</para>
+</listitem>
+<listitem>
+<para>
 400 URI Failure - URI has failed to acquire
 </para>
 </listitem>
@@ -309,9 +314,9 @@ Authorization is User/Pass
 </itemizedlist>
 <para>
 Only the 6xx series of status codes is sent TO the method. Furthermore the
-method may not emit status codes in the 6xx range. The Codes 402 and 403
+method may not emit status codes in the 6xx range. The Codes 351, 402 and 403
 require that the method continue reading all other 6xx codes until the proper
-602/603 code is received. This means the method must be capable of handling an
+600/602/603 code is received. This means the method must be capable of handling an
 unlimited number of 600 messages.
 </para>
 <para>
@@ -567,6 +572,20 @@ Size, Last-Modified, Filename, MD5-Hash
 </listitem>
 </varlistentry>
 <varlistentry>
+<term>351 Aux Request</term>
+<listitem>
+<para>
+Indicates a request for an auxiliary file to be downloaded by the acquire system
+(via another method) and made available for the requesting method. The requestor
+will get a <emphasis>600 URI Acquire</emphasis> with the URI it requested and the
+filename will either be an existing file if the request was success or if the
+acquire failed for some reason the file will not exist.
+Fields: URI (of the file causing the need for the auxiliary file), MaximumSize,
+Aux-ShortDesc, Aux-Description, Aux-URI
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
 <term>400 URI Failure</term>
 <listitem>
 <para>
diff --git a/doc/po4a.conf b/doc/po4a.conf
index 955ebbc76..587215abc 100644
--- a/doc/po4a.conf
+++ b/doc/po4a.conf
@@ -27,6 +27,9 @@
 [type: manpage] apt-sortpkgs.1.xml $lang:$lang/apt-sortpkgs.$lang.1.xml add_$lang:xml.add
 [type: manpage] apt-ftparchive.1.xml $lang:$lang/apt-ftparchive.$lang.1.xml add_$lang:xml.add
 [type: manpage] apt_auth.conf.5.xml $lang:$lang/apt_auth.conf.$lang.5.xml add_$lang:xml.add
+[type: manpage] apt-transport-http.1.xml $lang:$lang/apt-transport-http.$lang.1.xml add_$lang:xml.add
+[type: manpage] apt-transport-https.1.xml $lang:$lang/apt-transport-https.$lang.1.xml add_$lang:xml.add
+[type: manpage] apt-transport-mirror.1.xml $lang:$lang/apt-transport-mirror.$lang.1.xml add_$lang:xml.add
 
 [type: docbook] guide.dbk $lang:$lang/guide.$lang.dbk
 #                add_$lang::$lang/addendum/docbook_$lang.add
diff --git a/doc/sources.list.5.xml b/doc/sources.list.5.xml
index 694082bea..5572b8da3 100644
--- a/doc/sources.list.5.xml
+++ b/doc/sources.list.5.xml
@@ -352,17 +352,15 @@ deb-src [ option1=value1 option2=value2 ] uri suite [component1] [component2] [.
 
     <para>The currently recognized URI types are:
    <variablelist>
-    <varlistentry><term><command>http</command></term>
+    <varlistentry><term><command>http</command> (&apt-transport-http;)</term>
     <listitem><para>
     The http scheme specifies an HTTP server for an archive and is the most
-    commonly used method, with many options in the
-    <literal>Acquire::http</literal> scope detailed in &apt-conf;. The URI can
-    directly include login information if the archive requires it, but the use
-    of &apt-authconf; should be preferred. The method also supports SOCKS5 and
-    HTTP(S) proxies either configured via apt-specific configuration or
-    specified by the environment variable <envar>http_proxy</envar> in the
-    format (assuming an HTTP proxy requiring authentication)
-    <replaceable>http://user:pass@server:port/</replaceable>.
+    commonly used method. The URI can directly include login information if the
+    archive requires it, but the use of &apt-authconf; should be preferred.
+    The method also supports SOCKS5 and HTTP(S) proxies either configured via
+    apt-specific configuration or specified by the environment variable
+    <envar>http_proxy</envar> in the format (assuming an HTTP proxy requiring
+    authentication) <replaceable>http://user:pass@server:port/</replaceable>.
     The authentication details for proxies can also be supplied via
     &apt-authconf;.</para>
     <para>Note that these forms of authentication are insecure as the whole
@@ -373,7 +371,7 @@ deb-src [ option1=value1 option2=value2 ] uri suite [component1] [component2] [.
     chosen transport method. See &apt-secure; for details.</para></listitem>
     </varlistentry>
 
-    <varlistentry><term><command>https</command></term>
+    <varlistentry><term><command>https</command> (&apt-transport-https;)</term>
     <listitem><para>
     The https scheme specifies an HTTPS server for an archive and is very
     similar in use and available options to the http scheme. The main