summaryrefslogtreecommitdiff
path: root/methods
AgeCommit message (Collapse)Author
2017-10-23CMake: methods: Cleanup link libraries, use OBJECT librariesJulian Andres Klode
Use OBJECT libraries for http and connect stuff, and move the seccomp link expression into a global link_libraries() call. This also fixes a bug where only the http target pulled in the gnutls header arguments despite gnutls being used in connect.cc, and thus by mirror and ftp as well. Adjust translation support to ignore TARGET_OBJECTS sources and add the OBJECT libraries to the translated files.
2017-10-23seccomp: Conditionalize statx() whitelistingJulian Andres Klode
statx was introduced in 4.11, so it fails to build in stretch if we just unconditionally use it.
2017-10-23seccomp: Add missing syscalls for ppc64el, i386, and othersJulian Andres Klode
These are a few overlooked syscalls. Also add readv(), writev(), renameat2(), and statx() in case libc uses them. Gbp-Dch: ignore
2017-10-22Sandbox methods with seccomp-BPF; except cdrom, gpgv, rshJulian Andres Klode
This reduces the number of syscalls to about 140 from about 350 or so, significantly reducing security risks. Also change prepare-release to ignore the architecture lists in the build dependencies when generating the build-depends package for travis. We might want to clean up things a bit more and/or move it somewhere else.
2017-10-22methods/basehttp.cc: Remove proxy autodetect debugging codeJulian Andres Klode
This was a left over from the autodetect move. Gbp-Dch: ignore
2017-10-22methods/mirror: Enable sandboxing and other aptMethod featuresJulian Andres Klode
Sandboxing was turned off because we called pkgAcqMethod's Configuration() instead of aptMethod's.
2017-10-22Run Proxy-Auto-Detect script from main processJulian Andres Klode
This avoids running the Proxy-Auto-Detect script inside the untrusted (well, less trusted for now) sandbox. This will allow us to restrict the http method from fork()ing or exec()ing via seccomp.
2017-09-26proper error reporting for v3 onion servicesDavid Kalnischkies
APT connects just fine to any .onion address given, only if the connect fails somehow it will perform checks on the sanity of which in this case is checking the length as they are well defined and as the strings are arbitrary a user typing them easily mistypes which apt should can be slightly more helpful in figuring out by saying the onion hasn't the required length.
2017-09-24Drop curl method and apt-transport-https packageJulian Andres Klode
This automatically removes any old apt-transport-https, as apt now Breaks it unversioned.
2017-07-26allow the auth.conf to be root:root ownedDavid Kalnischkies
Opening the file before we drop privileges in the methods allows us to avoid chowning in the acquire main process which can apply to the wrong file (imagine Binary scoped settings) and surprises users as their permission setup is overridden. There are no security benefits as the file is open, so an evil method could as before read the contents of the file, but it isn't worse than before and we avoid permission problems in this setup.
2017-07-26lookup login info for proxies in auth.confDavid Kalnischkies
On HTTP Connect we since recently look into the auth.conf file for login information, so we should really look for all proxies into the file as the argument is the same as for sources entries and it is easier to document (especially as the manpage already mentions it as supported).
2017-07-26reimplement and document auth.confDavid Kalnischkies
We have support for an netrc-like auth.conf file since 0.7.25 (closing 518473), but it was never documented in apt that it even exists and netrc seems to have fallen out of usage as a manpage for it no longer exists making the feature even more arcane. On top of that the code was a bit of a mess (as it is written in c-style) and as a result the matching of machine tokens to URIs also a bit strange by checking for less specific matches (= without path) first. We now do a single pass over the stanzas. In practice early adopters of the undocumented implementation will not really notice the differences and the 'new' behaviour is simpler to document and more usual for an apt user. Closes: #811181
2017-07-26fail early in http if server answer is too small as wellDavid Kalnischkies
Failing on too much data is good, but we can do better by checking for exact filesizes as we know with hashsums how large a file should be, so if we get a file which has a size we do not expect we can drop it directly, regardless of if the file is larger or smaller than what we expect which should catch most cases which would end up as hashsum errors later now a lot sooner.
2017-07-26fail earlier if server answers with too much dataDavid Kalnischkies
We tend to operate on rather large static files, which means we usually get Content-Length information from the server. If we combine this information with the filesize we are expecting (factoring in pipelining) we can avoid reading a bunch of data we are ending up rejecting anyhow by just closing the connection saving bandwidth and time both for the server as well as the client.
2017-07-26don't try to parse all fields starting with HTTP as status-lineDavid Kalnischkies
It is highly unlikely to encounter fields which start with HTTP in practice, but we should really be a bit more restrictive here.
2017-07-12Reformat and sort all includes with clang-formatJulian Andres Klode
This makes it easier to see which headers includes what. The changes were done by running git grep -l '#\s*include' \ | grep -E '.(cc|h)$' \ | xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/' To modify all include lines by adding a space, and then running ./git-clang-format.sh.
2017-07-12methods/aptmethod.h: Add missing fileutl.h includeJulian Andres Klode
2017-07-03Stop bragging about old speeds in http.cc commentsJulian Andres Klode
That's just ridiculous these days. Gbp-Dch: ignore
2017-07-03don't set ip addresses as server names for SNIDavid Kalnischkies
It is kinda unlikely that apt will ever encounter a certificate for an IP and a user actually using it, but the API documentation for gnutls_server_name_set explicitly says that "IPv4 or IPv6 addresses are not permitted to be set by this function.", so we should follow it. [jak@d.o: Slightly rebased]
2017-07-03Swap file descriptors before the handshakeJulian Andres Klode
This makes more sense. If the handshake failed midway, we still should run the gnutls bye stuff. The thinking here is to only set the fd after the session setup, as we do not modify it before, so if it fails in session setup, you retain a usable file descriptor. Gbp-Dch: ignore
2017-07-03Do not error out, only warn if ca certificates are not availableJulian Andres Klode
This probably makes more sense if Verify-Peer is set to off.
2017-07-03tls: Add more details to error messages, and detect more errorsJulian Andres Klode
This should make it easier to figure out what was going on.
2017-07-01http: A response with Content-Length: 0 has no contentJulian Andres Klode
APT considered any response with a Content-Length to have a body, even if the value of the header was 0. A 0 length body however, is equal to no body.
2017-07-01Make Verify-Host and Verify-Peer independent againJulian Andres Klode
We can actually just pass null as a hostname, so let's just do that when Verify-Host is set to false.
2017-06-30TLS support: Error out on unsupported curl optionsJulian Andres Klode
Silently ignoring the options might be a security issue, so produce an error instead.
2017-06-30Improve closing the TLS connectionJulian Andres Klode
If gnutls_session_bye() exited with an error, we never closed the underlying file descriptor, causing the method to think the connection was still open. This caused problems especially in test-partial-file-support where we checked that a "complete" file and an incomplete file work. The first GET returns a 416 with Connection: close, and the next GET request then accidentally reads the body of the 416 as the header for its own request.
2017-06-30Switch to 'http' as the default https methodJulian Andres Klode
The old curl based method is still available as 'curl', 'curl+http', and 'curl+https'.
2017-06-30http: Add support for https:// proxiesJulian Andres Klode
HTTPS proxies just require unwrapping the TLS layer at the proxy connection, that's easy, and of course sending proxy-specific headers that are sent on "http" proxies.
2017-06-30http: Add support for CONNECT proxying to HTTPS locationsJulian Andres Klode
Proxying HTTPS traffic requires the proxy providing the CONNECT method. This implements the client side of it, although it is a bit hacky. HTTP connect is a normal HTTP CONNECT request, followed by a normal HTTP response, just that the body of the response is the TCP stream of the target host. We use a special wrapper in case there are data bytes in the header packets - in that case, the bytes are stored in a buffer and the buffer will be drained first, afterwards the connection continues directly with the TCP stream (with one more vcall). Also: Do not send full URI to https destinations when proxying, as we are directly interfacing with the destination data stream.
2017-06-30Allow running the TLS stack on any lower connectionJulian Andres Klode
This is especially needed if we use an HTTPS proxy to CONNECT to an HTTPS URI, as we run TLS-inside-TLS then.
2017-06-30Reset failure reason when connection was successfulJulian Andres Klode
When APT was trying multiple addresses, any later error somewhere else would be reported with ConnectionRefused or ConnectionTimedOut as the FailReason because that was set by early connect attempts. This causes APT to handle the failures differently, leading to some weirdly breaking test cases (like the changed one). Add debugging to the previously failing test case so we can find out when something goes wrong there again.
2017-06-30Don't read CaInfo if not specified (missing else)Julian Andres Klode
This fixes a regression from ~alpha2. Closes: #866559 Gbp-Dch: Full
2017-06-29http: Only use system CA store if CaInfo is not setJulian Andres Klode
It turns out that curl only sets the system trust store if the CaInfo option is not set, so let's do the same here.
2017-06-29Improve error message if system CA store is emptyJulian Andres Klode
Tell the user to install ca-certificates. Closes: #866377
2017-06-29use port from SRV record instead of initial portDavid Kalnischkies
An SRV record includes a portnumber to use with the host given, but apt was ignoring the portnumber and instead used either the port given by the user for the initial host or the default port for the service. In practice the service usually runs on another host on the default port, so it tends to work as intended and even if not and apt can't get a connection there it will gracefully fallback to contacting the initial host with the right port, so its a user invisible bug most of the time.
2017-06-28support tor+https being handled by httpDavid Kalnischkies
The apt-transport-tor package operates via simple symlinks which can result in 'http' being called as 'tor+https', so it must pick up the right configuration pieces and trigger https support also in plus names.
2017-06-28Introduce Acquire::AllowTLS to turn off TLS supportJulian Andres Klode
As requested by Henrique de Moraes Holschuh, here comes an option to disable TLS support. If the option is set to false, the internal TLS layer is disabled.
2017-06-28Fix https->http redirect issuesDavid Kalnischkies
Gbp-Dch: ignore
2017-06-28methods: http: Drain pending data before selectingJulian Andres Klode
GnuTLS can already have data pending in its buffers, we need to to drain that first otherwise select() might block indefinitely. Gbp-Dch: ignore
2017-06-28Allow building without curlJulian Andres Klode
This makes testing easier and prepares us for the transition.
2017-06-28methods: Add HTTPS support to http method, using GnuTLSJulian Andres Klode
The http method will eventually replace the curl-based https method, but for now, this is an opt-in experiment that can be enabled by setting Dir::Bin::Methods::https to "http". Known issues: - We do not support HTTPS proxies yet - We do not support proxying HTTPS connections yet (CONNECT) - IssuerCert and SslForceVersion are unsupported Gbp-Dch: Full
2017-06-28methods: connect: Switch from int fds to new MethodFdJulian Andres Klode
Use std::unique_ptr<MethodFd> everywhere we used an integer-based file descriptor before. This allows us to implement stuff like TLS support easily. Gbp-Dch: ignore
2017-06-28methods: connect: Change PkgAcqMethod to aptMethodJulian Andres Klode
This will allow us to access ConfigFind() and stuff which makes it possible for us to implement TLS support. Gbp-Dch: ignore
2017-06-26deal with 3xx httpcodes as required by HTTP/1.1 specDavid Kalnischkies
An unknown code should be handled the same as the x00 code of this group, but for redirections we used to treat 300 (and a few others) as an error while unknown codes were considered redirections. Instead we check now explicitly for the redirection codes we support for redirecting (and add the 308 defined in RFC 7538) to avoid future problems if new 3xx codes are added expecting certain behaviours. Potentially strange would have been e.g. "305 Use Proxy" sending a Location for the proxy to use – which wouldn't have worked and resulted in an error anyhow, but probably confused users in the process.
2017-06-26avoid changing directory in mirror methodDavid Kalnischkies
2017-06-26Annotate intended switch fall through in httpsDavid Kalnischkies
Reported-By: gcc-7 Gbp-Dch: Ignore
2017-01-24basehttp: Only read Content-Range on 416 and 206 responsesJulian Andres Klode
This fixes issues with sourceforge where the redirector includes such a Content-Range in a 302 redirect. Since we do not really know what file is meant in a redirect, let's just ignore it for all responses other than 416 and 206. Maybe we should also get rid of the other errors, and just ignore the field in those cases as well? LP: #1657567
2017-01-19fix various typos reported by spellintianDavid Kalnischkies
Most of them in (old) code comments. The two instances of user visible string changes the po files of the manpages are fixed up as well. Gbp-Dch: Ignore Reported-By: spellintian
2017-01-19stop rred from leaking debug messages on recovered errorsDavid Kalnischkies
rred can fail for a plentory of reasons, but its failure is usually recoverable (Ign lines) so it shouldn't leak unrequested debug messages to an observing user. Closes: #850759
2017-01-17https: Quote path in URL before passing it to curlJulian Andres Klode
Curl requires URLs to be urlencoded. We are however giving it undecoded URLs. This causes it go completely nuts if there is a space in the URI, producing requests like: GET /a file HTTP/1.1 which the servers then interpret as a GET request for "/a" with HTTP version "file" or some other non-sense. This works around the issue by encoding the path component of the URL. I'm not sure if we should encode other parts of the URL as well, this one seems to do the trick for the actual issue at hand. A more correct fix is to avoid the dequoting and (re-)quoting of URLs when a redirect occurs / a new request is sent. That's been on the radar for probably a year or two now, but nobody bothered implementing that yet. LP: #1651923