summaryrefslogtreecommitdiff
path: root/methods
AgeCommit message (Collapse)Author
2018-01-03non-local mirrorlists shouldn't redirect to localDavid Kalnischkies
A mirror list we get from an non-local source like http shouldn't be able to include e.g. file sources and even with other online sources we need to be careful: They also shouldn't include prefixed methods like 'tor+http'. So apply magic based on how the method is called: mirror+file will be allowed to redirect to any source while tor+mirror+file allows all, but sends them to their tor+ variant.
2018-01-03add tag-based control over mirror choices from the listDavid Kalnischkies
The old implementation used to construct a query string including the release(s) the mirrorlist should be for, but that is hard to deal with as this rules out that partial mirrors are included in the list and it turns out that nobody ended up implementing it on the server side. Controlling this on the client side allows partial mirrors to be included and as a bonus prevents that we tell the mirrorlist server (this rather generic) user information.
2018-01-03require methods to request AuxRequest capability at startupDavid Kalnischkies
Allowing a method to request work from other methods is a powerful capability which could be misused or exploited, so to slightly limited the surface let method opt-in into this capability on startup.
2018-01-03reimplement and simplify mirror:// methodDavid Kalnischkies
Embedding an entire acquire stack and HTTP logic in the mirror method made it rather heavy weight and fragile. This reimplement goes the other way by doing only the bare minimum in the method itself and instead redirect the actual download of files to their proper methods. The reimplementation drops the (in the real world) unused query-string feature as it isn't really implementable in the new architecture.
2018-01-03Correctly report transient errors againJulian Andres Klode
Commit 47c0bdc310c8cd62374ca6e6bb456dd183bdfc07 ("report transient errors as transient error") accidentally changed some connection failures to become non-transient, because the result of the error checks where being ignored and then fatal error was returned if an error was pending - even if that error was trivial. After the merge of pu/happy-eyeballs2a this becomes a lot clearer, and easy to fix. Gbp-Dch: ignore Regression-Of: 47c0bdc310c8cd62374ca6e6bb456dd183bdfc07
2018-01-03Add rapid "happy eyeballs" connection fallback (RFC 8305)Julian Andres Klode
Try establishing connections in alternating address families in rapid intervals of 250 ms, adding more connections to the wait list until one succeeds (RFC 8305, happy eyeballs 2). It is important that WaitAndCheckErrors() waits until it has a successful connection, a time out, or all connections failed - otherwise the timing between tries might be wrong, and the final long wait might exit early because one connection failed without trying the others. Timing wise, this only works correctly on Linux, as select() counts down there. But we rely on that in some other places too, so this is not the time to fix that. Timeouts are only reported in the final long wait - the short inner waits are expected to time out more often, and multiple times, we do not want to report them. Closes: #668948 LP: #1308200 Gbp-Dch: paragraph
2018-01-03connect: Extract Connection::CheckError() methodJulian Andres Klode
Extracting the error checking method allows us to reuse it in different places, so we can move the waiting and checking out of DoConnect() eventually. Gbp-Dch: ignore
2018-01-03connect: Store the IP used when picking a connectionJulian Andres Klode
There's no real point in storing the IP address while resolving it - failure messages include the IP address in any case. Do this when picking the connection for actual use instead.
2018-01-03connect: Extract a Connection structJulian Andres Klode
This struct holds information about a connection attempt, like the addrinfo, the resolved address, the fd for the connection, and so on. Gbp-Dch: ignore
2018-01-03connect: Alternate address families for addressesJulian Andres Klode
As a first step to implementing Happy Eyeballs version 2, we need to order the list of hosts getaddrinfo() gave us so it alternates between preferred and other address families. RFC: https://tools.ietf.org/html/rfc8305 Gbp-Dch: ignore
2017-12-13report transient errors as transient errorsDavid Kalnischkies
The Fail method for acquire methods has a boolean parameter indicating the transient-nature of a reported error. The problem with this is that Fail is called very late at a point where it is no longer easily identifiable if an error is indeed transient or not, so some calls were and some weren't and the acquire system would later mostly ignore the transient flag and guess by using the FailReason instead. Introducing a tri-state enum we can pass the information about fatal or transient errors through the callstack to generate the correct fails.
2017-12-13mark some 500 HTTP codes as transient acquire errorsDavid Kalnischkies
If retries are enabled only transient errors are retried, which are very few errors. At least for some HTTP codes it could be beneficial to retry them through so adding them seems like a good idea if only to be more consistent in what we report.
2017-12-13avoid some useless casts reported by -Wuseless-castDavid Kalnischkies
The casts are useless, but the reports show some where we can actually improve the code by replacing them with better alternatives like converting whatever int type into a string instead of casting to a specific one which might in the future be too small. Reported-By: gcc -Wuseless-cast
2017-11-19Also look at https_proxy for https URLsJulian Andres Klode
We accidentally regressed here in 1.5 when replacing the https method.
2017-11-12Do not attempt seccomp under qemu-user and drop EFAULT workaroundJulian Andres Klode
qemu-user passes prctl()-based seccomp through to the kernel, umodified. That's bad, as it blocks the wrong syscalls. We ignored EFAULT which fixed the problem for targets with different pointer sizes from the host, but was a bad hack. In order to identify qemu we can rely on the fact that qemu-user prints its version and exits with 0 if QEMU_VERSION is set to an unsupported value. If we run a command that should fail in such an environment, and it exits with 0, then we are running in qemu-user. apt-helper is an obvious command to run. The tests ensure it exits with 1, and it only prints usage information. We also could not use /bin/false because apt might just as well be from a foreign arch while /bin/false is not. Closes: #881519
2017-10-27seccomp: Allow clock_nanosleep() and nanosleep() syscallsJulian Andres Klode
We sleep in http.cc, so we should allow the sleeping syscalls.
2017-10-26Drop unused gzip, lzma, bzip2, and xz symlinks of storeJulian Andres Klode
The store method replaced them all, the symlinks where mostly for partial upgrades or whatever, they should not be needed any longer.
2017-10-26seccomp: Allow ipc() for fakeroot, and allow sysinfo() for sortJulian Andres Klode
Sorting apparently calls sysconf() which calls sysinfo() to get free pages or whatever. Closes: #879814, #879826
2017-10-26Print syscall number and arch to stderr when trapped by seccompJulian Andres Klode
This should help debugging crashes. The signal handler is a C++11 lambda, yay! Special care has been taken to only use signal handler -safe functions inside there.
2017-10-25Only warn about seccomp() EINVAL (normal) and EFAULT (qemu) errorsJulian Andres Klode
If seccomp is disabled, we fallback to running without it. Qemu fails in the seccomp() call, returning ENOSYS and libseccomp falls back to prctl() without adjusting the pointer, causing the EFAULT. I hope qemu gets fixed at some point to return EINVAL for seccomp via prctl. Bug-Qemu: https://bugs.launchpad.net/qemu/+bug/1726394
2017-10-25methods: Enable additional syscalls (SYSV IPC) in fakerootJulian Andres Klode
If FAKED_MODE is set, enable SYSV IPC so we don't crash when running in fakeroot. Closes: #879662
2017-10-23CMake: methods: Cleanup link libraries, use OBJECT librariesJulian Andres Klode
Use OBJECT libraries for http and connect stuff, and move the seccomp link expression into a global link_libraries() call. This also fixes a bug where only the http target pulled in the gnutls header arguments despite gnutls being used in connect.cc, and thus by mirror and ftp as well. Adjust translation support to ignore TARGET_OBJECTS sources and add the OBJECT libraries to the translated files.
2017-10-23seccomp: Conditionalize statx() whitelistingJulian Andres Klode
statx was introduced in 4.11, so it fails to build in stretch if we just unconditionally use it.
2017-10-23seccomp: Add missing syscalls for ppc64el, i386, and othersJulian Andres Klode
These are a few overlooked syscalls. Also add readv(), writev(), renameat2(), and statx() in case libc uses them. Gbp-Dch: ignore
2017-10-22Sandbox methods with seccomp-BPF; except cdrom, gpgv, rshJulian Andres Klode
This reduces the number of syscalls to about 140 from about 350 or so, significantly reducing security risks. Also change prepare-release to ignore the architecture lists in the build dependencies when generating the build-depends package for travis. We might want to clean up things a bit more and/or move it somewhere else.
2017-10-22methods/basehttp.cc: Remove proxy autodetect debugging codeJulian Andres Klode
This was a left over from the autodetect move. Gbp-Dch: ignore
2017-10-22methods/mirror: Enable sandboxing and other aptMethod featuresJulian Andres Klode
Sandboxing was turned off because we called pkgAcqMethod's Configuration() instead of aptMethod's.
2017-10-22Run Proxy-Auto-Detect script from main processJulian Andres Klode
This avoids running the Proxy-Auto-Detect script inside the untrusted (well, less trusted for now) sandbox. This will allow us to restrict the http method from fork()ing or exec()ing via seccomp.
2017-09-26proper error reporting for v3 onion servicesDavid Kalnischkies
APT connects just fine to any .onion address given, only if the connect fails somehow it will perform checks on the sanity of which in this case is checking the length as they are well defined and as the strings are arbitrary a user typing them easily mistypes which apt should can be slightly more helpful in figuring out by saying the onion hasn't the required length.
2017-09-24Drop curl method and apt-transport-https packageJulian Andres Klode
This automatically removes any old apt-transport-https, as apt now Breaks it unversioned.
2017-07-26allow the auth.conf to be root:root ownedDavid Kalnischkies
Opening the file before we drop privileges in the methods allows us to avoid chowning in the acquire main process which can apply to the wrong file (imagine Binary scoped settings) and surprises users as their permission setup is overridden. There are no security benefits as the file is open, so an evil method could as before read the contents of the file, but it isn't worse than before and we avoid permission problems in this setup.
2017-07-26lookup login info for proxies in auth.confDavid Kalnischkies
On HTTP Connect we since recently look into the auth.conf file for login information, so we should really look for all proxies into the file as the argument is the same as for sources entries and it is easier to document (especially as the manpage already mentions it as supported).
2017-07-26reimplement and document auth.confDavid Kalnischkies
We have support for an netrc-like auth.conf file since 0.7.25 (closing 518473), but it was never documented in apt that it even exists and netrc seems to have fallen out of usage as a manpage for it no longer exists making the feature even more arcane. On top of that the code was a bit of a mess (as it is written in c-style) and as a result the matching of machine tokens to URIs also a bit strange by checking for less specific matches (= without path) first. We now do a single pass over the stanzas. In practice early adopters of the undocumented implementation will not really notice the differences and the 'new' behaviour is simpler to document and more usual for an apt user. Closes: #811181
2017-07-26fail early in http if server answer is too small as wellDavid Kalnischkies
Failing on too much data is good, but we can do better by checking for exact filesizes as we know with hashsums how large a file should be, so if we get a file which has a size we do not expect we can drop it directly, regardless of if the file is larger or smaller than what we expect which should catch most cases which would end up as hashsum errors later now a lot sooner.
2017-07-26fail earlier if server answers with too much dataDavid Kalnischkies
We tend to operate on rather large static files, which means we usually get Content-Length information from the server. If we combine this information with the filesize we are expecting (factoring in pipelining) we can avoid reading a bunch of data we are ending up rejecting anyhow by just closing the connection saving bandwidth and time both for the server as well as the client.
2017-07-26don't try to parse all fields starting with HTTP as status-lineDavid Kalnischkies
It is highly unlikely to encounter fields which start with HTTP in practice, but we should really be a bit more restrictive here.
2017-07-12Reformat and sort all includes with clang-formatJulian Andres Klode
This makes it easier to see which headers includes what. The changes were done by running git grep -l '#\s*include' \ | grep -E '.(cc|h)$' \ | xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/' To modify all include lines by adding a space, and then running ./git-clang-format.sh.
2017-07-12methods/aptmethod.h: Add missing fileutl.h includeJulian Andres Klode
2017-07-03Stop bragging about old speeds in http.cc commentsJulian Andres Klode
That's just ridiculous these days. Gbp-Dch: ignore
2017-07-03don't set ip addresses as server names for SNIDavid Kalnischkies
It is kinda unlikely that apt will ever encounter a certificate for an IP and a user actually using it, but the API documentation for gnutls_server_name_set explicitly says that "IPv4 or IPv6 addresses are not permitted to be set by this function.", so we should follow it. [jak@d.o: Slightly rebased]
2017-07-03Swap file descriptors before the handshakeJulian Andres Klode
This makes more sense. If the handshake failed midway, we still should run the gnutls bye stuff. The thinking here is to only set the fd after the session setup, as we do not modify it before, so if it fails in session setup, you retain a usable file descriptor. Gbp-Dch: ignore
2017-07-03Do not error out, only warn if ca certificates are not availableJulian Andres Klode
This probably makes more sense if Verify-Peer is set to off.
2017-07-03tls: Add more details to error messages, and detect more errorsJulian Andres Klode
This should make it easier to figure out what was going on.
2017-07-01http: A response with Content-Length: 0 has no contentJulian Andres Klode
APT considered any response with a Content-Length to have a body, even if the value of the header was 0. A 0 length body however, is equal to no body.
2017-07-01Make Verify-Host and Verify-Peer independent againJulian Andres Klode
We can actually just pass null as a hostname, so let's just do that when Verify-Host is set to false.
2017-06-30TLS support: Error out on unsupported curl optionsJulian Andres Klode
Silently ignoring the options might be a security issue, so produce an error instead.
2017-06-30Improve closing the TLS connectionJulian Andres Klode
If gnutls_session_bye() exited with an error, we never closed the underlying file descriptor, causing the method to think the connection was still open. This caused problems especially in test-partial-file-support where we checked that a "complete" file and an incomplete file work. The first GET returns a 416 with Connection: close, and the next GET request then accidentally reads the body of the 416 as the header for its own request.
2017-06-30Switch to 'http' as the default https methodJulian Andres Klode
The old curl based method is still available as 'curl', 'curl+http', and 'curl+https'.
2017-06-30http: Add support for https:// proxiesJulian Andres Klode
HTTPS proxies just require unwrapping the TLS layer at the proxy connection, that's easy, and of course sending proxy-specific headers that are sent on "http" proxies.
2017-06-30http: Add support for CONNECT proxying to HTTPS locationsJulian Andres Klode
Proxying HTTPS traffic requires the proxy providing the CONNECT method. This implements the client side of it, although it is a bit hacky. HTTP connect is a normal HTTP CONNECT request, followed by a normal HTTP response, just that the body of the response is the TCP stream of the target host. We use a special wrapper in case there are data bytes in the header packets - in that case, the bytes are stored in a buffer and the buffer will be drained first, afterwards the connection continues directly with the TCP stream (with one more vcall). Also: Do not send full URI to https destinations when proxying, as we are directly interfacing with the destination data stream.