summaryrefslogtreecommitdiff
path: root/methods
AgeCommit message (Collapse)Author
2017-06-28methods: Add HTTPS support to http method, using GnuTLSJulian Andres Klode
The http method will eventually replace the curl-based https method, but for now, this is an opt-in experiment that can be enabled by setting Dir::Bin::Methods::https to "http". Known issues: - We do not support HTTPS proxies yet - We do not support proxying HTTPS connections yet (CONNECT) - IssuerCert and SslForceVersion are unsupported Gbp-Dch: Full
2017-06-28methods: connect: Switch from int fds to new MethodFdJulian Andres Klode
Use std::unique_ptr<MethodFd> everywhere we used an integer-based file descriptor before. This allows us to implement stuff like TLS support easily. Gbp-Dch: ignore
2017-06-28methods: connect: Change PkgAcqMethod to aptMethodJulian Andres Klode
This will allow us to access ConfigFind() and stuff which makes it possible for us to implement TLS support. Gbp-Dch: ignore
2017-06-26deal with 3xx httpcodes as required by HTTP/1.1 specDavid Kalnischkies
An unknown code should be handled the same as the x00 code of this group, but for redirections we used to treat 300 (and a few others) as an error while unknown codes were considered redirections. Instead we check now explicitly for the redirection codes we support for redirecting (and add the 308 defined in RFC 7538) to avoid future problems if new 3xx codes are added expecting certain behaviours. Potentially strange would have been e.g. "305 Use Proxy" sending a Location for the proxy to use – which wouldn't have worked and resulted in an error anyhow, but probably confused users in the process.
2017-06-26avoid changing directory in mirror methodDavid Kalnischkies
2017-06-26Annotate intended switch fall through in httpsDavid Kalnischkies
Reported-By: gcc-7 Gbp-Dch: Ignore
2017-01-24basehttp: Only read Content-Range on 416 and 206 responsesJulian Andres Klode
This fixes issues with sourceforge where the redirector includes such a Content-Range in a 302 redirect. Since we do not really know what file is meant in a redirect, let's just ignore it for all responses other than 416 and 206. Maybe we should also get rid of the other errors, and just ignore the field in those cases as well? LP: #1657567
2017-01-19fix various typos reported by spellintianDavid Kalnischkies
Most of them in (old) code comments. The two instances of user visible string changes the po files of the manpages are fixed up as well. Gbp-Dch: Ignore Reported-By: spellintian
2017-01-19stop rred from leaking debug messages on recovered errorsDavid Kalnischkies
rred can fail for a plentory of reasons, but its failure is usually recoverable (Ign lines) so it shouldn't leak unrequested debug messages to an observing user. Closes: #850759
2017-01-17https: Quote path in URL before passing it to curlJulian Andres Klode
Curl requires URLs to be urlencoded. We are however giving it undecoded URLs. This causes it go completely nuts if there is a space in the URI, producing requests like: GET /a file HTTP/1.1 which the servers then interpret as a GET request for "/a" with HTTP version "file" or some other non-sense. This works around the issue by encoding the path component of the URL. I'm not sure if we should encode other parts of the URL as well, this one seems to do the trick for the actual issue at hand. A more correct fix is to avoid the dequoting and (re-)quoting of URLs when a redirect occurs / a new request is sent. That's been on the radar for probably a year or two now, but nobody bothered implementing that yet. LP: #1651923
2016-12-31rename ServerMethod to BaseHttpMethodDavid Kalnischkies
This 'method' is the abstract base for http and https and should as such be called out like this rather using an easily confused name. Gbp-Dch: Ignore
2016-12-31separating state variables regarding server/requestDavid Kalnischkies
Having a Reset(bool) method to partially reset certain variables like the download size always were strange, so this commit splits the ServerState into an additional RequestState living on the stack for as long as we deal with this request causing an automatic "reset". There is much to do still to make this code look better, but this is a good first step which compiles cleanly and passes all tests, so keeping it as history might be beneficial and due to avoiding explicit memory allocations it ends up fixing a small memory leak in https, too. Closes: #440057
2016-12-08Honour Acquire::ForceIPv4/6 in the https transportLukasz Kawczynski
2016-11-25gpgv: Untrust SHA1, RIPE-MD/160, but allow downgrading to weakJulian Andres Klode
Change the trust level check to allow downgrading an Untrusted option to weak (APT::Hashes::SHA1::Weak "yes";), so it prints a warning instead of an error; and change the default values for SHA1 and RIPE-MD/160 from Weak to Untrusted.
2016-11-24report apt-key errors via status-fd messagesDavid Kalnischkies
We report warnings from apt-key this way already since 29c590951f812d9e9c4f17706e34f2c3315fb1f6, so reporting errors seems like a good addition. Most of those errors aren't really from apt-key through, but from the code setting up and actually calling it which used to just print to stderr which might or might not intermix them with (other) progress lines in update calls. Having them as proper error messages in the system means that the errors are actually collected later on for the list instead of ending up with our relatively generic but in those cases bogus hint regarding "is gpgv installed?". The effective difference is minimal as the errors apply mostly to systems which have far worse problems than a not as nice looking error message, which makes this pretty hard to test – but at least now the hint that your system is broken can be read in proper order (= there aren't many valid cases in which the permissions of /tmp are messed up…). LP: #1522988
2016-11-11http: skip connection cleanup if we close it anyhowDavid Kalnischkies
Suggested in #529794
2016-11-11http: clear content before reporting the failureEdgar Fuß
[Comment from commiter:] I have the feeling that the issue itself is fixed for a while already as nowadays we have testcases involving a webserver closing the connection on error (look for "closeOnError") and no even remotely recent reports about it, but moving the content clearance above the failure report is a valid change and shouldn't hurt. Closes: #465572
2016-11-10improve SOCKS error messages for http slightlyDavid Kalnischkies
The 0.0.0.0:0 tor reports is pretty useless by itself, but even if an IP would be reported it is better to show the user the hostname we wanted the proxy to connect to in the same error message. We improve upon it further by looking for this bind address in particular and remap error messages slightly to give users a better chance of figuring out what went wrong. Upstream Tor can't do that as it is technically wrong.
2016-09-04abort connection on '.' target replies in SRVDavid Kalnischkies
Commit 3af3ac2f5ec007badeded46a94be2bd06b9917a2 (released in 1.3~pre1) implements proper fallback for SRV, but that works actually too good as the RFC defines that such an SRV record should indicate that the server doesn't provide this service and apt should respect this. The solution is hence to fail again as requested even if that isn't what the user (and perhaps even the server admins) wanted. At least we will print a message now explicitly mentioning SRV to point people in the right direction. Reported-In: https://bugs.kali.org/view.php?id=3525 Reported-By: Raphaël Hertzog
2016-09-01support long keyid and fingerprint in gpgv's GOODSIGDavid Kalnischkies
In gpgv1 GOODSIG (and the other messages of status-fd) are documented as sending the long keyid. In gpgv2 it is documented to be either long keyid or the fingerprint. At the moment it is still the long keyid, but the documentation hints at the possibility of changing this. We care about this for Signed-By support as we detect this way if the right fingerprint has signed this file (or not). The check itself is done via VALIDSIG which always is a fingerprint, but there must also be a GOODSIG (as expired sigs are valid, too) found to be accepted which wouldn't be found in the fingerprint-case and the signature hence refused.
2016-09-01try not to call memcpy with length 0 in hash calculationsDavid Kalnischkies
memcpy is marked as nonnull for its input, but ignores the input anyhow if the declared length is zero. Our SHA2 implementations do this as well, it was "just" MD5 and SHA1 missing, so we add the length check here as well as along the callstack as it is really pointless to do all these method calls for "nothing". Reported-By: gcc -fsanitize=undefined
2016-08-27Merge branch 'portability/freebsd'Julian Andres Klode
2016-08-26methods/connect.cc: Only use AI_IDN if definedJulian Andres Klode
Gbp-Dch: ignore
2016-08-26CMake: Do not use -lresolv if res_init exists in libcJulian Andres Klode
Gbp-Dch: ignore
2016-08-25show apt-key warnings in apt updateDavid Kalnischkies
In 105503b4b470c124bc0c271bd8a50e25ecbe9133 we got a warning implemented for unreadable files which greatly improves the behavior of apt update already as everything will work as long as we don't need the keys included in these files. The behavior if they are needed is still strange through as update will fail claiming missing keys and a manual test (which the user will likely perform as root) will be successful. Passing the new warning generated by apt-key through to apt is a bit strange from an interface point of view, but basically duplicating the warning code in multiple places doesn't feel right either. That means we have no translation for the message through as apt-key has no i18n yet. It also means that if the user has a bunch of sources each of them will generate a warning for each unreadable file which could result in quite a few duplicated warnings, but "too many" is better than none. Closes: 834973
2016-08-17methods: read config in most to least specific orderDavid Kalnischkies
The implementation of the generic config fallback did the fallback in the wrong order so that the least specific option wasn't the last value picked but in fact the first one… doh! So in the bugreports case http -> https -> http::<hostname> -> https::<hostname> while it should have been the reverse as before. Regression-In: 30060442025824c491f58887ca7369f3c572fa57 Closes: 834642
2016-08-16don't try pipelining if server closes connectionsDavid Kalnischkies
If a server closes a connection after sending us a file that tends to mean that its a type of server who always closes the connection – it is therefore relatively pointless to try pipelining with it even if it isn't a problem by itself: apt is just restarting the pipeline each time after it got served one file and the connection is closed. The problem starts if one or more proxies are between the server and apt and they disagree about how the connection should be as in the bugreporters case where the responses apt gets contain both Keep-Alive and Proxy-Connection headers (which apt both ignores) indicating a proxy is trying to keep a connection open while the response also contains "Connection: close" indicating the opposite which apt understands and respects as it is required to do. We avoid stepping into this abyss by not performing pipelining anymore if we got a respond with the indication to close connection if the response was otherwise a success – error messages are sent by some servers via this method as their pages tend to be created dynamically and hence their size isn't known a priori to them. Closes: #832113
2016-08-16don't sent Range requests if we know its not acceptedDavid Kalnischkies
If the server told us in a previous request that it isn't supporting Ranges with bytes via an Accept-Ranges header missing bytes, we don't try to formulate requests using Ranges.
2016-08-16reorganize server-states resetting in http/httpsDavid Kalnischkies
We keep various information bits about the server around, some only effecting the currently handled file (like sizes) while others should be persistent (like pipeline detections). http used to reset all file-related manually, which is a bit silly if we already have a Reset() method – which does reset all through –, so extending it with a parameter for reuse and calling it from https too (as this was previously resetting by just creating a new state struct – it uses no value of the persistent state-keeping yet as it supports no pipelining). Gbp-Dch: Ignore
2016-08-13http(s): allow empty values for header fieldsDavid Kalnischkies
It seems completely pointless from a server-POV to sent empty header fields, so most of them don't do it (simply proven by this limitation existing since day one) – but it is technically allowed by the RFC as the surounding whitespaces are optional and Github seems to like sending "X-Geo-Block-List:\r\n" since recently (bug reports in other http clients indicate July) at least sometimes as the reporter claims to have seen it on https only even through it can happen with both. Closes: 834048
2016-08-11http: auto-configure for local Tor proxy if called as 'tor'David Kalnischkies
With apts http transport supporting socks5h proxies and all the work in terms of configuration of methods based on the name it is called with it becomes surprisingly easy to implement Tor support equally (and perhaps even a bit exceeding) what is available currently in apt-transport-tor. How this will turn out to be handled packaging wise we will see in https://lists.debian.org/deity/2016/08/msg00012.html , but until this is resolved we can add the needed support without actively enabling it for now, so that this can be tested better.
2016-08-11block direct connections to .onion domains (RFC7687)David Kalnischkies
Doing a direct connect to an .onion address (if you don't happen to use it as a local domain, which you shouldn't) is bound to fail and does leak the information that you do use Tor and which hidden service you wanted to connect to to a DNS server. Worse, if the DNS is poisoned and actually resolves tricking a user into believing the setup would work correctly… This does block also the usage of wrappers like torsocks with apt, but with native support available and advertised in the error message this shouldn't really be an issue. Inspired-by: https://bugzilla.mozilla.org/show_bug.cgi?id=1228457
2016-08-10implement socks5h proxy support for http methodDavid Kalnischkies
Socks support is a requested feature in sofar that the internet is actually believing Acquire::socks::Proxy would exist. It doesn't and this commit isn't adding it as that isn't how our configuration works, but it allows Acquire::http::Proxy="socks5h://…". The HTTPS method was changed already to support socks proxies (all versions) via curl. This commit implements only SOCKS5 (RFC1928) with no auth or pass&user auth (RFC1929), but not GSSAPI which is required by the RFC. The 'h' in the protocol name further indicates that DNS resolution is delegated to the socks proxy rather than performed locally. The implementation works and was tested with Tor as socks proxy for which implementing socks5h only can actually be considered a feature. Closes: 744934
2016-08-10implement generic config fallback for methodsDavid Kalnischkies
The https method implemented for a long while now a hardcoded fallback to the same options in http, which, while it works, is rather inflexible if we want to allow the methods to use another name to change their behavior slightly, like apt-transport-tor does to https – most of the diff being s#https#tor#g which then fails to do the full circle fallthrough tor -> https -> http for https sources. With this config infrastructure this could be implemented now.
2016-08-10use the same redirection handling for http and httpsDavid Kalnischkies
cURL which backs our https implementation can handle redirects on its own, but by dealing with them on our own we gain finer control over which redirections will be performed (we don't like https → http) and by whom so that redirections to other hosts correctly spawn a new https method dealing with these instead of letting the current one deal with it.
2016-08-10detect redirection loops in acquire instead of workersDavid Kalnischkies
Having the detection handled in specific (http) workers means that a redirection loop over different hostnames isn't detected. Its also not a good idea have this implement in each method independently even if it would work
2016-08-10fail on unsupported http/https proxy settingsDavid Kalnischkies
Closes: #623443
2016-08-10support all socks-proxy known to curl in https methodDavid Kalnischkies
2016-08-10Get rid of the old buildsystemJulian Andres Klode
Bye, bye, old friend.
2016-08-06CMake: Add basic CMake build systemJulian Andres Klode
Introduce an initial CMake buildsystem. This build system can build a fully working apt system without translation or documentation. The FindBerkelyDB module is from kdelibs, with some small adjustements to also look in db5 directories. Initial work on this CMake build system started in 2009, and was resumed in August 2016.
2016-07-30prevent C++ locale number formatting in text APIs (try 2)David Kalnischkies
Followup of b58e2c7c56b1416a343e81f9f80cb1f02c128e25. Still a regression of sorts of 8b79c94af7f7cf2e5e5342294bc6e5a908cacabf. Closes: 832044
2016-07-27rred: truncate result file before writing to itDavid Kalnischkies
If another file in the transaction fails and hence dooms the transaction we can end in a situation in which a -patched file (= rred writes the result of the patching to it) remains in the partial/ directory. The next apt call will perform the rred patching again and write its result again to the -patched file, but instead of starting with an empty file as intended it will override the content previously in the file which has the same result if the new content happens to be longer than the old content, but if it isn't parts of the old content remain in the file which will pass verification as the new content written to it matches the hashes and if the entire transaction passes the file will be moved the lists/ directory where it might or might not trigger errors depending on if the old content which remained forms a valid file together with the new content. This has no real security implications as no untrusted data is involved: The old content consists of a base file which passed verification and a bunch of patches which all passed multiple verifications as well, so the old content isn't controllable by an attacker and the new one isn't either (as the new content alone passes verification). So the best an attacker can do is letting the user run into the same issue as in the report. Closes: #831762
2016-07-27http: skip requesting if pipeline is fullDavid Kalnischkies
The rewrite in 742f67eaede80d2f9b3631d8697ebd63b8f95427 is based on the assumption that the pipeline will always be at least one item short each time it is called, but the logs in #832113 suggest that this isn't always the case. I fail to see how at the moment, but the old implementation had this behavior, so restoring it can't really hurt, can it?
2016-07-27use proper warning for automatic pipeline disableDavid Kalnischkies
Also fixes message itself to mention the correct option name as noticed in #832113.
2016-07-26verify hash of input file in rredDavid Kalnischkies
We read the entire input file we want to patch anyhow, so we can also calculate the hash for that file and compare it with what he had expected it to be. Note that this isn't really a security improvement as a) the file we patch is trusted & b) if the input is incorrect, the result will hardly be matching, so this is just for failing slightly earlier with a more relevant error message (althrough, in terms of rred its ignored and complete download attempt instead).
2016-07-06keep trying with next if connection to a SRV host failedDavid Kalnischkies
Instead of only trying the first host we get via SRV, we try them all as we are supposed to and if that isn't working we try to connect to the host itself as if we hadn't seen any SRV records. This was already the intend of the old code, but it failed to hide earlier problems for the next call, which would unconditionally fail then resulting in an all around failure to connect. With proper stacking we can also keep the error messages of each call around (and in the order tried) so if the entire connection fails we can report all the things we have tried while we discard the entire stack if something works out in the end.
2016-07-06report all instead of first error up the acquire chainDavid Kalnischkies
If we don't give a specific error to report up it is likely that all error currently in the error stack are equally important, so reporting just one could turn out to be confusing e.g. if name resolution failed in a SRV record list.
2016-07-06don't change owner/perms/times through file:// symlinksDavid Kalnischkies
If we have files in partial/ from a previous invocation or similar such those could be symlinks created by file:// sources. The code is expecting only real files through and happily changes owner, modification times and permission on the file the symlink points to which tend to be files we have no business in touching in this way. Permissions of symlinks shouldn't be changed, changing owner is usually pointless to, but just to be sure we pick the easy way out and use lchown, check for symlinks before chmod/utimes. Reported-By: Mattia Rizzolo on IRC
2016-07-05avoid 416 response teardown binding to null pointerDavid Kalnischkies
methods/http.cc:640:13: runtime error: reference binding to null pointer of type 'struct FileFd' This reference is never used in the cases it has a nullptr, so the practical difference is non-existent, but its a bug still. Reported-By: gcc -fsanitize=undefined
2016-07-02use +0000 instead of UTC by default as timezone in outputDavid Kalnischkies
All apt versions support numeric as well as 3-character timezones just fine and its actually hard to write code which doesn't "accidently" accepts it. So why change? Documenting the Date/Valid-Until fields in the Release file is easy to do in terms of referencing the datetime format used e.g. in the Debian changelogs (policy §4.4). This format specifies only the numeric timezones through, not the nowadays obsolete 3-character ones, so in the interest of least surprise we should use the same format even through it carries a small risk of regression in other clients (which encounter repositories created with apt-ftparchive). In case it is really regressing in practice, the hidden option -o APT::FTPArchive::Release::NumericTimezone=0 can be used to go back to good old UTC as timezone. The EDSP and EIPP protocols use this 'new' format, the text interface used to communicate with the acquire methods does not for compatibility reasons even if none of our methods would be effected and I doubt any other would (in these instances the timezone is 'GMT' as that is what HTTP/1.1 requires). Note that this is only true for apt talking to methods, (libapt-based) methods talking to apt will respond with the 'new' format. It is therefore strongly adviced to support both also in method input.