summaryrefslogtreecommitdiff
path: root/methods/https.cc
AgeCommit message (Collapse)Author
2015-04-19calculate hashes while downloading in httpsDavid Kalnischkies
We do this in HTTP already to give the CPU some exercise while the disk is heavily spinning (or flashing?) to store the data avoiding the need to reread the entire file again later on to calculate the hashes – which happens outside of the eyes of progress reporting, so you might ended up with a bunch of https workers 'stuck' at 100% while they were busy calculating hashes. This is a bummer for everyone using apt as a connection speedtest as the https method works slower now (not really, it just isn't reporting done too early anymore).
2015-04-19calculate only expected hashes in methodsDavid Kalnischkies
Methods get told which hashes are expected by the acquire system, which means we can use this list to restrict what we calculate in the methods as any extra we are calculating is wasted effort as we can't compare it with anything anyway. Adding support for a new hash algorithm is therefore 'free' now and if a algorithm is no longer provided in a repository for a file, we automatically stop calculating it. In practice this results in a speed-up in Debian as we don't have SHA512 here (so far), so we practically stop calculating it.
2015-04-19improve https method queue progress reportingDavid Kalnischkies
The worker expects that the methods tell him when they start or finish downloading a file. Various information pieces are passed along in this report including the (expected) filesize. https was using a "global" struct for reporting which made it 'reuse' incorrect values in some cases like a non-existent InRelease fallbacking to Release{,.gpg} resulting in a size-mismatch warning. Reducing the scope and redesigning the setting of the values we can fix this and related issues. Closes: 777565, 781509 Thanks: Robert Edmonds and Anders Kaseorg for initial patchs
2015-04-19do not unlink https file on general errorDavid Kalnischkies
It might be quite interesting which file (content) made curl freak out and other methods keep the file around as well. Git-Dch: Ignore
2015-03-16derive more of https from http methodDavid Kalnischkies
Bug #778375 uncovered that https wasn't properly integrated in the class family tree of http as it was supposed to be leading to a NULL pointer dereference. Fixing this 'properly' was deemed to much diff for practically no gain that late in the release, so commit 0c2dc43d4fe1d026650b5e2920a021557f9534a6 just fixed the synptom, while this commit here is fixing the cause plus adding a test.
2015-03-16merge debian/sid into debian/experimentalDavid Kalnischkies
2015-01-05Fix missing URIStart() for https downloadsMichael Vogt
Add a explicit ReceivedData to HttpsMethod that indicates when we got data from the connection so that we can send URISTart() to the parent. This is needed because URIStart got moved in f9b4f12d from the progress_callback to write_data() and it only checks for Res.Size. In the old code if progress_callback is called by libcurl (and sets Res.Size) before write_data is called then URIStart() is never send. Making this a explicit ReceivedData variable fixes this issue.
2014-12-22dispose http(s) 416 error page as non-contentDavid Kalnischkies
Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. (The tests and their binary helpers had to be slightly modified to apply, but the patch to fix the issue itself is unchanged.) Closes: 768797
2014-12-09dispose http(s) 416 error page as non-contentDavid Kalnischkies
Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. Closes: 768797
2014-10-13Fix backward compatiblity of the new pkgAcquireMethod::DropPrivsOrDie()Michael Vogt
Do not drop privileges in the methods when using a older version of libapt that does not support the chown magic in partial/ yet. To do this DropPrivileges() now will ignore a empty Apt::Sandbox::User. Cleanup all hardcoded _apt along the way.
2014-10-07Send "Fail-Reason: MaximumSizeExceeded" from the methodMichael Vogt
Communicate the fail reason from the methods to the parent and Rename() failed files.
2014-10-07make expected-size a maximum-size check as this is what we want at this pointMichael Vogt
2014-10-07add ftp expected size checkMichael Vogt
2014-10-07methods/https.cc: use File->Tell() here tooMichael Vogt
2014-10-06make http size check workMichael Vogt
2014-09-24methods: Fail if we cannot drop privilegesJulian Andres Klode
2014-09-24Drop Privileges to "Debian-apt" in most acquire methodsMichael Vogt
Add a new "Debian-apt" user that owns the /var/lib/apt/lists and /var/cache/apt/archive directories. The methods http, https, ftp, gpgv, gzip switch to this user when they start. Thanks to Julian and "ioerror" and tors "switch_id()" code.
2014-09-02Make Proxy-Auto-Detect check for each hostMichael Vogt
When doing Acquire::http{,s}::Proxy-Auto-Detect, run the auto-detect command for each host instead of only once. This should make using "proxy" from libproxy-tools feasible which can then be used for PAC style or other proxy configurations. Closes: #759264
2014-08-26make https honor ExpectedSize as wellMichael Vogt
2014-04-26enforce LFS for partial files in https range requestsDavid Kalnischkies
2014-03-13cleanup headers and especially #includes everywhereDavid Kalnischkies
Beside being a bit cleaner it hopefully also resolves oddball problems I have with high levels of parallel jobs. Git-Dch: Ignore Reported-By: iwyu (include-what-you-use)
2014-03-13warning: unused parameter ‘foo’ [-Wunused-parameter]David Kalnischkies
Reported-By: gcc -Wunused-parameter Git-Dch: Ignore
2014-03-13warning: extra ‘;’ [-Wpedantic]David Kalnischkies
Git-Dch: Ignore Reported-By: gcc -Wpedantic
2014-02-22Fix typos in documentation (codespell)Michael Vogt
2014-02-14add a testcase to check for forbidden https→http downgradesDavid Kalnischkies
Git-Dch: Ignore
2014-02-14disable https->http redirects in libcurlMichael Vogt
This change prevents changing the protocol from https to http.
2014-02-14report https download start only if we really get itDavid Kalnischkies
Reporting it via progress means that e.g. a redirect will trigger it, too, so you get a Get & Hit while http only reports a Hit as it should be.
2014-02-11use utimes instead of utimensat/futimensDavid Kalnischkies
cppcheck complains about the obsolete utime as it was removed in POSIX1.2008 and recommends usage of utimensat/futimens instead as those are in POSIX and so commit 9ce3cfc9 switched to them. It is just that they aren't as portable as the standard suggests: At least our kFreeBSD and Hurd ports stumble over it at runtime. So to make both, the ports and cppcheck happy, we use utimes instead. Closes: 738567
2014-01-16correct some style/performance/warnings from cppcheckDavid Kalnischkies
The most "visible" change is from utime to utimensat/futimens as the first one isn't part of POSIX anymore. Reported-By: cppcheck Git-Dch: Ignore
2013-10-01handle complete responses to https range requestsDavid Kalnischkies
Servers might respond with a complete file either because they don't support Ranges at all or the If-Range condition isn't statisfied, so we have to parse the headers curl gets ourself to seek or truncate the file we have so far. This also finially adds the testcase testing a bunch of partial situations for both, http and https - which is now all green. Closes: 617643, 667699 LP: 1157943
2013-10-01fix partial (206 and 416) support in httpsDavid Kalnischkies
As lengthy discussed in lp:1157943 partial https support was utterly broken as a 206 response was handled as an (unhandled) error. This is the first part of fixing it by supporting a 206 response and starting to deal with 416.
2013-05-08merged patch from Daniel Hartwig to fix URI and proxy releated issuesMichael Vogt
2013-05-08merged from the debian-wheezy branchMichael Vogt
2013-05-08properly handle if-modfied-since with libcurl/https Michael Vogt
(closes: #705648)
2013-01-09* methods/https.cc:Michael Vogt
- reuse connection in https, thanks to Thomas Bushnell, BSG for the patch. LP: #1087543, Closes: #695359
2012-05-17add spaces around PACKAGE_VERSION to fix FTBFS with -std=c++11David Kalnischkies
2012-03-22the previously used VERSION didn't work everywhere so we are switchingDavid Kalnischkies
to the more standard PACKAGE_VERSION and make it work in every file
2012-01-30* methods/http{s,}.cc:David Kalnischkies
- if a file without an extension is requested send an 'Accept: text/*' header to avoid that the server chooses unsupported compressed files in a content-negotation attempt (Closes: #657560)
2012-01-19* methods/https.cc:David Kalnischkies
- use curls list append instead of appending Range and If-Range by hand which generates malformed requests, thanks Mel Collins for the hint! (Closes: #646381)
2011-12-17try to avoid direct usage of .Fd() if possible and do read()s and coDavid Kalnischkies
on the FileFd instead
2011-10-05merge with current debian apt/experimentalDavid Kalnischkies
2011-10-05* apt-pkg/contrib/configuration.cc:Michael Vogt
- fix double delete (LP: #848907) - ignore only the invalid regexp instead of all options * apt-pkg/acquire-item.h, apt-pkg/deb/debmetaindex.cc: - fix fetching language information by adding OptionalSubIndexTarget * methods/https.cc: - cleanup broken downloads properly * ftparchive/cachedb.cc: - fix buffersize in bytes2hex * apt-pkg/deb/deblistparser.cc: - fix crash when the dynamic mmap needs to be grown in LoadReleaseInfo (LP: #854090)
2011-09-20* methods/https.cc:Michael Vogt
- cleanup broken downloads properly
2011-09-19use forward declaration in headers if possible instead of includesDavid Kalnischkies
2011-09-13Support large files in the complete toolset. Indexes of thisDavid Kalnischkies
size are pretty unlikely for now, but we need it for deb packages which could become bigger than 4GB now (LP: #815895)
2011-09-13reorder includes: add <config.h> if needed and include it at firstDavid Kalnischkies
2011-01-12* methods/https.cc:David Kalnischkies
- fix CURLOPT_SSL_VERIFYHOST by really passing 2 to it if enabled
2010-01-08* French manpage translation updateMichael Vogt
* spot & fix various typos in all manpages * German manpage translation update * cmdline/apt-cache.cc: - remove translatable marker from the "%4i %s\n" string * buildlib/po4a_manpage.mak: - instruct debiandoc to build files with utf-8 encoding * buildlib/tools.m4: - fix some warning from the buildtools * apt-pkg/acquire-item.cc: - add configuration PDiffs::Limit-options to not download too many or too big patches (Closes: #554349) * debian/control: - let all packages depend on ${misc:Depends} * share/*-archive.gpg: - remove the horrible outdated files. We already depend on the keyring so we don't need to ship our own version * cmdline/apt-key: - errors out if wget is not installed (Closes: #545754) - add --keyring option as we have now possibly many * methods/gpgv.cc: - pass all keyrings (TrustedParts) to gpgv instead of using only one trusted.gpg keyring (Closes: #304846) * methods/https.cc: - finally merge the rest of the patchset from Arnaud Ebalard with the CRL and Issuers options, thanks! (Closes: #485963)
2009-12-14merged patches from david (many thanks)Michael Vogt
2009-12-11fix the backport of the https methods as they would require an ABI breakDavid Kalnischkies
otherwise in the Configuration class.