summaryrefslogtreecommitdiff
path: root/methods
AgeCommit message (Collapse)Author
2015-06-09rework hashsum verification in the acquire systemDavid Kalnischkies
Having every item having its own code to verify the file(s) it handles is an errorprune process and easy to break, especially if items move through various stages (download, uncompress, patching, …). With a giant rework we centralize (most of) the verification to have a better enforcement rate and (hopefully) less chance for bugs, but it breaks the ABI bigtime in exchange – and as we break it anyway, it is broken even harder. It shouldn't effect most frontends as they don't deal with the acquire system at all or implement their own items, but some do and will need to be patched (might be an opportunity to use apt on-board material). The theory is simple: Items implement methods to decide if hashes need to be checked (in this stage) and to return the expected hashes for this item (in this stage). The verification itself is done in worker message passing which has the benefit that a hashsum error is now a proper error for the acquire system rather than a Done() which is later revised to a Failed().
2015-05-22Merge branch 'debian/sid' into debian/experimentalMichael Vogt
Conflicts: apt-pkg/pkgcache.h debian/changelog methods/https.cc methods/server.cc test/integration/test-apt-download-progress
2015-05-22Update methods/https.cc now that ServerState::Size is renamedMichael Vogt
Git-Dch: ignore
2015-05-22Merge remote-tracking branch 'upstream/debian/jessie' into debian/sidMichael Vogt
Conflicts: apt-pkg/deb/dpkgpm.cc
2015-05-22Rename "Size" in ServerState to TotalFileSizeMichael Vogt
The variable "Size" was misleading and caused bug #1445239. To avoid similar issues in the future, rename it to make the meaning more obvious. git-dch: ignore
2015-05-22Fix endless loop in apt-get update that can cause disk fillupMichael Vogt
The apt http code parses Content-Length and Content-Range. For both requests the variable "Size" is used and the semantic for this Size is the total file size. However Content-Length is not the entire file size for partital file requests. For servers that send the Content-Range header first and then the Content-Length header this can lead to globbing of Size so that its less than the real file size. This may lead to a subsequent passing of a negative number into the CircleBuf which leads to a endless loop that writes data. Thanks to Anton Blanchard for the analysis and initial patch. LP: #1445239
2015-05-13detect Releasefile IMS hits even if the server doesn'tDavid Kalnischkies
Not all servers we are talking to support If-Modified-Since and some are not even sending Last-Modified for us, so in an effort to detect such hits we run a hashsum check on the 'old' compared to the 'new' file, we got the hashes for the 'new' already for "free" from the methods anyway and hence just need to calculated the old ones. This allows us to detect hits even with unsupported servers, which in turn means we benefit from all the new hit behavior also here.
2015-05-12detect 416 complete file in partial by expected hashDavid Kalnischkies
If we have the expected hashes we can check with them if the file we have in partial we got a 416 for is the expected file. We detected this with same-size before, but not every server sends a good Content-Range header with a 416 response.
2015-04-19calculate hashes while downloading in httpsDavid Kalnischkies
We do this in HTTP already to give the CPU some exercise while the disk is heavily spinning (or flashing?) to store the data avoiding the need to reread the entire file again later on to calculate the hashes – which happens outside of the eyes of progress reporting, so you might ended up with a bunch of https workers 'stuck' at 100% while they were busy calculating hashes. This is a bummer for everyone using apt as a connection speedtest as the https method works slower now (not really, it just isn't reporting done too early anymore).
2015-04-19calculate only expected hashes in methodsDavid Kalnischkies
Methods get told which hashes are expected by the acquire system, which means we can use this list to restrict what we calculate in the methods as any extra we are calculating is wasted effort as we can't compare it with anything anyway. Adding support for a new hash algorithm is therefore 'free' now and if a algorithm is no longer provided in a repository for a file, we automatically stop calculating it. In practice this results in a speed-up in Debian as we don't have SHA512 here (so far), so we practically stop calculating it.
2015-04-19handle servers closing encoded connections correctlyDavid Kalnischkies
Servers who advertise that they close the connection get the 'Closes' encoding flag, but this conflicts with servers who response with a transfer-encoding (e.g. encoding) as it is saved in the same flag. We have a better flag for the keep-alive (or not) of the connection anyway, so we check this instead of the encoding. This is in practice not much of a problem as real servers we talk to are HTTP1.1 servers (with keep-alive) and there isn't much point in doing chunked encoding if you are going to close anyway, but our simple testserver stumbles over this if pressed and its a bit cleaner, too. Git-Dch: Ignore
2015-04-19send Alt-* info for uncompressed based on any compressionsDavid Kalnischkies
file sends information about the uncompressed file if it can find it as well as for the compressed file. This was done only for gzip so far, but we support more compression types. That this information isn't used a lot is a different story. Git-Dch: Ignore
2015-04-19remove duplicated check for same file copyDavid Kalnischkies
Git-Dch: Ignore
2015-04-19improve https method queue progress reportingDavid Kalnischkies
The worker expects that the methods tell him when they start or finish downloading a file. Various information pieces are passed along in this report including the (expected) filesize. https was using a "global" struct for reporting which made it 'reuse' incorrect values in some cases like a non-existent InRelease fallbacking to Release{,.gpg} resulting in a size-mismatch warning. Reducing the scope and redesigning the setting of the values we can fix this and related issues. Closes: 777565, 781509 Thanks: Robert Edmonds and Anders Kaseorg for initial patchs
2015-04-19do not unlink https file on general errorDavid Kalnischkies
It might be quite interesting which file (content) made curl freak out and other methods keep the file around as well. Git-Dch: Ignore
2015-04-13Revert "HttpsMethod::Fetch(): Zero the FetchResult object when leaving due ↵Michael Vogt
to 404" This reverts commit 1296bc7c466181a7978c313c40a041b34ce3eaeb.
2015-04-10reimplement the last uses of sprintfDavid Kalnischkies
Working with strings c-style is complicated and error-prune, so by converting to c++ style we gain some simplicity and avoid buffer overflows by later extensions. Git-Dch: Ignore
2015-04-07properly handle expected filesize in httpsDavid Kalnischkies
The worker expects that the methods tell him when they start or finish downloading a file. Various information pieces are passed along in this report including the (expected) filesize. https is using a "global" struct for reporting which made it 'reuse' incorrect values in some cases like a non-existent InRelease fallbacking to Release{,.gpg} resulting in an incorrect size-mismatch warning scaring and desensitizing users as well as being subject to a race between the write_data and progress callbacks generating incorrect progress reporting and potentially the same error message. Other branches as well as the bugreports contain 'better' fixes making the struct local and other sensible changes, but are larger as a result, so in this version we opted for short diff with minimal effect above else instead. Closes: 777565, 781509 Thanks: Robert Edmonds and Anders Kaseorg for initial patchs
2015-04-07HttpsMethod::Fetch(): Zero the FetchResult object when leaving due to 404Robert Edmonds
2015-03-16derive more of https from http methodDavid Kalnischkies
Bug #778375 uncovered that https wasn't properly integrated in the class family tree of http as it was supposed to be leading to a NULL pointer dereference. Fixing this 'properly' was deemed to much diff for practically no gain that late in the release, so commit 0c2dc43d4fe1d026650b5e2920a021557f9534a6 just fixed the synptom, while this commit here is fixing the cause plus adding a test.
2015-03-16merge debian/sid into debian/experimentalDavid Kalnischkies
2015-02-23Fix crash in the apt-transport-https when Owner is NULLTomasz Buchert
Do not crash in ServerState::HeaderLine if there is no Owner. Closes: #778375
2015-01-05Fix missing URIStart() for https downloadsMichael Vogt
Add a explicit ReceivedData to HttpsMethod that indicates when we got data from the connection so that we can send URISTart() to the parent. This is needed because URIStart got moved in f9b4f12d from the progress_callback to write_data() and it only checks for Res.Size. In the old code if progress_callback is called by libcurl (and sets Res.Size) before write_data is called then URIStart() is never send. Making this a explicit ReceivedData variable fixes this issue.
2014-12-22dispose http(s) 416 error page as non-contentDavid Kalnischkies
Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. (The tests and their binary helpers had to be slightly modified to apply, but the patch to fix the issue itself is unchanged.) Closes: 768797
2014-12-09dispose http(s) 416 error page as non-contentDavid Kalnischkies
Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. Closes: 768797
2014-11-09use getline() instead of rolling our ownDavid Kalnischkies
We use it in other places already as well even though it is farly new addition to the POSIX family with 2008, but rolling our own here is really something which should be avoided in such a important method. Git-Dch: Ignore
2014-11-08Assert statement calls a function which may have desired side effects: ↵David Kalnischkies
'pos_is_okay' It does not have any desired sideeffect, so we just mark it as const to properly advertise this fact to developer, compiler and linter alike. Reported-By: cppcheck Git-Dch: Ignore
2014-10-14Merge branch 'debian/sid' into debian/experimentalMichael Vogt
2014-10-13Fix backward compatiblity of the new pkgAcquireMethod::DropPrivsOrDie()Michael Vogt
Do not drop privileges in the methods when using a older version of libapt that does not support the chown magic in partial/ yet. To do this DropPrivileges() now will ignore a empty Apt::Sandbox::User. Cleanup all hardcoded _apt along the way.
2014-10-08methods/rsh.cc: replace strcat with std::stringMichael Vogt
Instead of using strcat use a C++ std::string to avoid overflowing this buffer. Thanks to David Garfield Closes: #76442
2014-10-08Fix ServerMethod::FindMaximumObjectSizeInQueue()Michael Vogt
Git-Dch: ignore
2014-10-08Fix http pipeline messup detectionMichael Vogt
The Maximum-Size protection breaks the http pipeline reorder code because it relies on that the object got fetched entirely so that it can compare the hash of the downloaded data. So instead of stopping when the Maximum-Size of the expected item is reached we only stop when the maximum size of the biggest item in the queue is reached. This way the pipeline reoder code keeps working.
2014-10-07Send "Fail-Reason: MaximumSizeExceeded" from the methodMichael Vogt
Communicate the fail reason from the methods to the parent and Rename() failed files.
2014-10-07Merge branch 'feature/acq-trans' into feature/expected-sizeMichael Vogt
2014-10-07make expected-size a maximum-size check as this is what we want at this pointMichael Vogt
2014-10-07add ftp expected size checkMichael Vogt
2014-10-07methods/https.cc: use File->Tell() here tooMichael Vogt
2014-10-06make http size check workMichael Vogt
2014-09-29Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans Conflicts: apt-pkg/acquire-item.cc apt-pkg/acquire-item.h methods/gpgv.cc
2014-09-29Merge branch 'debian/sid' into debian/experimentalMichael Vogt
Conflicts: apt-pkg/acquire-item.cc
2014-09-27fix: %i in format string (no. 1) requires 'int' but the argument type isDavid Kalnischkies
'unsigned int' Git-Dch: Ignore Reported-By: cppcheck
2014-09-27correct the error messages to refer to apt-key instead of gpgvDavid Kalnischkies
Git-Dch: Ignore
2014-09-27fix: Member variable 'X' is not initialized in the constructor.David Kalnischkies
Reported-By: cppcheck Git-Dch: Ignore
2014-09-26Disable Mth.DropPrivsOrDie() in copy.cc for nowMichael Vogt
Dch-Ignore: true
2014-09-25Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans
2014-09-24releasing package apt version 1.1~exp3Michael Vogt
2014-09-24methods: Fail if we cannot drop privilegesJulian Andres Klode
2014-09-24Drop Privileges to "Debian-apt" in most acquire methodsMichael Vogt
Add a new "Debian-apt" user that owns the /var/lib/apt/lists and /var/cache/apt/archive directories. The methods http, https, ftp, gpgv, gzip switch to this user when they start. Thanks to Julian and "ioerror" and tors "switch_id()" code.
2014-09-23Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans Conflicts: apt-pkg/acquire-item.cc apt-pkg/acquire-item.h methods/copy.cc test/integration/test-hashsum-verification
2014-09-23Merge branch 'debian/sid' into debian/experimentalMichael Vogt
Conflicts: apt-pkg/acquire-item.cc apt-pkg/acquire-item.h apt-pkg/cachefilter.h configure.ac debian/changelog