summaryrefslogtreecommitdiff
path: root/apt-pkg/acquire-item.cc
AgeCommit message (Collapse)Author
2015-06-10store all targets data in IndexTarget structDavid Kalnischkies
We still need an API for the targets, so slowly prepare the IndexTargets to let them take this job. Git-Dch: Ignore
2015-06-09remove debianism file-content verificationDavid Kalnischkies
The code requires every index file we download to have a Package field, but that doesn't hold true for all index we might want to download in the future. Some might not even be deb822 formatted files… The check was needed as apt used to accept unverifiable files like Translation-*, but nowadays it requires hashes for these as well. Even for unsigned repositories we interpret the Release file as binding now, which means this check isn't triggerable expect for repositories which do not have a Release file at all – something which is highly discouraged! Git-Dch: Ignore
2015-06-09do not request files if we expect an IMS hitDavid Kalnischkies
If we have a file on disk and the hashes are the same in the new Release file and the old one we have on disk we know that if we ask the server for the file, we will at best get an IMS hit – at worse the server doesn't support this and sends us the (unchanged) file and we have to run all our checks on it again for nothing. So, we can save ourselves (and the servers) some unneeded requests if we figure this out on our own.
2015-06-09cleanup pdiff support detection decisionDavid Kalnischkies
Its a bit unclean to create an item just to let the item decide that it can't do anything and let it fail, so instead we let the item creator decide in all cases if patching should be attempted. Also pulls a small trick to get the hashes for the current file without calculating them by looking at the 'old' Release file if we have it. Git-Dch: Ignore
2015-06-09support hashes for compressed pdiff filesDavid Kalnischkies
At the moment we only have hashes for the uncompressed pdiff files, but via the new '$HASH-Download' field in the .diff/Index hashes can be provided for the .gz compressed pdiff file, which apt will pick up now and use to verify the download. Now, we "just" need a buy in from the creators of repositories…
2015-06-09check patch hashes in rred worker instead of in the handlerDavid Kalnischkies
rred is responsible for unpacking and reading the patch files in one go, but we currently only have hashes for the uncompressed patch files, so the handler read the entire patch file before dispatching it to the worker which would read it again – both with an implicit uncompress. Worse, while the workers operate in parallel the handler is the central orchestration unit, so having it busy with work means the workers do (potentially) nothing. This means rred is working with 'untrusted' data, which is bad. Yet, having the unpack in the handler meant that the untrusted uncompress was done as root which isn't better either. Now, we have it at least contained in a binary which we can harden a bit better. In the long run, we want hashes for the compressed patch files through to be safe.
2015-06-09rework hashsum verification in the acquire systemDavid Kalnischkies
Having every item having its own code to verify the file(s) it handles is an errorprune process and easy to break, especially if items move through various stages (download, uncompress, patching, …). With a giant rework we centralize (most of) the verification to have a better enforcement rate and (hopefully) less chance for bugs, but it breaks the ABI bigtime in exchange – and as we break it anyway, it is broken even harder. It shouldn't effect most frontends as they don't deal with the acquire system at all or implement their own items, but some do and will need to be patched (might be an opportunity to use apt on-board material). The theory is simple: Items implement methods to decide if hashes need to be checked (in this stage) and to return the expected hashes for this item (in this stage). The verification itself is done in worker message passing which has the benefit that a hashsum error is now a proper error for the acquire system rather than a Done() which is later revised to a Failed().
2015-06-07don't try other compressions on hashsum mismatchDavid Kalnischkies
If we e.g. fail on hash verification for Packages.xz its highly unlikely that it will be any better with Packages.gz, so we just waste download bandwidth and time. It also causes us always to fallback to the uncompressed Packages file for which the error will finally be reported, which in turn confuses users as the file usually doesn't exist on the mirrors, so a bug in apt is suspected for even trying it…
2015-05-18treat older Release files than we already have as an IMSHitDavid Kalnischkies
Valid-Until protects us from long-living downgrade attacks, but not all repositories have it and an attacker could still use older but still valid files to downgrade us. While this makes it sounds like a security improvement now, its a bit theoretical at best as an attacker with capabilities to pull this off could just as well always keep us days (but in the valid period) behind and always knows which state we have, as we tell him with the If-Modified-Since header. This is also why this is 'silently' ignored and treated as an IMSHit rather than screamed at the user as this can at best be an annoyance for attackers. An error here would 'regularily' be encountered by users by out-of-sync mirrors serving a single run (e.g. load balancer) or in two consecutive runs on the other hand, so it would just help teaching people ignore it. That said, most of the code churn is caused by enforcing this additional requirement. Crisscross from InRelease to Release.gpg is e.g. very unlikely in practice, but if we would ignore it an attacker could sidestep it this way.
2015-05-13detect Releasefile IMS hits even if the server doesn'tDavid Kalnischkies
Not all servers we are talking to support If-Modified-Since and some are not even sending Last-Modified for us, so in an effort to detect such hits we run a hashsum check on the 'old' compared to the 'new' file, we got the hashes for the 'new' already for "free" from the methods anyway and hence just need to calculated the old ones. This allows us to detect hits even with unsupported servers, which in turn means we benefit from all the new hit behavior also here.
2015-05-12implement VerifyFile as all-hashes checkDavid Kalnischkies
It isn't used much compared to what the methodname suggests, but in the remaining uses it can't hurt to check more than strictly necessary by calculating and verifying with all hashes we can compare with rather than "just" the best known hash.
2015-05-11improve partial/ cleanup in abort and failure casesDavid Kalnischkies
Especially pdiff-enhanced downloads have the tendency to fail for various reasons from which we can recover and even a successful download used to leave the old unpatched index in partial/. By adding a new method responsible for making the transaction of an individual file happen we can at specialisations especially for abort cases to deal with the cleanup. This also helps in keeping the compressed indexes around if another index failed instead of keeping the decompressed files, which we wouldn't pick up in the next call.
2015-04-19hide first pdiff merge failure debug messageDavid Kalnischkies
The sibling of this message are all guarded as debug messages, just this one had it missing an subsequently causes display issues if triggered. Git-Dch: Ignore
2015-04-19a hit on Release files means the indexes will be hits tooDavid Kalnischkies
If we get a IMSHit for the Transaction-Manager (= the InRelease file or as its still supported fallback Release + Release.gpg combo) we can assume that every file we would queue based on this manager, but already have locally is current and hence would get an IMSHit, too. We therefore save us and the server the trouble and skip the queuing in this case. Beside speeding up repetative executions of 'apt-get update' this way we also avoid hitting hashsum errors if the indexes are in fact already updated, but the Release file isn't yet as it is the case on well behaving mirrors as Release files is updated last. The implementation is a bit harder than the theory makes it sound as we still have to keep reverifying the Release files (e.g. to detect now expired once to avoid an attacker being able to silently stale us) and have to handle cases in which the Release file hits, but some indexes aren't present (e.g. user added a new foreign architecture).
2015-04-19refactor calculation of final lists/ name from URIDavid Kalnischkies
Calculating the final name of an item which it will have after everything is done and verified successfully is suprisingly complicated as while they all follow a simple pattern, the URI and where it is stored varies between the items. With some (abibreaking) redesign we can abstract this similar to how it is already down for the partial file location. Git-Dch: Ignore
2015-04-19unsigned Release files can expire, tooDavid Kalnischkies
Checking Valid-Until on an unsigned Release file doesn't give us any security brownie points as an attacker could just change the date and in practice repositories with unsigned Release files will very likely not have a Valid-Until date, but for symetry and the fact that being unsigned is currently just a warning, while expired is a fatal error.
2014-11-08guard const-ification API changesDavid Kalnischkies
Git-Dch: Ignore
2014-11-08replace ignore-deprecated #pragma dance with _PragmaDavid Kalnischkies
For compatibility we use/provide and fill quiet some deprecated methods and fields, which subsequently earns us a warning for using them. These warnings therefore have to be disabled for these codeparts and that is what this change does now in a slightly more elegant way. Git-Dch: Ignore
2014-11-08reenable patchsize limit option for pdiffsDavid Kalnischkies
One word: "doh!" Commit f6d4ab9ad8a2cfe52737ab620dd252cf8ceec43d disabled the check to prevent apt from downloading bigger patches than the index it tries to patch. Happens rarly of course, but still. Detected by scan-build complaining about a dead assignment. To make up for the mistake a test is included as well.
2014-11-06Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/no-more-acquire-guessing Conflicts: apt-pkg/acquire-item.cc
2014-11-04Call "Dequeue()" for items in AbortTransaction() to fix raceMichael Vogt
The pkgAcquire::Run() code works uses a while(ToFetch > 0) loop over the items queued for fetching. This means that we need to Deqeueue the item if we call AbortTransaction() to avoid a hang.
2014-10-29Only support Translation-* that are listed in the {In,}Release fileMichael Vogt
Handle Translation-* files exactly like Packages files (with the expection that it is ok if a download of them fails). Remove all "guessing" on apts side. This will elimimnate a bunch of errors releated to captive portals and similar. Its also more correct and removes another potential attack vector.
2014-10-26move permission changing from -item to -workerDavid Kalnischkies
The worker is the part closest to the methods, which will call the item methods according to what it gets back from the methods, it is therefore a better place to change permissions as it is very central and can do it now at the point the item is assigned to a method rather than then it is queued for download (and as before while dequeued via Done/Failure). Git-Dch: Ignore
2014-10-23chown finished partial files earlierDavid Kalnischkies
partial files are chowned by the Item baseclass to let the methods work with them. Now, this baseclass is also responsible for chowning the files back to root instead of having various deeper levels do this. The consequence is that all overloaded Failed() methods now call the Item::Failed base as their first step. The same is done for Done(). The effect is that even in partial files usually don't belong to _apt anymore, helping sneakernets and reducing possibilities of a bad method modifying files not belonging to them. The change is supported by the framework not only supporting being run as root, but with proper permission management, too, so that privilege dropping can be tested with them.
2014-10-20use c++ style instead of the last two c-arraysDavid Kalnischkies
Git-Dch: Ignore
2014-10-20aborted reverify restores file owner and permissionDavid Kalnischkies
If we get an IMS hit for an InRelease file we use the file we already have and pass it into reverification, but this changes the permissions and on abort of the transaction they weren't switched back. This is now done, additionally, every file in partial which hasn't failed gets permission and owner changed for root access as well, as it is very well possible that the next invocation will (re)use these files.
2014-10-20run acquire transactions only onceDavid Kalnischkies
Transactions are run and completed from multiple places, so it happens for unsigned repos that the Release file was commited even if it was previously aborted (due to --no-allow-insecure-repositories). The reason is simply that the "failure" of getting an InRelease/Release.gpg is currently ignored, so that the acquire process believes that nothing bad happened and commits the transaction even though the same transaction was previously aborted.
2014-10-20mark --allow-insecure-repositories message as translateableDavid Kalnischkies
Refactors a bit to ensure the same message is used in all three cases as well. Git-Dch: Ignore
2014-10-15ignore Acquire::GzipIndexes for cdrom sourcesDavid Kalnischkies
We do not support compressed indexes for cdrom sources as we rewrite some of them, so supporting it correctly could be hard. What we do instead in the meantime is probably disabling it for cdrom sources.
2014-10-15Merge branch 'debian/sid' into debian/experimentalDavid Kalnischkies
The acquire code changed completely, so this is more an import of the testcase and a new fix than the merge of an existent fix. Conflicts: apt-pkg/acquire-item.cc
2014-10-15don't cleanup cdrom files in apt-get updateDavid Kalnischkies
Regression from merging 801745284905e7962aa77a9f37a6b4e7fcdc19d0 and b0f4b486e6850c5f98520ccf19da71d0ed748ae4. While fine by itself, merged the part fixing the filename is skipped if a cdrom source is encountered, so that our list-cleanup removes what seems to be orphaned files. Closes: 765458
2014-10-13do not load filesize in pkgAcqIndexTrans explicitlyDavid Kalnischkies
The constructor is calling the baseclass pkgAcqIndex which does this already – and also does it correctly for compressed files which would overwise lead to the size of uncompressed files to be expected. Git-Dch: Ignore
2014-10-13fix compile and tests errorDavid Kalnischkies
I am pretty sure I did that before committing broken stuff… Git-Dch: Ignore
2014-10-13Fix backward compatiblity of the new pkgAcquireMethod::DropPrivsOrDie()Michael Vogt
Do not drop privileges in the methods when using a older version of libapt that does not support the chown magic in partial/ yet. To do this DropPrivileges() now will ignore a empty Apt::Sandbox::User. Cleanup all hardcoded _apt along the way.
2014-10-13trusted=yes sources are secure, we just don't know whyDavid Kalnischkies
Do not require a special flag to be present to update trusted=yes sources as this flag in the sources.list is obviously special enough. Note that this is just disabling the error message, the user will still be warned about all the (possible) failures the repository generated, it is just triggering the acceptance of the warnings on a source-by-source level. Similarily, the trusted=no flag doesn't require the user to pass additional flags to update, if the repository looks fine in the view of apt it will update just fine. The unauthenticated warnings will "just" be presented then the data is used. In case you wonder: Both was the behavior in previous versions, too.
2014-10-13display a warning for unsigned reposDavid Kalnischkies
The same message is used for InRelease if fails in gpgv, but the Release/Release.gpg duo needs to handle the failing download case as well (InRelease just defers to the duo if download fails) and print a message accompaning the insecure error to provide a hint on what is going on.
2014-10-13make --allow-insecure-repositories message an errorDavid Kalnischkies
Not using this option, but using unsigned (and co) repositories will cause these repositories to be ignored and data acquiring from them fails, so this is very well in the realms of an error and helps in making 'apt-get update' fail with a non-zero error code as well.
2014-10-10remove useless pdiff filename outputDavid Kalnischkies
Looks like a leftover from debugging. Absolutely no need for it and destroys progess reporting completely. Closes: 764737
2014-10-08Only rename StatError files in AbortTransaction()Michael Vogt
This fixes a race that we see in travis when two copy operations finish at about the same time but the bad one first. This lead to a rename of the good one and triggers a error when apt tries to verify the good version but can no longer find it.
2014-10-08pkgAcqArchive::QueueNext(): change owner/permission of DestFileMichael Vogt
The code was using FinalFile before but we only test the existance of DestFile so we use that instead.
2014-10-07Send "Fail-Reason: MaximumSizeExceeded" from the methodMichael Vogt
Communicate the fail reason from the methods to the parent and Rename() failed files.
2014-10-07Add new Acquire::MaxReleaseFileSize=10*1000*1000 optionMichael Vogt
This option controls the maximum size of Release/Release.gpg/InRelease files. The rational is that we do not know the size of these files in advance and we want to protect against a denial of service attack where someone sends us endless amounts of data until the disk is full (we do know the size all other files (Packages/Sources/debs)).
2014-10-07Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans Conflicts: apt-pkg/acquire-item.cc
2014-10-07UpperCase some functions for consistencyMichael Vogt
changeOwnerAndPermissionOfFile->ChangeOwnerAndPermissionOfFile preparePartialFile->GetPartialFileName preparePartialFileFromURI->GetPartialFileNameFromURI Git-Dch: ignore
2014-10-07display errortext for all Err as well as Ign logsDavid Kalnischkies
consistently using Item::Failed in all specializec classes helps setting up some information bits otherwise unset, so some errors had an empty reason as an error. Ign is upgraded to display the error message we ignored to further help in understanding what happens.
2014-10-07use _apt:root only for partial directoriesDavid Kalnischkies
Using a different user for calling methods is intended to protect us from methods running amok (via remotely exploited bugs) by limiting what can be done by them. By using root:root for the final directories and just have the files in partial writeable by the methods we enhance this in sofar as a method can't modify already verified data in its parent directory anymore. As a side effect, this also clears most of the problems you could have if the final directories are shared without user-sharing or if these directories disappear as they are now again root owned and only the partial directories contain _apt owned files (usually none if apt isn't running) and the directory itself is autocreated with the right permissions.
2014-10-07fixup foldmarkers in acquire-item.ccDavid Kalnischkies
Git-Dch: Ignore
2014-10-06fix warningsMichael Vogt
2014-10-06rename StopAuthentication -> CheckStopAuthentication and make it protectedMichael Vogt
2014-10-06rename AuthDone() -> CheckAuthDone()Michael Vogt