summaryrefslogtreecommitdiff
path: root/apt-pkg/acquire-item.h
AgeCommit message (Collapse)Author
2015-06-15deal better with acquiring the same URI multiple timesDavid Kalnischkies
This is an unlikely event for indexes and co, but it can happen quiet easily e.g. for changelogs where you want to get the changelogs for multiple binary package(version)s which happen to all be built from a single source. The interesting part is that the Acquire system actually detected this already and set the item requesting the URI again to StatDone - expect that this is hardly sufficient: an Item must be Complete=true as well to be considered truely done and that is only the tip of the ::Done handling iceberg. So instead of this StatDone hack we allow QItems to be owned by multiple items and notify all owners about everything now, so that for the point of each item they got it downloaded just for them.
2015-06-15provide a public interface for acquiring changelogsDavid Kalnischkies
Provided is a specialized acquire item which given a version can figure out the correct URI to try by itself and if not provides an error message alongside with static methods to get just the URI it would try to download if it should just be displayed or similar such. The URI is constructed as follows: Release files can provide an URI template in the "Changelogs" field, otherwise we lookup a configuration item based on the "Label" or "Origin" of the Release file to get a (hopefully known) default value for now. This template should contain the string CHANGEPATH which is replaced with the information about the version we want the changelog for (e.g. main/a/apt/apt_1.1). This middleway was choosen as this path part was consistent over the three known implementations (+1 defunct), while the rest of the URI varies widely between them. The benefit of this construct is that it is now easy to get changelogs for Debian packages on Ubuntu and vice versa – even at the moment where the Changelogs field is present nowhere. Strictly better than what apt-get had before as it would even fail to get changelogs from security… Now it will notice that security identifies as Origin: Debian and pick this setting (assuming again that no Changelogs field exists). If on the other hand security would ship its changelogs in a different location we could set it via the Label option overruling Origin. Closes: 687147, 739854, 784027, 787190
2015-06-11use IndexTarget to get to IndexFileDavid Kalnischkies
Removes a bunch of duplicated code in the deb-specific parts. Especially the Description part is now handled centrally by IndexTarget instead of being duplicated to the derivations of IndexFile. Git-Dch: Ignore
2015-06-10stop using IndexTarget pointers which are never freedDavid Kalnischkies
Creating and passing around a bunch of pointers of IndexTargets (and of a vector of pointers of IndexTargets) is probably done to avoid the 'costly' copy of container, but we are really not in a timecritical operation here and move semantics will help us even further in the future. On the other hand we never do a proper cleanup of these pointers, which is very dirty, even if structures aren't that big… The changes will effecting many items only effect our own hidden class, so we can do that without fearing breaking interfaces or anything. Git-Dch: Ignore
2015-06-10store all targets data in IndexTarget structDavid Kalnischkies
We still need an API for the targets, so slowly prepare the IndexTargets to let them take this job. Git-Dch: Ignore
2015-06-09remove debianism file-content verificationDavid Kalnischkies
The code requires every index file we download to have a Package field, but that doesn't hold true for all index we might want to download in the future. Some might not even be deb822 formatted files… The check was needed as apt used to accept unverifiable files like Translation-*, but nowadays it requires hashes for these as well. Even for unsigned repositories we interpret the Release file as binding now, which means this check isn't triggerable expect for repositories which do not have a Release file at all – something which is highly discouraged! Git-Dch: Ignore
2015-06-09cleanup pdiff support detection decisionDavid Kalnischkies
Its a bit unclean to create an item just to let the item decide that it can't do anything and let it fail, so instead we let the item creator decide in all cases if patching should be attempted. Also pulls a small trick to get the hashes for the current file without calculating them by looking at the 'old' Release file if we have it. Git-Dch: Ignore
2015-06-09support hashes for compressed pdiff filesDavid Kalnischkies
At the moment we only have hashes for the uncompressed pdiff files, but via the new '$HASH-Download' field in the .diff/Index hashes can be provided for the .gz compressed pdiff file, which apt will pick up now and use to verify the download. Now, we "just" need a buy in from the creators of repositories…
2015-06-09check patch hashes in rred worker instead of in the handlerDavid Kalnischkies
rred is responsible for unpacking and reading the patch files in one go, but we currently only have hashes for the uncompressed patch files, so the handler read the entire patch file before dispatching it to the worker which would read it again – both with an implicit uncompress. Worse, while the workers operate in parallel the handler is the central orchestration unit, so having it busy with work means the workers do (potentially) nothing. This means rred is working with 'untrusted' data, which is bad. Yet, having the unpack in the handler meant that the untrusted uncompress was done as root which isn't better either. Now, we have it at least contained in a binary which we can harden a bit better. In the long run, we want hashes for the compressed patch files through to be safe.
2015-06-09rework hashsum verification in the acquire systemDavid Kalnischkies
Having every item having its own code to verify the file(s) it handles is an errorprune process and easy to break, especially if items move through various stages (download, uncompress, patching, …). With a giant rework we centralize (most of) the verification to have a better enforcement rate and (hopefully) less chance for bugs, but it breaks the ABI bigtime in exchange – and as we break it anyway, it is broken even harder. It shouldn't effect most frontends as they don't deal with the acquire system at all or implement their own items, but some do and will need to be patched (might be an opportunity to use apt on-board material). The theory is simple: Items implement methods to decide if hashes need to be checked (in this stage) and to return the expected hashes for this item (in this stage). The verification itself is done in worker message passing which has the benefit that a hashsum error is now a proper error for the acquire system rather than a Done() which is later revised to a Failed().
2015-05-18treat older Release files than we already have as an IMSHitDavid Kalnischkies
Valid-Until protects us from long-living downgrade attacks, but not all repositories have it and an attacker could still use older but still valid files to downgrade us. While this makes it sounds like a security improvement now, its a bit theoretical at best as an attacker with capabilities to pull this off could just as well always keep us days (but in the valid period) behind and always knows which state we have, as we tell him with the If-Modified-Since header. This is also why this is 'silently' ignored and treated as an IMSHit rather than screamed at the user as this can at best be an annoyance for attackers. An error here would 'regularily' be encountered by users by out-of-sync mirrors serving a single run (e.g. load balancer) or in two consecutive runs on the other hand, so it would just help teaching people ignore it. That said, most of the code churn is caused by enforcing this additional requirement. Crisscross from InRelease to Release.gpg is e.g. very unlikely in practice, but if we would ignore it an attacker could sidestep it this way.
2015-05-13detect Releasefile IMS hits even if the server doesn'tDavid Kalnischkies
Not all servers we are talking to support If-Modified-Since and some are not even sending Last-Modified for us, so in an effort to detect such hits we run a hashsum check on the 'old' compared to the 'new' file, we got the hashes for the 'new' already for "free" from the methods anyway and hence just need to calculated the old ones. This allows us to detect hits even with unsupported servers, which in turn means we benefit from all the new hit behavior also here.
2015-05-11improve partial/ cleanup in abort and failure casesDavid Kalnischkies
Especially pdiff-enhanced downloads have the tendency to fail for various reasons from which we can recover and even a successful download used to leave the old unpatched index in partial/. By adding a new method responsible for making the transaction of an individual file happen we can at specialisations especially for abort cases to deal with the cleanup. This also helps in keeping the compressed indexes around if another index failed instead of keeping the decompressed files, which we wouldn't pick up in the next call.
2015-04-19a hit on Release files means the indexes will be hits tooDavid Kalnischkies
If we get a IMSHit for the Transaction-Manager (= the InRelease file or as its still supported fallback Release + Release.gpg combo) we can assume that every file we would queue based on this manager, but already have locally is current and hence would get an IMSHit, too. We therefore save us and the server the trouble and skip the queuing in this case. Beside speeding up repetative executions of 'apt-get update' this way we also avoid hitting hashsum errors if the indexes are in fact already updated, but the Release file isn't yet as it is the case on well behaving mirrors as Release files is updated last. The implementation is a bit harder than the theory makes it sound as we still have to keep reverifying the Release files (e.g. to detect now expired once to avoid an attacker being able to silently stale us) and have to handle cases in which the Release file hits, but some indexes aren't present (e.g. user added a new foreign architecture).
2015-04-19refactor calculation of final lists/ name from URIDavid Kalnischkies
Calculating the final name of an item which it will have after everything is done and verified successfully is suprisingly complicated as while they all follow a simple pattern, the URI and where it is stored varies between the items. With some (abibreaking) redesign we can abstract this similar to how it is already down for the partial file location. Git-Dch: Ignore
2014-11-08guard const-ification API changesDavid Kalnischkies
Git-Dch: Ignore
2014-11-08mark internal interfaces as hiddenDavid Kalnischkies
We have a bunch of classes which are of no use for the outside world, but were still exported and so needed to preserve ABI/API. Marking them as hidden to not export them any longer is a big API break in theory, but in practice nobody is using them – as if they would its a bug.
2014-11-06Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/no-more-acquire-guessing Conflicts: apt-pkg/acquire-item.cc
2014-11-04apt-pkg/acquire-item.h: make friend declaration compatible with older gccMichael Vogt
2014-11-04Call "Dequeue()" for items in AbortTransaction() to fix raceMichael Vogt
The pkgAcquire::Run() code works uses a while(ToFetch > 0) loop over the items queued for fetching. This means that we need to Deqeueue the item if we call AbortTransaction() to avoid a hang.
2014-10-29Only support Translation-* that are listed in the {In,}Release fileMichael Vogt
Handle Translation-* files exactly like Packages files (with the expection that it is ok if a download of them fails). Remove all "guessing" on apts side. This will elimimnate a bunch of errors releated to captive portals and similar. Its also more correct and removes another potential attack vector.
2014-10-23chown finished partial files earlierDavid Kalnischkies
partial files are chowned by the Item baseclass to let the methods work with them. Now, this baseclass is also responsible for chowning the files back to root instead of having various deeper levels do this. The consequence is that all overloaded Failed() methods now call the Item::Failed base as their first step. The same is done for Done(). The effect is that even in partial files usually don't belong to _apt anymore, helping sneakernets and reducing possibilities of a bad method modifying files not belonging to them. The change is supported by the framework not only supporting being run as root, but with proper permission management, too, so that privilege dropping can be tested with them.
2014-10-07Add new Acquire::MaxReleaseFileSize=10*1000*1000 optionMichael Vogt
This option controls the maximum size of Release/Release.gpg/InRelease files. The rational is that we do not know the size of these files in advance and we want to protect against a denial of service attack where someone sends us endless amounts of data until the disk is full (we do know the size all other files (Packages/Sources/debs)).
2014-10-07Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans Conflicts: apt-pkg/acquire-item.cc
2014-10-07use _apt:root only for partial directoriesDavid Kalnischkies
Using a different user for calling methods is intended to protect us from methods running amok (via remotely exploited bugs) by limiting what can be done by them. By using root:root for the final directories and just have the files in partial writeable by the methods we enhance this in sofar as a method can't modify already verified data in its parent directory anymore. As a side effect, this also clears most of the problems you could have if the final directories are shared without user-sharing or if these directories disappear as they are now again root owned and only the partial directories contain _apt owned files (usually none if apt isn't running) and the directory itself is autocreated with the right permissions.
2014-10-06rename StopAuthentication -> CheckStopAuthentication and make it protectedMichael Vogt
2014-10-06rename AuthDone() -> CheckAuthDone()Michael Vogt
2014-10-06cleanup pkgAcq*::Failed()Michael Vogt
2014-10-06Rework pkgAcqMeta{Index,Sig,ClearSig}::Done() for readabilityMichael Vogt
Move common code out but do not use subclassing for ::Done to make it easier to understand what each class is doing when its done
2014-10-06add new "SetActiveSubprocess()Michael Vogt
2014-10-02cleanup around pkgAcqMetaSig and improved testsMichael Vogt
2014-10-02add a bunch of docstrings etcMichael Vogt
2014-10-02fix crashMichael Vogt
2014-10-02donkults fixesMichael Vogt
2014-10-02Cleanup pkgAcqIndexMichael Vogt
2014-10-01refactor and add pkgAcqIndex::ValidateFile()Michael Vogt
2014-09-30support parsing of all hashes for pdiffDavid Kalnischkies
The fileformat of a pdiff index stores currently only SHA1 hashes. With this change, we look for all other hashes we support as well and take what we get, so that we can work after the release of jessie to get right of SHA1 if we want to. Note that the completely patched file is and was checked against the hashes collected from the Release file, so this transition isn't mission critical.
2014-09-30mark private methods as hiddenDavid Kalnischkies
We are the only possible users of private methods, so we are also the only users who can potentially export them via using them in inline methods. The point is: We don't need these symbols exported if we don't do this, so marking them as hidden removes some methods from the API without breaking anything as nobody could have used them. Git-Dch: Ignore
2014-09-29cleanupMichael Vogt
2014-09-29refactorMichael Vogt
2014-09-29Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans Conflicts: apt-pkg/acquire-item.cc apt-pkg/acquire-item.h methods/gpgv.cc
2014-09-28replace c-string Mode with c++-string ActiveSubprocessDavid Kalnischkies
A long-lasting FIXME in the acquire code points out the problem that we e.g. for decompressors assign c-string representations of c++-strings to the Mode variable, which e.g. cppcheck points out as very bad. In practice, nothing major happens as the c++-strings do not run out of scope until Mode would do, but that is bad style and fragile, so the obvious proper fix is to use a c++ string for storage to begin with. The slight complications stems from the fact that progress reporting code in frontends potentially uses Mode and compares it with NULL, which can't be done with std::string, so instead of just changing the type we introduce a new variable and deprecate the old one. Git-Dch: Ignore
2014-09-26Do not allow going from authenticated to unauthenticated repoMichael Vogt
Also rework the way we load the Release file, so it only after Release.gpg verified the Release file. The rational is that we never want to load untrusted data into our parsers. Only stuff verified with gpg or by its hashes get loaded. To load untrusted data you now need to use apt-get update --allow-unauthenticated.
2014-09-25Revert making pkgAcquire::Item::DescURI() "const"Michael Vogt
Revert because its a API change and the gain does not justify the extra work to make the required changes in the consumers of this interface at this point.
2014-09-23make pdiff transactional (but at the cost of a CopyFile()Michael Vogt
2014-09-23cleanup, fix test-apt-update-unauth as the behavior of apt changedMichael Vogt
2014-09-23cleanupMichael Vogt
2014-09-23Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans Conflicts: apt-pkg/acquire-item.cc apt-pkg/acquire-item.h methods/copy.cc test/integration/test-hashsum-verification
2014-09-23Merge branch 'debian/sid' into debian/experimentalMichael Vogt
Conflicts: apt-pkg/acquire-item.cc apt-pkg/acquire-item.h apt-pkg/cachefilter.h configure.ac debian/changelog
2014-09-17fix gcc warningsMichael Vogt