| Age | Commit message (Collapse) | Author |
|---|
|
Beside being a bit cleaner it hopefully also resolves oddball problems
I have with high levels of parallel jobs.
Git-Dch: Ignore
Reported-By: iwyu (include-what-you-use)
|
|
Reported-By: gcc -Wunused-parameter
Git-Dch: Ignore
|
|
Git-Dch: Ignore
Reported-By: gcc -Wuseless-cast
|
|
|
|
|
|
With APT::Get::List-Cleanup disabled the ed-style patch files are
lingering in the lists/ directory otherwise. That was kinda okay in the
old none-client-merge as the filename was always the same so it was
constantly overridden, but now with different names for client-merge
quiet a few could pill up on the system and are used by the next call
as it picks them up based on the filename.
|
|
Providing the benefits of both without the downsides :)
(ABI breaks or external dependencies)
For this Anthonys rred is equipped with:
- magic-filename-pickup of patches rather than explicit messages
- use of FileFd instead of FILE* to get on-the-fly uncompress
of the gzip compressed pdiff patches
The acquire code in turn stops checking for apt-file's helper
as our own rred is now clever enough for our needs.
|
|
In 51fc6def77edfb1f429a48e5169519e9e05a759b we limited the amount of
pdiff to be downloaded per index to 20. This was a compromise between
not letting it go overboard (becoming even slower) and not using
bandwidth needlessly. Now that with the POC the speed reason is gone it
makes sense again to download as much files as we possible can via pdiff
to save bandwidth (and possibly even time).
It also avoids problems with the limit in cases we were we deal with a
server merged archieve as this limit assumes a strict patch progression.
|
|
|
|
The idea of pdiffs is to avoid downloading the hole file by patching the
existing index. This works very well, but becomes slow if a lot of
patches needs to be applied to reconstruct an up-to-date index and in
recent years more and more dinstall (or similar) runs are executed
creating more and more pdiffs in the same amount of time, so pdiffs
became less useful.
The solution is simple: Reduce the amount of patches (which are very
small) which need to be applied on top of the index we have available
(which is usually pretty big).
This can be done in two ways: Either merge the patches on the
server-side so that the client has to download only one patch or the
patches are all downloaded and merged on the client-side.
The first needs a client who is doing one step at a time who can also
skip patches if it needs (APT supports this for a long time now).
The later is implemented by this commit, but depends on the server NOT
merging the patches and the patches being in a strict order in which no
patch is skipped.
This is traditionally the case for dak, but other repository creators
support merging – e.g. reprepro (which helpfully adds a flag indicating
that the patches are merged). To support both or even mixes a client
needs more information which isn't available for now.
This POC uses the external diffindex-rred included in apt-file to
do the heavy lifting of merging & applying all patches in one pass,
hence to test this feature apt-file needs to be installed.
|
|
The previous code already did this, this is just being a hell of a lot
more obvious, so that it isn't that easy to break in the future.
Git-Dch: Ignore
|
|
|
|
With a bit of trickery we can reuse the usual infrastructure we have in
place to acquire deb files for the 'download' operation as well, which
gains us authentification check & display, error messages, correct
filenames and "downloads" from the root-owned archives.
|
|
This helps ensure three things:
- each error is reported via ReportMirrorFailure
- if DestFile doesn't exist, do not attempt rename
- renames happen for every error
The last one wasn't the case for Size mismatches, which isn't nice, but
not a exploitable problem per-se as the file isn't picked up and remains
in partial/ where the following download-try will at most take it for a
partial request which fails the hashsum verification later on
Git-Dch: Ignore
|
|
--allow-unauthenticated switches the download to a pre-0.6 system in
which a package can come from any source, rather than that trusted
packages can only come from trusted sources.
To allow this the flag used to set all packages as untrusted, which is a
bit much, so we check now if the package can be acquired via an
untrusted source and only if this is the case set it as untrusted.
As APT nowadays supports setting sources as trusted via a flag in the
sources.list this mode shouldn't be used that much anymore though.
[Note that this is not the patch from the BTS]
Closes: 617690
|
|
|
|
Conflicts:
apt-pkg/tagfile.h
|
|
The constructors of our (clear)sign-acquire-items move a pre-existent
file for error-recovery away, which gets restored or discarded later as
the acquire progresses, but --print-uris never really starts the
acquire process, so the files aren't restored (as they should).
To fix this both get a destructor which checks for signs of acquire
doing anything and if it hasn't the file is restored.
Note that these virtual destructors theoretically break the API, but
only with classes extending the sign-acquire-items and nobody does this,
as it would be insane for library users to fiddle with Acquire
internals – and these classes are internals.
Closes: 719263
|
|
|
|
On CD-ROMs Translation-* files are only in compressed form included in
the Release file. This used to work while we had no record of
Translation-* files in the Release file at all as APT would have just
guessed the (compressed) filename and accepted it (unchecked), but now
that it checks for the presents of entries and if it finds records it
expects the uncompressed to be verifiable.
This commit relaxes this requirement again to fix the regression.
We are still secure "enough" as we can validate the compressed file we have
downloaded, so we don't loose anything by not requiring a hashsum for
the uncompressed files to double-check them.
Closes: 717665
|
|
Before we download the 'new' InRelease file the old file will be moved
out of the way with the name 'foobar_InRelease.reverify', so if no
partial file for the 'new' file exists take the modification time from
this reverify file, so that if we get an IMS hit for the InRelease file
we can move back the reverify file as new file rather than downloading
the 'new' file even though we already have it.
We do the same for Release files and this happened to work until the
reverify renaming was corrected for InRelease files.
|
|
translations. Closes: #705087
|
|
|
|
|
|
- keep the last good InRelease file around just as we do it with
Release.gpg in case the new one we download isn't good for us
|
|
|
|
|
|
|
|
- remove 'old' InRelease file if we can't get a new one before
proceeding with Release.gpg to avoid the false impression of a still
trusted repository by a (still present) old InRelease file.
Thanks to Simon Ruderich for reporting this issue! (CVE-2012-0214)
Effected are all versions >= 0.8.11
Possible attack summary:
- Attacker needs to find a user which has run at least one successful
'apt-get update' against an archive providing InRelease files.
- Create a Packages file with his preferred content.
- Attacker then prevents the download of InRelease, Release and
Release.gpg (alternatively he creates a valid Release file and sends
this, the other two files need to be missing either way).
- User updates against this, getting the modified Packages file without
any indication of being unsigned (beside the "Ign InRelease" and
"Ign Release.gpg" in the output of 'apt-get update').
=> deb files from this source are considered 'trusted' (and therefore the
user isn't asked for an additional confirmation before install)
|
|
Beware: pdiffs for Translation-* are only acquired if their
availability is advertised in the Release file.
|
|
- drop support for i18n/Index file (introduced in 0.8.11) and use
the Release file instead to get the Translations (Closes: #649314)
* ftparchive/writer.cc:
- add 'Translation-*' to the default patterns
i18n/Index was never used outside debian - and even here it isn't used
consistently as only 'main' has such a file. As the Release file now
includes the Translation-* files we therefore drop support for i18n/Index.
A version supporting it was never part of a debian release and still
supporting it would mean that we get 99% of the time a 404 as response
to the request anyway and confuse archive maintainers who want to
provide all files APT tries to acquire.
|
|
on the FileFd instead
|
|
|
|
|
|
|
|
|
|
- if no Release.gpg file is found try to verify with hashes,
but do not fail if a hash can't be found
|
|
|
|
|
|
done on the mirco-optimazation level, so lets fix them:
(performance) Possible inefficient checking for emptiness.
(performance) Prefer prefix ++/-- operators for non-primitive types.
|
|
|
|
|
|
- add regression test for hashsum verification
* apt-pkg/acquire-item.cc:
- if no Release.gpg file is found, still load the hashes for
verification (closes: #636314) and add test
|
|
- if no Release.gpg file is found, still load the hashes for
verification (closes: #636314) and add test
|
|
|
|
- do not check for a "Package" tag in optional index targets
like the translations index
|
|
- cleanup around OptionalIndexTarget and SubIndexTarget
|
|
|
|
- try even harder to support really big files in the fetcher by
converting (hopefully) everything to 'long long' (Closes: #632271)
|
|
|
