| Age | Commit message (Collapse) | Author |
|---|
|
We do not deal a lot with URIs which need encoding, but then we do it is
a pain that we store it decoded in the acquire system as it means we
have to decode and reencode URIs eventually which is potentially giving
us slightly different URIs.
We see that in our own testing framework while setting up redirects as
the config options are effectively double-encoded and decoded to pass
them around successfully as otherwise %2f and / in an URI are treated
the same.
This commit adds the infrastructure for methods to opt into getting URIs
send in encoded form (and returning them to us in encoded form, too) so
that we eventually do not have to touch the URIs which is how it should
be. This means though that we have to deal with methods who do not
support this yet (aka: all at the moment) for which we decode and encode
while communicating with them.
|
|
|
|
|
|
|
|
These status fields belong to the current item, move them there. This
prepares us for eventually having multiple current items.
|
|
|
|
Clean up the code, make it neat, lalala
|
|
Allowing a method to request work from other methods is a powerful
capability which could be misused or exploited, so to slightly limited
the surface let method opt-in into this capability on startup.
|
|
This makes it easier to see which headers includes what.
The changes were done by running
git grep -l '#\s*include' \
| grep -E '.(cc|h)$' \
| xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/'
To modify all include lines by adding a space, and then running
./git-clang-format.sh.
|
|
Having messages being printed on the error stack and confirm them by
commandline flags is an okayish first step, but some frontends will
probably want to have a more interactive feeling here with a proper
question the user can just press yes/no for as for some frontends a
commandline flag makes no sense…
|
|
Most of them in (old) code comments. The two instances of user visible
string changes the po files of the manpages are fixed up as well.
Gbp-Dch: Ignore
Reported-By: spellintian
|
|
Employ a priority queue instead of a normal queue to hold
the items; and only add items to the running pipeline if
their priority is the same or higher than the priority
of items in the queue.
The priorities are designed for a 3 stage pipeline system:
In stage 1, all Release files and .diff/Index files are fetched. This
allows us to determine what files remain to be fetched, and thus
ensures a usable progress reporting.
In stage 2, all Pdiff patches are fetched, so we can apply them
in parallel with fetching other files in stage 3.
In stage 3, all other files are fetched (complete index files
such as Contents, Packages).
Performance improvements, mainly from fetching the pdiff patches
before complete files, so they can be applied in parallel:
For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update
of Debian unstable and testing with Contents and appstream for
amd64 and i386, update time reduced from 37 seconds to 24-28
seconds.
Previously, apt would first download new DEP11 icon tarballs
and metadata files, causing the CPU to be idle. By fetching
the diffs in stage 2, we can now patch our contents and Packages
files while we are downloading the DEP11 stuff.
|
|
Git-Dch: Ignore
|
|
This fixes a regression caussed by commit
95278287f4e1eeaf5d96749d6fc9bfc53fb400d0
that moved the error detection of RunFds() later into the loop.
However this broke detecting issues like dead acquire methods.
Instead of relying on the global error state (which is bad)
we now pass a boolean value back from RunFds() and break on
false.
Closes: #806406
|
|
The acquire system actually uses usec pulse intervals, so the
previous value was correct (500ms) whereas the new value is
now 5s.
It's a bit unfortunate that the two systems use different units
for pulse intervals, but probably not much we can do about it.
This partially reverts commit eaf21c2144fa8dc4be8581dc69cf88cb38e30ce2.
|
|
Closes: #799857
|
|
We had a very similar method previously for our own private usage, but
with some generalisation we can move this check into the acquire system
proper so that all frontends profit from this compatibility change.
As we are disabling a security feature here a warning is issued and
frontends are advised to consider reworking their download logic if
possible.
Note that this is implemented as an all or nothing situation: We can't
just (not) drop privileges for a subset of the files in a fetcher, so in
case you have to download some files with and some without you need to
use two fetchers.
|
|
Reported-By: codespell
|
|
CurrentItem previously was a DescItem, so let's make QItem a
DescItem to not break things.
|
|
Doing this disables the implicit copy assignment operator (among others)
which would cause hovac if used on the classes as it would just copy the
pointer, not the data the d-pointer points to. For most of the classes
we don't need a copy assignment operator anyway and in many classes it
was broken before as many contain a pointer of some sort.
Only for our Cacheset Container interfaces we define an explicit copy
assignment operator which could later be implemented to copy the data
from one d-pointer to the other if we need it.
Git-Dch: Ignore
|
|
Some of them modify the ABI, but given that we prepare a big one
already, these few hardly count for much.
Git-Dch: Ignore
|
|
To have a chance to keep the ABI for a while we need all three to team
up. One of them missing and we might loose, so ensuring that they are
available is a very tedious but needed task once in a while.
Git-Dch: Ignore
|
|
This is an unlikely event for indexes and co, but it can happen quiet
easily e.g. for changelogs where you want to get the changelogs for
multiple binary package(version)s which happen to all be built from a
single source.
The interesting part is that the Acquire system actually detected this
already and set the item requesting the URI again to StatDone - expect
that this is hardly sufficient: an Item must be Complete=true as well
to be considered truely done and that is only the tip of the ::Done
handling iceberg. So instead of this StatDone hack we allow QItems to be
owned by multiple items and notify all owners about everything now,
so that for the point of each item they got it downloaded just for them.
|
|
If we get a IMSHit for the Transaction-Manager (= the InRelease file or
as its still supported fallback Release + Release.gpg combo) we can
assume that every file we would queue based on this manager, but already
have locally is current and hence would get an IMSHit, too. We therefore
save us and the server the trouble and skip the queuing in this case.
Beside speeding up repetative executions of 'apt-get update' this way we
also avoid hitting hashsum errors if the indexes are in fact already
updated, but the Release file isn't yet as it is the case on well
behaving mirrors as Release files is updated last.
The implementation is a bit harder than the theory makes it sound as we
still have to keep reverifying the Release files (e.g. to detect now expired
once to avoid an attacker being able to silently stale us) and have to
handle cases in which the Release file hits, but some indexes aren't
present (e.g. user added a new foreign architecture).
|
|
partial files are chowned by the Item baseclass to let the methods work
with them. Now, this baseclass is also responsible for chowning the
files back to root instead of having various deeper levels do this.
The consequence is that all overloaded Failed() methods now call the
Item::Failed base as their first step. The same is done for Done().
The effect is that even in partial files usually don't belong to
_apt anymore, helping sneakernets and reducing possibilities of a bad
method modifying files not belonging to them.
The change is supported by the framework not only supporting being run
as root, but with proper permission management, too, so that privilege
dropping can be tested with them.
|
|
Reimplementing an inline method is opening a can of worms we don't want
to open if we ever want to us a d-pointer in those classes, so we do the
only thing which can save us from hell: move the destructors into the cc
sources and we are good.
Technically not an ABI break as the methods inline or not do the same
(nothing), so a program compiled against the old version still works
with the new version (beside that this version is still in experimental,
so nothing really has been build against this library anyway).
Git-Dch: Ignore
|
|
Reworks the API involved in creating and setting up the fetcher to be a
bit more pleasent to look at and work with as e.g. an empty string for
no lock isn't very nice. With the lock we can also stop creating all our
partial directories "just in case". This way we can also be a bit more
aggressive with the partial directory itself as with a lock, we know we
will gone need it.
|
|
apt-get download and changelog as well as apt-helper reuse the acquire
system for their own proposes without requiring the directories the
fetcher wants to create, which is a problem if you run them as non-root
and the directories do not exist as it greets you with:
E: Archives directory /var/cache/apt/archives/partial is missing. -
Acquire (13: Permission denied)
Closes: 762898
|
|
Reported-By: cppcheck
Git-Dch: Ignore
|
|
for both items and bytes
|
|
Git-Dch: Ignore
Reported-By: gcc -Wsuggest-attribute={pure,const,noreturn}
|
|
Beside being a bit cleaner it hopefully also resolves oddball problems
I have with high levels of parallel jobs.
Git-Dch: Ignore
Reported-By: iwyu (include-what-you-use)
|
|
Git-Dch: Ignore
|
|
The breakage is just to big for now, so guard the change with
#ifndef APT_8_CLEANER_HEADERS and be nice to library users
|
|
|
|
* apt-pkg/contrib/sha1.cc:
- fix illegally casts of on-stack buffer to a type requiring more
alignment than it has resulting in segfaults on sparc (Closes: #634696)
* apt-pkg/contrib/cdromutl.cc:
- fix escape problem when looking for the mounted devices
* apt-pkg/contrib/strutl.{h,cc}, test/libapt/strutil_test.cc:
- add new DeEscapeString() similar to DeQuoteString but
unescape character escapes like \0XX and \xXX (plus added
test)
* refresh po/*
* cmdline/apt-get.cc:
- fix missing download progress in apt-get download
- do not require unused partial dirs in 'source' (Closes: #633510)
- buildconflicts effect all architectures
- implement MultiarchCross for build-dep and source (Closes: #632221)
* cmdline/apt-key:
- use a tmpfile instead of /etc/apt/secring.gpg (Closes: #632596)
* debian/apt.postinst:
- remove /etc/apt/secring.gpg if it is an empty file
* doc/apt-cache.8.xml:
- apply madison typofix from John Feuerstein, thanks! (Closes: #633455)
* apt-pkg/policy.cc:
- emit an error on unknown APT::Default-Release value (Closes: #407511)
* apt-pkg/aptconfiguration.cc:
- ensure that native architecture is if not specified otherwise the
first architecture in the Architectures vector
* apt-pkg/deb/deblistparser.cc:
- Strip only :any and :native if MultiArch should be stripped as it is
save to ignore them in non-MultiArch contexts but if the dependency
is a specific architecture (and not the native) do not strip
|
|
|
|
|
|
- try even harder to support really big files in the fetcher by
converting (hopefully) everything to 'long long' (Closes: #632271)
|
|
|
|
- create Dir::Log if needed to support /var/log as tmpfs or similar,
inspired by Thomas Bechtold, thanks! (Closes: #523919, LP: #220239)
Easily done by moving a private method from pkgAcquire into the public
area of fileutl.cc to be able to use it also in here
|
|
- switch i{Download,Usr}Size from double to (un)signed long long
The biggest reason is that this saves a lot of float point operations
we do in AddSizes() on integers. The only reason i see that this was
a double is that it was 64bit long and can therefore store bigger
values than int/long, but with the availablity of (un)signed long long
we are now also at 64bit and can store sizes more than 8 Exabytes big -
by the time this will be a limit the C/C++ Standard will have bigger types,
hopefully.
|
|
- add a class WeakPointable which allows one to register weak pointers to
an object which will be set to NULL when the object is deallocated.
* [ABI break] apt-pkg/acquire{-worker,-item,}.h:
- subclass pkgAcquire::{Worker,Item,ItemDesc} from WeakPointable.
|
|
- add a parent-guarded "mkdir -p" as CreateDirectory()
* apt-pkg/acquire.{cc,h}:
- add a delayed constructor with Setup() for success reporting
- check for and create directories in Setup if needed instead of
error out unfriendly in the Constructor (Closes: #523920, #525783)
- optional handle a lock file in Setup()
* cmdline/apt-get.cc:
- remove the lock file handling and let Acquire take care of it instead
|
|
|
|
|
|
|
|
- deal better with duplicated sources.list entries (avoid
double queuing of URLs) - this fixes hangs in bzip/gzip
|
|
|
|
|
