| Age | Commit message (Collapse) | Author |
|---|
|
Progress reporting used an "upper bound" on files we might get, expect
that this wasn't correct in case pdiff entered the picture. So instead
of calculating a value which is perhaps incorrect, we just accept that
we can't tell how many files we are going to download and just keep at
0% until we know. Additionally, if we have pdiffs we wait until we got
these (sub)index files, too.
That could all be done better by downloading all Release files first and
planing with them in hand accordingly, but one step at a time.
|
|
Queues feeding workers like rred are created in a random pattern to get
a few of them to run in parallel – but if we already have an idling queue
we don't need to assign it to a (potentially new) random queue as that
saves us the (agruably small) overhead of starting up a new queue,
avoids adding jobs to an already busy queue while others idle and as
a bonus reduces the size of debug logs a bit.
We also keep starting new queues now until we reach our limit before
we assign work at random to them, which should give us a more effective
utilisation overall compared to potentially adding work to busy queues
while we haven't reached our queue limit yet.
|
|
For the non-pdiff case, we have can have accurate progress
reporting because after fetching the {,In}Release files we know
how many IndexFiles will be fetched and what size they have.
Therefore init the filesize early (in pkgAcqIndex::Init) and
ensure that in Acquire::Pulse() looks at already downloaded
bits when calculating the progress in Acquire::Pulse.
Also improve debug output of Debug::acquire::progress
|
|
pkgAcqChangelog has the default behaviour of downloading a changelog to
a temporary directory (inside /tmp, not /tmp directly), which is cleaned
up on shutdown, but this can be overridden to store the changelog more
permanently – but that caries a permission problem.
For changelog we can 'easily' solve this by always downloading to a
temporary directory and only move it out of there on done.
|
|
Introduced in 9d2a8a7388cf3b0bbbe92f6b0b30a533e1167f40 apt tries to
merge actions like downloading the same (as judged by hashes) file
into doing it once. The implementation was very simple in that it isn't
planing at all. Turns out that it works 90% of the time just fine, but
has issues in more complicated situations in which items can be in
different stages downloading different files emitting potentially the
"wrong" hash – like while pdiffs are worked on we might end up copying
the patch instead of the result file giving us very strange errors in
return. Reverting the change until we can implement a better planing
solution seems to be the best course of action even if its sad.
Closes: 810046
|
|
The maximum parallelization soft limit is the number of CPU
cores * 2 on systems defining _SC_NPROCESSORS_ONLN. The hard
limit in all cases is Acquire::QueueHost::Limit.
|
|
Gbp-Dch: ignore
|
|
We need to pass 0llu instead of 0 as the init value, otherwise
std::accumulate will calculate with ints.
Reported-by: Raphaël Hertzog
|
|
If it does not exist disabled priv dropping as there is nothing
we can drop to. This will unblock people with special chroots
or systems that deleted the "_apt" user.
Closes: #806406
|
|
This fixes a regression caussed by commit
95278287f4e1eeaf5d96749d6fc9bfc53fb400d0
that moved the error detection of RunFds() later into the loop.
However this broke detecting issues like dead acquire methods.
Instead of relying on the global error state (which is bad)
we now pass a boolean value back from RunFds() and break on
false.
Closes: #806406
|
|
In ce1f3a2c we started warning about failing unlinking, which we
consistently do for directories. That isn't a problem as directories
usually aren't in the places we do want to clean up – with the potential
exeception of "lost+found", so lets ignore it like we ignore our own
partial/ subdirectory.
Closes: 805424
|
|
In 0940230d we started dropping privileges for file (and a bit later for
copy, too) with the intend of uniforming this for all methods. The
commit message says that the source will likely fail based on the
compressors already – and there isn't much secret in the repository
content. After all, after apt has run the update everyone can access the
content via apt anyway…
There are sources through which worked before which are mostly
single-deb (and those with the uncompressed files available).
The first one being especially surprising for users maybe, so instead of
failing, we make it so that apt detects that it can't access a source as
_apt and if so doesn't drop (for all sources!) privileges – but we limit
this to file/copy, so the uncompress which might be needed will still
fail – but that failed before this regression.
We display a notice about this, mostly so that if it still fails (e.g.
compressed) the user has some idea what is wrong.
Closes: 805069
|
|
Unlinking /dev/null is bad, we shouldn't do that. Also, we should print
at least a warning if we tried to unlink a file but didn't manage to
pull it of (ignoring the case were the file is /dev/null or doesn't
exist in the first place).
This got triggered by a relatively unlikely to cause problem in
pkgAcquire::Worker::PrepareFiles which would while temporary
uncompressed files (which are set to keep compressed) figure out that to
files are the same and prepare for sharing by deleting them. Bad move.
That also shows why not printing a warning is a bad idea as this hide
the error for in non-root test runs.
Git-Dch: Ignore
|
|
Reported-By: gcc
Understandable: no
Git-Dch: Ignore
|
|
Reported-By: scan-build
Git-Dch: Ignore
|
|
Our error reporting is historically grown into some kind of mess.
A while ago I implemented stacking for the global error which is used in
this commit now to wrap calls to functions which do not report (all)
errors via return, so that only failures in those calls cause a failure
to propergate down the chain rather than failing if anything
(potentially totally unrelated) has failed at some point in the past.
This way we can avoid stopping the entire acquire process just because a
single source produced an error for example. It also means that after
the acquire process the cache is generated – even if the acquire
process had failures – as we still have the old good data around we can and
should generate a cache for (again).
There are probably more instances of this hiding, but all these looked
like the easiest to work with and fix with reasonable (aka net-positive)
effects.
|
|
Various smaller improvements so that the check deals better with already
downloaded files, relative paths and other things.
Git-Dch: Ignore
|
|
We had a very similar method previously for our own private usage, but
with some generalisation we can move this check into the acquire system
proper so that all frontends profit from this compatibility change.
As we are disabling a security feature here a warning is issued and
frontends are advised to consider reworking their download logic if
possible.
Note that this is implemented as an all or nothing situation: We can't
just (not) drop privileges for a subset of the files in a fetcher, so in
case you have to download some files with and some without you need to
use two fetchers.
|
|
Git-Dch: ignore
|
|
|
|
Doing this disables the implicit copy assignment operator (among others)
which would cause hovac if used on the classes as it would just copy the
pointer, not the data the d-pointer points to. For most of the classes
we don't need a copy assignment operator anyway and in many classes it
was broken before as many contain a pointer of some sort.
Only for our Cacheset Container interfaces we define an explicit copy
assignment operator which could later be implemented to copy the data
from one d-pointer to the other if we need it.
Git-Dch: Ignore
|
|
Some of them modify the ABI, but given that we prepare a big one
already, these few hardly count for much.
Git-Dch: Ignore
|
|
To have a chance to keep the ABI for a while we need all three to team
up. One of them missing and we might loose, so ensuring that they are
available is a very tedious but needed task once in a while.
Git-Dch: Ignore
|
|
Progress reports once in a while which is a bit to unpredictable for
testcases, so we enforce a steady progress for them in the hope that
this makes the tests (mostly test-apt-progress-fd) a bit more stable.
Git-Dch: Ignore
|
|
It shouldn't be too common, but sometimes people have multiple mirrors
in the sources or otherwise repositories with the same content. Now that
we gracefully can handle multiple requests to the same URI, we can also
fold multiple requests with the same expected hashes into one. Note that
this isn't trying to find oppertunities for merging, but just merges if
it happens to encounter the oppertunity for it.
This is most obvious in the new testcase actually as it needs to delay
the action to give the acquire system enough time to figure out that
they can be merged.
|
|
This is an unlikely event for indexes and co, but it can happen quiet
easily e.g. for changelogs where you want to get the changelogs for
multiple binary package(version)s which happen to all be built from a
single source.
The interesting part is that the Acquire system actually detected this
already and set the item requesting the URI again to StatDone - expect
that this is hardly sufficient: an Item must be Complete=true as well
to be considered truely done and that is only the tip of the ::Done
handling iceberg. So instead of this StatDone hack we allow QItems to be
owned by multiple items and notify all owners about everything now,
so that for the point of each item they got it downloaded just for them.
|
|
We autocreate for a while now the last two directories in /var/lib/apt/lists
(similar for /var/cache/apt/archives) which is very nice for systems having
any of those on tmpfs or other non-persistent storage. This also means
though that this creation is effected by the default umask, so for
people with aggressive umasks like 027 the directories will be created
with 750, which means all non-root users are left out, which is usually
exactly what we want then this umask is set, but the cache and lib
directories contain public knowledge. There isn't any need to protect
them from viewers and they render apt completely useless if not
readable.
|
|
Do the same with less code in apt-get. This especially ensures that the
lock file (and the parent directories) exist before we are trying to
lock. It also means that clean now creates the directories if they are
missing so we returned to a proper clean state now.
Git-Dch: Ignore
|
|
partial files are chowned by the Item baseclass to let the methods work
with them. Now, this baseclass is also responsible for chowning the
files back to root instead of having various deeper levels do this.
The consequence is that all overloaded Failed() methods now call the
Item::Failed base as their first step. The same is done for Done().
The effect is that even in partial files usually don't belong to
_apt anymore, helping sneakernets and reducing possibilities of a bad
method modifying files not belonging to them.
The change is supported by the framework not only supporting being run
as root, but with proper permission management, too, so that privilege
dropping can be tested with them.
|
|
Git-Dch: Ignore
|
|
Ensure in SetupAPTPartialDirectory() that the /etc/apt/auth.conf file
can be read by the priv sep apt methods.
|
|
I am pretty sure I did that before committing broken stuff…
Git-Dch: Ignore
|
|
Do not drop privileges in the methods when using a older version of
libapt that does not support the chown magic in partial/ yet. To
do this DropPrivileges() now will ignore a empty Apt::Sandbox::User.
Cleanup all hardcoded _apt along the way.
|
|
Reimplementing an inline method is opening a can of worms we don't want
to open if we ever want to us a d-pointer in those classes, so we do the
only thing which can save us from hell: move the destructors into the cc
sources and we are good.
Technically not an ABI break as the methods inline or not do the same
(nothing), so a program compiled against the old version still works
with the new version (beside that this version is still in experimental,
so nothing really has been build against this library anyway).
Git-Dch: Ignore
|
|
Reworks the API involved in creating and setting up the fetcher to be a
bit more pleasent to look at and work with as e.g. an empty string for
no lock isn't very nice. With the lock we can also stop creating all our
partial directories "just in case". This way we can also be a bit more
aggressive with the partial directory itself as with a lock, we know we
will gone need it.
|
|
feature/acq-trans
Conflicts:
apt-pkg/acquire-item.cc
apt-pkg/acquire-item.h
methods/gpgv.cc
|
|
apt-get download and changelog as well as apt-helper reuse the acquire
system for their own proposes without requiring the directories the
fetcher wants to create, which is a problem if you run them as non-root
and the directories do not exist as it greets you with:
E: Archives directory /var/cache/apt/archives/partial is missing. -
Acquire (13: Permission denied)
Closes: 762898
|
|
Reported-By: cppcheck
Git-Dch: Ignore
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Conflicts:
apt-pkg/deb/deblistparser.cc
doc/po/apt-doc.pot
doc/po/de.po
doc/po/es.po
doc/po/fr.po
doc/po/it.po
doc/po/ja.po
doc/po/pl.po
doc/po/pt.po
doc/po/pt_BR.po
po/da.po
po/mr.po
po/vi.po
|
|
Having "/" here is most likely a user configuration error and
may cause removal of import symlinks like /vmlinuz
Closes: #753531
|
|
|
|
for both items and bytes
|
|
with the correct extension
|
