| Age | Commit message (Collapse) | Author |
|---|
|
Having every item having its own code to verify the file(s) it handles
is an errorprune process and easy to break, especially if items move
through various stages (download, uncompress, patching, …). With a giant
rework we centralize (most of) the verification to have a better
enforcement rate and (hopefully) less chance for bugs, but it breaks the
ABI bigtime in exchange – and as we break it anyway, it is broken even
harder.
It shouldn't effect most frontends as they don't deal with the acquire
system at all or implement their own items, but some do and will need to
be patched (might be an opportunity to use apt on-board material).
The theory is simple: Items implement methods to decide if hashes need to
be checked (in this stage) and to return the expected hashes for this
item (in this stage). The verification itself is done in worker message
passing which has the benefit that a hashsum error is now a proper error
for the acquire system rather than a Done() which is later revised to a
Failed().
|
|
The change itself is no problem ABI wise, but the remove of the old
undynamic hashtables is, so we bring it back for older abis and happily
use the now available free space to backport more recent additions like
the dynamic hashtable itself.
Git-Dch: Ignore
|
|
Git-Dch: Ignore
|
|
We are the only possible users of private methods, so we are also the
only users who can potentially export them via using them in inline
methods. The point is: We don't need these symbols exported if we don't
do this, so marking them as hidden removes some methods from the API
without breaking anything as nobody could have used them.
Git-Dch: Ignore
|
|
Accessing the package records to acquire this information is pretty
costly, so that information wasn't used so far in many places. The most
noticeable user by far is EDSP at the moment, but there are ideas to
change that which this commit tries to enable.
|
|
Strings like Section names or architectures are needed vary often.
Instead of writing them each time we need them, we deploy sharing for
these special strings. Until now, this was done with a linked list of
strings in which we would search, which was stored in the cache.
It turns out we can do this just as well in memory as well with a bunch
of std::map's.
In memory means here that it isn't available anymore if we have a partly
invalid cache, but that isn't much of a problem in practice as the
status file is compared to the other files we parse very small and includes
mostly duplicates, so the space we would gain by storing is more or less
equal to the size of the stored linked list…
|
|
They both store the same information, so this field just takes up space
in the Package struct for no good reason. We mark it "just" as deprecated
instead of instantly removing it though as it isn't misleading like
Section was and is potentially used in the wild more often.
|
|
A version belongs to a section and has hence a section member of its
own. A package on the other hand can have multiple versions from
different sections. This was "solved" by using the section which was
parsed first as order of sources.list defines, but that is obviously a
horribly unpredictable thing.
We therefore directly remove this struct member to free some space and
mark the access method as deprecated, which is told to return the
section of the 'newest' known version, which is at least predictable,
but possible not what it returned before – but nobody knows.
Users are way better of with the Section() as returned by the version
they are dealing with. It is likely the same for all versions of a
package, but in the few cases it isn't, it is important (like packages
moving from main/* to contrib/* or into oldlibs …).
|
|
We had a wild mixture of (unsigned) int, long and long long here without
much sense, so this commit adds a few typedefs to get some sense in the
typesystem and ensures that a ID isn't sometimes computed as int, stored
as long and compared with a long long… as this could potentially bite us
later on as the size of the archive only increases over time.
|
|
It also makes the size configureable, so it can be adapted in the future
without the need for an abi break - and even by users…
The increase was long overdue as it gives a >10% decrease in runtime of
e.g. 'apt-get check -s'. Some (useless) benchmark with 69933 groups and
187796 packages without a pre-built cache:
time apt-get check -so APT::Cache-HashTableSize=1 → 20m
time apt-get check -so APT::Cache-HashTableSize=1000 → 6,41s
time apt-get check -so APT::Cache-HashTableSize=2000 → 5,64s (old)
time apt-get check -so APT::Cache-HashTableSize=3000 → 5,30s
time apt-get check -so APT::Cache-HashTableSize=5000 → 5,08s
time apt-get check -so APT::Cache-HashTableSize=6000 → 5,05s
time apt-get check -so APT::Cache-HashTableSize=7000 → 5,02s
time apt-get check -so APT::Cache-HashTableSize=8000 → 5,00s
time apt-get check -so APT::Cache-HashTableSize=9000 → 4,98s
time apt-get check -so APT::Cache-HashTableSize=10000 → 4,96s (new)
time apt-get check -so APT::Cache-HashTableSize=15000 → 4,90s
time apt-get check -so APT::Cache-HashTableSize=20000 → 4,86s
time apt-get check -so APT::Cache-HashTableSize=30000 → 4,77s
time apt-get check -so APT::Cache-HashTableSize=40000 → 4,74s
time apt-get check -so APT::Cache-HashTableSize=50000 → 4,73s
time apt-get check -so APT::Cache-HashTableSize=60000 → 4,71s
The gap increases further for operations which have more package
lookups. Factor 5 was chosen as higher values do not provide any
really significant timing advantage anymore compared to the memory
increase in my testing and there is always the possibility to increase
it now if that changes. (also most users will not have 3 releases and
4 architectures in the cache, so theirs will be much smaller and faster).
|
|
Conflicts:
apt-pkg/acquire-item.cc
apt-pkg/acquire-item.h
apt-pkg/deb/debmetaindex.h
apt-pkg/pkgcache.cc
test/integration/test-apt-ftparchive-src-cachedb
|
|
This is a internal struct not a external interface so the actual
breakage should be small.
|
|
|
|
The cache heavily depends on the architecture(s) it is build for,
especially if you move from single- to multiarch. Adding a new
architecture to dpkg therefore has to be detected and must invalidate
the cache so that we don't operate on incorrect data.
The incorrect data will prevent us from doing otherwise sensible
actions (it doesn't allow bad things to happen) and the recovery is
simple and automatic in most cases, so this hides pretty well and is
also not as serious as it might sound at first.
Closes: 745036
|
|
Conflicts:
apt-pkg/cachefilter.h
apt-pkg/contrib/fileutl.cc
apt-pkg/contrib/netrc.h
apt-pkg/deb/debsrcrecords.cc
apt-pkg/init.h
apt-pkg/pkgcache.cc
debian/apt.install.in
debian/changelog
|
|
Git-Dch: Ignore
Reported-By: gcc -Wsuggest-attribute={pure,const,noreturn}
|
|
Reported-By: gcc -Wignored-qualifiers
Git-Dch: Ignore
|
|
Git-Dch: Ignore
Reported-By: gcc -Wpedantic
|
|
Git-Dch: Ignore
|
|
Conflicts:
apt-private/private-list.cc
configure.ac
debian/apt.install.in
debian/changelog
|
|
|
|
- adjust pkgCache::State::VerPriority enum, to match reality
|
|
The breakage is just to big for now, so guard the change with
#ifndef APT_8_CLEANER_HEADERS and be nice to library users
|
|
- prefer native providers over foreigns even if the chain is foreign
The code preferred real over virtual packages and based on priorities.
This is changed in so far that a real package from any arch is preferred
over any virtual provider and if priorities doesn't help in choosing the
best provider we choose it based on architectures
|
|
|
|
|
|
- readd All{Foreign,Allowed} as suggested by Julian to
remain strictly API compatible
|
|
- clean up mess with the "all" handling in MultiArch to
fix LP: #733741 cleanly for everyone now
|
|
- use the native Architecture stored in the cache header instead of
loading it from configuration as suggested by Julian Andres Klode
|
|
- return "all" instead of native architecture without breaking the abi
(too much) by extending enum instead of using bitflags (LP: #733741)
With the next abi break that enum should be a char bitflag instead
|
|
Multi-Arch; instead, Arch: all packages only satisfy dependencies for
the native arch, except where the Arch: all package is declared
Multi-Arch: foreign. (Closes: #613584)
This has the sideeffect that arch:all packages internally show up as
coming from the native arch - so packages with the architecture "all"
doesn't exist any longer in the pkgcache
|
|
- support 100-pinning in Release file with ButAutomaticUpgrades
as requested by the backports crew (Closes: #596097)
* apt-pkg/deb/deblistparser.cc:
- overrule NotAutomatic in case of ButAutomaticUpgrades
|
|
- make the used MMap moveable (and therefore dynamic resizeable) by
applying (some) mad pointer magic (Closes: #195018)
|
|
- switch {,Install-}Size to unsigned long long
* apt-pkg/depcache.cc:
- deal with long long, not with int to remove 2GB Limit (LP: #250909)
|
|
- enhance the Groups ABI by providing a ID as the other structs does
- check also the size of the Group struct then checking for the others
|
|
- drop the file in favor of inplace documentation with doxygen
|
|
use it as a fallback if multiarch is not enabled. The effect is barly
noticeable but SingleArch is the realworld scenario.
|
|
Arch all packages are now represented by arch depending packages which
all depend on a package with the same name and the special arch "all".
This packages has NO dependencies, but beside this the same information.
It is the only package which has a size, the arch depending ones all
have a zero size. While the arch depending pseudo packages are used
for dependency resolution the arch "all" package is used for downloading
and ordering of the package.
|
|
the information from the parent package now
|
|
for creating the dependencies needed for our groups:
For now for all groups only one package can be installed at the same
time which conflicts with each other packages in the group.
The exceptions are architecture all package.
Also, the Multi-Arch field is now parsed, but not used for now.
|
|
infrastructor for packages.
APT is now aware of the fact that a package A in architecture X can't
satisfy a dependency on package A in architecture Y - to handle these
packages are now identified by name and architecture, so different
architectures of the same package are handled internally as completly
different packages. This is great for pinning, dependency checking and
in many other situations, but sometimes we need to know which archs are
available for a given package: Here Groups come to our rescue!
|
|
This should not change the public interface, but it removes the
friend connection between the iterators and pkgcache as it is unused.
|
|
patch from Víctor Manuel Jáquez Leal, thanks! (Closes: #538917)
|
|
|
|
|
|
|
|
* MatchType::Release checks first for archive than for codename equality
* new n= option in apt_preference to be able to pin based on a codeName
|
|
- fix TimeToStr i18n (LP: #289807)
* [ABI break] merge support for http redirects, thanks to
Jeff Licquia and Anthony Towns
* [ABI break] use int for the package IDs (thanks to Steve Cotton)
* apt-pkg/pkgcache.cc:
- do not run "dpkg --configure pkg" if pkg is in trigger-awaited
state (LP: #322955)
* methods/https.cc:
- add Acquire::https::AllowRedirect support
* Clarify the --help for 'purge' (LP: #243948)
* cmdline/apt-get.cc
- fix "apt-get source pkg" if there is a binary package and
a source package of the same name but from different
packages (LP: #330103)
* cmdline/acqprogress.cc:
- Call pkgAcquireStatus::Pulse even if quiet, so that we still get
dlstatus messages on the status-fd (LP: #290234).
|
|
|
|
* pulled in the other remaining ubuntu changes
|
