| Age | Commit message (Collapse) | Author |
|---|
|
Multiple targets downloading the same file is bad™ as it leads us to all
sorts of problems like the acquire system breaking or simply a problem
of which settings to use for them. Beside that this is most likely a
mistake and silently ignoring it doesn't help the user realizing his
mistake…
On the other hand, we have 'duplicates' which are 'created' by how we
create indextargets, so we have to prevent those from being created to
but do not emit a warning for them as this is an implementation detail.
And then, there is the absolute and most likely user mistake: Having the
same target(s) activated in multiple entries.
|
|
|
|
Some targets like Contents-udeb are special-needs targets. Shipping the
configuration snippet for them is okay, but they shouldn't be downloaded
by default. Forcing the user to enable targets by uncommenting targets
is wrong and this would still not really solve the problem completely as
even if you want to download some -udebs it will probably not be for all
sources you have enabled, so having the possibility of disabling a
target by default, but giving the user the option to enable it on a
per-source entry basis is better.
|
|
Nobody knows what makes the 'unable to optimize loop' warning to appear
in the sourceslist minus-options parsing, especially if we use a foreach
loop, but we can replace it with some nice c++11 algorithm+lambda usage,
which also helps in making even clearer what happens here.
And as this would be a lonely change, lets do it for a few more loops as
well where I might or might not have seen the warning at some point in
time, too.
Git-Dch: Ignore
|
|
Some additional files like 'Contents' are very big and should therefore
kept compressed on the disk, which apt-file did in the past. It also
implemented pdiff patching of these files by un- and recompressing these
files on-the-fly, with this commit we can do the same – but we can do
this in both pdiff patching styles (client and server merging) and
secured by hashes.
Hashes are in so far slightly complicated as we can't compare the hashes
of the compressed files as we might compress them differently than the
server would (different compressor versions, options, …), so we must
compare the hashes of the uncompressed content.
While this commit has changes in public headers, the classes it changes
are marked as hidden, so nobody can use them directly, which means the
ABI break is internal only.
|
|
Disabling pdiffs can be useful occasionally, like if you have a fast
local mirror where the download doesn't matter, but still want to use it
for non-local mirrors. Also, some users might prefer it to only use it
for very big indextargets like Contents.
|
|
This could allow an attacker to mark a package as installed in a
remote package index, as long as the package was not listed in
the dpkg status file.
This way, an attacker could force the installation of a package
during a dist-upgrade, by providing two packages in an index,
an older marked as installed, and a newer - apt would "upgrade"
to the newer version.
|
|
While Target{,-Add,-Remove} is available for configuring IndexTargets
already, allow Targets to be mentioned explicitely as yes/no options as
well, so that the Target 'Contents' can be disabled via 'Contents: no'
as well as 'Target-Remove: Contents'.
|
|
Sometimes too much refactoring can have bad effects.
Thanks: Niels Thykier for reporting on IRC
Git-Dch: Ignore
|
|
Git-Dch: ignore
|
|
|
|
Support for that variable was removed in dpkg in 1.15.6, in commit
6f037003e8b96878b485efb7cbd1f846e3bf4e97.
Closes: #765366
|
|
Gbp-Dch: ignore
Reported-By: g++ -Wsuggest-override
Thanks: g++ -Wsuggest-override
|
|
Git-Dch: Ignore
|
|
We archieve the same without the special handling now, so drop this code.
Makes supporting this abdomination a little longer bearable as well.
Git-Dch: Ignore
|
|
Now that we can dynamically create dependencies and provides as needed
rather than requiring to know with which architectures we will deal
before running we can allow the listparser to parse all records rather
than skipping records of "unknown" architectures.
This can e.g. happen if a user has foreign architecture packages in his
status file without dpkg knowing about this architecture (or apt
configured in this way).
A sideeffect is that now arch:all packages are (correctly) recorded as
available from any Packages file, not just from the native one – which
has its downsides for the resolver as mixed-arch source packages can
appear in different architectures at different times, but that is the
problem of the resolver and dealing with it in the parser is at best a
hack (and also depends on a helpful repository).
Another sideeffect is that his allows :none packages to appear in
Packages files again as we don't do any kind of checks now, but given
that they aren't really supported (anymore) by anyone we can live with
that.
|
|
Trade deduplication of code for a bunch of new virtuals, so it is
actually visible how the different indexes behave cleaning up the
interface at large in the process.
Git-Dch: Ignore
|
|
Sources are usually defined in sources.list (and co) and are pretty
stable, but once in a while a frontend might want to add an additional
"source" like a local .deb file to install this package (No support for
'real' sources being added this way as this is a multistep process).
We had a hack in place to allow apt-get and apt to pull this of for a
short while now, but other frontends are either left in the cold by this
and/or the code for it looks dirty with FIXMEs plastering it and has on
top of this also some problems (like including these 'volatile' sources
in the srcpkgcache.bin file).
So the biggest part in this commit is actually the rewrite of the cache
generation as it is now potentially a three step process. The biggest
problem with adding support now through is that this makes a bunch of
previously mostly unusable by externs and therefore hidden classes
public, so a bit of further tuneing on this now public API is in order…
|
|
Now that we deal with provides in a more dynamic fashion the last
remaining problem is explicit dependencies like 'Conflicts: foo' which
have to apply to all architectures, but creating them all at the same
time requires us to know all architectures ending up in the cache which
isn't needed to be the same set as all foreign architectures.
The effect is visible already now through as this prevents the creation
of a bunch of virtual packages for arch:all packages and as such also
many dependencies, just not very visible if you don't look at the stats…
Git-Dch Ignore
|
|
Expecting the worst is easy to code, but has its disadvantages e.g.
by creating package structures which otherwise would have never
existed. By creating the provides instead at the time a package
structure is added we are well prepared for the introduction of partial
architectures, massive amounts of M-A:foreign (and :allowed) and co as
far as provides are concerned at least. We have something relatively
similar for dependencies already.
Many tests are added for both M-A states and the code cleaned to
properly support implicit provides for foreign architectures and
architectures we 'just' happen to parse.
Git-Dch: Ignore
|
|
Before MultiArch implicits weren't a thing, so they were hidden by
default by definition. Adding them for MultiArch solved many problems,
but having no reliable way of detecting which dependency (and provides)
is implicit or not causes problems everytime we want to output
dependencies without confusing our observers with unneeded
implementation details.
The really notworthy point here is actually that we keep now a better
record of how a dependency came to be so that we can later reason about
it more easily, but that is hidden so deep down in the library internals
that change is more the problems it solves than the change itself.
|
|
We aren't and we will not be really compatible again with the previous
stable abi, so lets drop these markers (which never made it into a
released version) for good as they have outlived their intend already.
Git-Dch: Ignore
|
|
DepCache functions are called a lot, so if we can squeeze some drops out
of them for free we should do so. Takes also the opportunity to remove
some whitespace errors from these functions.
Git-Dch: Ignore
|
|
'files' is a bit too generic as a name for a command usually only used
programmatically (if at all) by developers, so instead of "wasting" this
generic name for this we use "indextargets" which is actually the name
of the datastructure the displayed data is stored in.
Along with this rename the config options are renamed accordingly.
|
|
C++11 adds the 'override' specifier to mark that a method is overriding
a base class method and error out if not. We hide it in the APT_OVERRIDE
macro to ensure that we keep compiling in pre-c++11 standards.
Reported-By: clang-modernize -add-override -override-macros
Git-Dch: Ignore
|
|
There is an option to keep all targets (Packages, Sources, …) compressed
for a while now, but the all-or-nothing approach is a bit limited for
our purposes with additional targets as some of them are very big
(Contents) and rarely used in comparison, so keeping them compressed by
default can make sense, while others are still unpacked.
Most interesting is the copy-change maybe: Copy is used by the acquire
system as an uncompressor and it is hence expected that it returns the
hashes for the "output", not the input. Now, in the case of keeping a
file compressed, the output is never written to disk, but generated in
memory and we should still validated it, so for compressed files copy is
expected to return the hashes of the uncompressed file. We used to use
the config option to enable on-the-fly decompress in the method, but in
reality copy is never used in a way where it shouldn't decompress a
compressed file to get its hashes, so we can save us the trouble of
sending this information to the method and just do it always.
|
|
Limits which key(s) can be used to sign a repository. Not immensely useful
from a security perspective all by itself, but if the user has
additional measures in place to confine a repository (like pinning) an
attacker who gets the key for such a repository is limited to its
potential and can't use the key to sign its attacks for an other (maybe
less limited) repository… (yes, this is as weak as it sounds, but having
the capability might come in handy for implementing other stuff later).
|
|
These options could be set via configuration before, but the connection
to the actual sources is so strong that they should really be set in the
sources.list instead – especially as this can be done a lot more
specific rather than e.g. disabling Valid-Until for all sources at once.
Valid-Until-* names are chosen instead of the Min/Max-ValidTime as this
seems like a better name and their use in the wild is probably low
enough that this isn't going to confuse anyone if we have to names for
the same thing in different areas.
In the longrun, the config options should be removed, but for now
documentation hinting at the new options is good enough as these are the
kind of options you set once across many systems with different apt
versions, so the new way should work everywhere first before we
deprecate the old way.
|
|
indexRecords was used to parse the Release file – mostly the hashes –
while metaIndex deals with downloading the Release file, storing all
indexes coming from this release and … parsing the Release file, but
this time mostly for the other fields.
That wasn't a problem in metaIndex as this was done in the type specific
subclass, but indexRecords while allowing to override the parsing method
did expect by default a specific format.
APT isn't really supporting different types at the moment, but this is
a violation of the abstraction we have everywhere else and, which is the
actual reason for this merge: Options e.g. coming from the sources.list
come to metaIndex naturally, which needs to wrap them up and bring them
into indexRecords, so the acquire system is told about it as they don't
get to see the metaIndex, but they don't really belong in indexRecords
as this is just for storing data loaded from the Release file… the
result is a complete mess.
I am not saying it is a lot prettier after the merge, but at least
adding new options is now slightly easier and there is just one place
responsible for parsing the Release file. That can't hurt.
|
|
A specific trust state can be enforced via a sources.list option, but it
effects all entries handled by the same Release file, not just the entry
it was given on so we enforce acknowledgement of this by requiring the
same value to be (not) set on all such entries.
|
|
We support arch= for a while, now we finally add lang= as well and as a
first simple way of controlling which targets to acquire also target=.
This asked for a redesign of the internal API of parsing and storing
information about 'deb' and 'deb-src' lines. As this API isn't visible
to the outside no damage done through.
Beside being a nice cleanup (= it actually does more in less lines) it
also provides us with a predictable order of architectures as provides
in the configuration rather than based on string sorting-order, so that
now the native architecture is parsed/displayed first. Observeable e.g.
in apt-get output.
|
|
Various small leaks here and there. Nothing particularily big, but still
good to fix. Found by the sanitizers while running our testcases.
Reported-By: gcc -fsanitize
Git-Dch: Ignore
|
|
Doing this disables the implicit copy assignment operator (among others)
which would cause hovac if used on the classes as it would just copy the
pointer, not the data the d-pointer points to. For most of the classes
we don't need a copy assignment operator anyway and in many classes it
was broken before as many contain a pointer of some sort.
Only for our Cacheset Container interfaces we define an explicit copy
assignment operator which could later be implemented to copy the data
from one d-pointer to the other if we need it.
Git-Dch: Ignore
|
|
Some of them modify the ABI, but given that we prepare a big one
already, these few hardly count for much.
Git-Dch: Ignore
|
|
To have a chance to keep the ABI for a while we need all three to team
up. One of them missing and we might loose, so ensuring that they are
available is a very tedious but needed task once in a while.
Git-Dch: Ignore
|
|
Translation-* files are internally handled as PackageFiles which isn't
super nice, but giving them their own struct is a bit overkill so let it
be for the moment. They always appeared in the policy output because of
this through and now that they are properly linked to a ReleaseFile they
even display all the pinning information on them, but they don't contain
any packages which could be pinned… No problem, but useless and
potentially confusing output.
Adding a 'NoPackages' flag which can be set on those files and be used
in applications seems like a simple way to fix this display issue.
|
|
This is mainly visible in the policy, so that you can now pin by b= and
let it only effect Packages files of this architecture and hence the
packages coming from it (which do not need to be from this architecture,
but very likely are in a normal repository setup).
If you should pin by architecture in this way is a different question…
Closes: 687255
|
|
Selecting targets based on the Release they belong to isn't to
unrealistic. In fact, it is assumed to be the most used case so it is
made the default especially as this allows to bundle another thing we
have to be careful with: Filenames and only showing targets we have
acquired.
Closes: 752702
|
|
We used to read the Release file for each Packages file and store the
data in the PackageFile struct even through potentially many Packages
(and Translation-*) files could use the same data. The point of the
exercise isn't the duplicated data through. Having the Release files as
first-class citizens in the Cache allows us to properly track their
state as well as allows us to use the information also for files which
aren't in the cache, but where we know to which Release file they
belong (Sources are an example for this).
This modifies the pkgCache structs, especially the PackagesFile struct
which depending on how libapt users access the data in these structs can
mean huge breakage or no visible change. As a single data point:
aptitude seems to be fine with this. Even if there is breakage it is
trivial to fix in a backportable way while avoiding breakage for
everyone would be a huge pain for us.
Note that not all PackageFile structs have a corresponding ReleaseFile.
In particular the dpkg/status file as well as *.deb files have not. As
these have only a Archive property need, the Component property takes
over this duty and the ReleaseFile remains zero. This is also the reason
why it isn't needed nor particularily recommended to change from
PackagesFile to ReleaseFile blindly. Sticking with the earlier is
usually the better option.
|
|
Downloading additional files is only half the job. We still need a way
to allow external tools to know where the files are they requested for
download given that we don't want them to choose their own location.
'apt-get files' is our answer to this showing by default in a deb822
format information about each IndexTarget with the potential to filter
the records based on lines and an option to change the output format.
The command serves also as an example on how to get to this information
via libapt.
|
|
Strings are easy to typo and we can keep the extensibility we require
here with a simple enum we can append to without endangering ABI.
Git-Dch: Ignore
|
|
Removes a bunch of duplicated code in the deb-specific parts. Especially
the Description part is now handled centrally by IndexTarget instead of
being duplicated to the derivations of IndexFile.
Git-Dch: Ignore
|
|
It is a rather strange sight that index items use SiteOnly which strips
the Path, while e.g. deb files are downloaded with NoUserPassword which
does not. Important to note here is that for the file transport Path is
pretty important as there is no Host which would be displayed by Site,
which always resulted in "interesting" unspecific errors for "file:".
Adding a 'middle' ground between the two which does show the Path but
potentially modifies it (it strips a pending / at the end if existing)
solves this "file:" issue, syncs the output and in the end helps to
identify which file is meant exactly in progress output and co as a
single site can have multiple repositories in different paths.
|
|
We need a general way to get from a sources.list entry to IndexTargets
and with this change we can move from pkgSourceList over the list of
metaIndexes it includes to the IndexTargets each metaIndex can have.
Git-Dch: Ignore
|
|
Creating and passing around a bunch of pointers of IndexTargets (and of
a vector of pointers of IndexTargets) is probably done to avoid the
'costly' copy of container, but we are really not in a timecritical
operation here and move semantics will help us even further in the
future. On the other hand we never do a proper cleanup of these
pointers, which is very dirty, even if structures aren't that big…
The changes will effecting many items only effect our own hidden class,
so we can do that without fearing breaking interfaces or anything.
Git-Dch: Ignore
|
|
We still need an API for the targets, so slowly prepare the IndexTargets
to let them take this job.
Git-Dch: Ignore
|
|
We have two places in the code which need to iterate over targets and do
certain things with it. The first one is actually creating these targets
for download and the second instance pepares certain targets for
reading.
Git-Dch: Ignore
|
|
First pass at making the acquire system capable of downloading files
based on configuration rather than hardcoded entries. It is now possible
to instruct 'deb' and 'deb-src' sources.list lines to download more than
just Packages/Translation-* and Sources files. Details on how to do that
can be found in the included documentation file.
|
|
Having every item having its own code to verify the file(s) it handles
is an errorprune process and easy to break, especially if items move
through various stages (download, uncompress, patching, …). With a giant
rework we centralize (most of) the verification to have a better
enforcement rate and (hopefully) less chance for bugs, but it breaks the
ABI bigtime in exchange – and as we break it anyway, it is broken even
harder.
It shouldn't effect most frontends as they don't deal with the acquire
system at all or implement their own items, but some do and will need to
be patched (might be an opportunity to use apt on-board material).
The theory is simple: Items implement methods to decide if hashes need to
be checked (in this stage) and to return the expected hashes for this
item (in this stage). The verification itself is done in worker message
passing which has the benefit that a hashsum error is now a proper error
for the acquire system rather than a Done() which is later revised to a
Failed().
|
|
In 66c3875df391b1120b43831efcbe88a78569fbfe we workaround/fixed a
problem where the code makes the assumption that the compiler uses
copy-on-write implementations for std::string. Turns out that for c++11
compatibility gcc >= 5 will stop doing this by default.
|
