| Age | Commit message (Collapse) | Author |
|---|
|
Thanks: Thomas Reusch
|
|
Git-Dch: Ignore
|
|
Architectures for packages which do not belong to the native nor a
foreign architecture (dubbed barbarian for now) which are marked
M-A:foreign still provide in their own architecture even if not for
others. Also, other M-A:foreign (and allowed) packages provide in these
barbarian architectures.
|
|
I overlooked this
Gbp-Dch: ignore
|
|
Downloading and storing are two different operations were different
compression types can be preferred. For downloading we provide the
choice via Acquire::CompressionTypes::Order as there is a choice to
be made between download size and speed – and limited by whats available
in the repository.
Storage on the other hand has all compressions currently supported by
apt available and to reduce runtime of tools accessing these files the
compression type should be a low-cost format in terms of decompression.
apt traditionally stores its indexes uncompressed on disk, but has
options to keep them compressed. Now that apt downloads additional files
we also deal with files which simply can't be stored uncompressed as
they are just too big (like Contents for apt-file). Traditionally they
are downloaded in a low-cost format (gz) as repositories do not provide
other formats, but there might be even lower-cost formats and for
download we could introduce higher-cost in the repositories.
Downloading an entire index potentially requires recompression to
another format, so an update takes potentially longer – but big files
are usually updated via pdiffs which has to de- and re-compress anyhow
and does it on the fly anyhow, so there is no extra time needed and in
general it seems to be benefitial to invest the time in update to save
time later on file access.
|
|
Do not create strings within the loop, that creates one string
per language and does more work than needed. Instead, reserve
enough space at the beginning and assign the prefix, and then
resize and append inside the loop.
Also call exists with the string itself instead of the c_str(),
this means that the lookup uses the size information in the
string now and does not have to call strlen() on it.
|
|
This improves performance, as we now can ignore unequal strings
based on their length already.
Gbp-Dch: ignore
|
|
This improves performance of the cache generation on my
ARM platform (4x Cortex A15) by about 10% to 20% from
2.35-2.50 to 2.1 seconds.
|
|
Now (55153bf94ff28a23318e79aa48242244c4d82b3c) that pkgTagFile can be
told to deal with all sorts of comments we can use this mode to parse
dsc (as by catch) and debian/control files properly even in the wake of
multiline fields spliced with comments like Build-Depends.
Closes: 806775
|
|
Debian has a Packages file for arch:all already, but the arch:any files
contain arch:all packages as well, so downloading it would be a total
waste of resources. Getting this solved is on the list of things to do,
but it is also the hardest part – for index targets like Contents the
situation is much easier and less server/client implementations are
involved so we might not want to stall them.
A repository can now declare via:
No-Support-for-Architecture-all: Packages
that even if an arch:all Packages exists, it shouldn't be downloaded, so
that support for Contents files can be added now.
See also 1dd20368486820efb6ef4476ad739e967174bec4 for the implementation
of downloading arch:all index targets, which this is limiting.
The field uses the name of the target from the apt configuration for
simplicity and is negative by design as this field is intended to be
supported/needed only for a "short" time (one or two Debian releases).
While this commit theoretically supports any target, its expected to
only see "Packages" as a value in reality.
|
|
We do not see those branches at all during normal mode of
operation (that is, during cache generation), so tell the
compiler about it.
|
|
The Set() method returns false if the input is no hex number,
so simply use that.
|
|
This makes the code parsing architecture lists slower, but on
the other hand, improves the more generic case of reading
dependencies from Packages files.
|
|
This converts all callers that read machine-generated data,
callers that might work with user input are not converted.
|
|
If we can't work with the hashes we parsed from the Release file we
display now an error message if the Release file includes only weak
hashes instead of downloading the indexes and failing to verify them
with "Hash Sum mismatch" even through the hashes didn't mismatch (they
were just weak).
If for some (unlikely) reason we have got weak hashes only for
individual targets we will show a warning to this effect (again, befor
downloading and failing the index itself).
Closes: 806459
|
|
dpkg does that when reading package files, so we should do
the same. This only deals with parsing names from binary
package paragraphs, it does not look at source package names
and/or the list of binaries in a dsc file.
Closes: #807012
|
|
We need to pass 0llu instead of 0 as the init value, otherwise
std::accumulate will calculate with ints.
Reported-by: Raphaël Hertzog
|
|
Otherwise a user is subject to unexpected content-injection depending on
which directory she happens to start apt in. This also cleans up the code
requiring less implementation details in build-dep which is always good.
Technically, this is an ABI break as we override virtual methods, but
that they weren't overridden was a mistake resulting in pure classes,
which shouldn't be pure, so they were unusable – and as they are new in
1.1 nobody is using them yet (and hopefully ever as they are borderline
implementation details).
Closes: 806693
|
|
Git-Dch: Ignore
|
|
Reference mail:
https://lists.debian.org/debian-l10n-english/2015/11/msg00006.html
|
|
Reported-By: cppcheck
Git-Dch: Ignore
|
|
A slightly unlikely bug, but lets fix it while slightly reworking this
whole function to be slightly saner to look at, even if still not good.
Git-Dch: Ignore
|
|
Unlinking /dev/null is bad, we shouldn't do that. Also, we should print
at least a warning if we tried to unlink a file but didn't manage to
pull it of (ignoring the case were the file is /dev/null or doesn't
exist in the first place).
This got triggered by a relatively unlikely to cause problem in
pkgAcquire::Worker::PrepareFiles which would while temporary
uncompressed files (which are set to keep compressed) figure out that to
files are the same and prepare for sharing by deleting them. Bad move.
That also shows why not printing a warning is a bad idea as this hide
the error for in non-root test runs.
Git-Dch: Ignore
|
|
Leading or trailing newlines can be confusing for our parser as it
expects two newlines to start/stop a new stanza. To solve this the lines
we wanna add are printed first, ignore any leading newlines and then add
the stanza as provided by dpkg-deb with or without trailing newlines as
the parser will look at the first stanza only anyway and removing
trailing newlines is considerably harder to do.
Closes: 802553
|
|
Based on a discussion with Niels Thykier who asked for Contents-all this
implements apt trying for all architecture dependent files to get a file
for the architecture all, which is treated internally now as an official
architecture which is always around (like native). This way arch:all
data can be shared instead of duplicated for each architecture requiring
the user to download the same information again and again.
There is one problem however: In Debian there is already a binary-all/
Packages file, but the binary-any files still include arch:all packages,
so that downloading this file now would be a waste of time, bandwidth
and diskspace. We therefore need a way to decide if it makes sense to
download the all file for Packages in Debian or not. The obvious answer
would be a special flag in the Release file indicating this, which would
need to default to 'no' and every reasonable repository would override
it to 'yes' in a few years time, but the flag would be there "forever".
Looking closer at a Release file we see the field "Architectures", which
doesn't include 'all' at the moment. With the idea outlined above that
'all' is a "proper" architecture now, we interpret this field as being
authoritative in declaring which architectures are supported by this
repository. If it says 'all', apt will try to get all, if not it will be
skipped. This gives us another interesting feature: If I configure a
source to download armel and mips, but it declares it supports only
armel apt will now print a notice saying as much. Previously this was a
very cryptic failure. If on the other hand the repository supports mips,
too, but for some reason doesn't ship mips packages at the moment, this
'missing' file is silently ignored (= that is the same as the repository
including an empty file).
The Architectures field isn't mandatory through, so if it isn't there,
we assume that every architecture is supported by this repository, which
skips the arch:all if not listed in the release file.
|
|
This was discussed a while ago on #debian-apt and now that I see myself
making this mistake lets bite the bullet and fix it in the easy way out
version: Using a new name which fits with a similar named setter and
deprecate the old method instead of 'hostily' changing API.
Closes: #803471
|
|
Showing just "Get: [1234 B]" looks very strange, so we now print the
filename and as usual the package name, version and architecture.
|
|
We had this code lying around in apt-mark for a while now, but other
frontends need this (and similar) functionality as well, so its high
time that we provide a public interface in libapt for this stuff.
|
|
We have a few places and there will be a few more still where we have to
call dpkg to detect/set certain features or settings. Centralizing the
calling infrastructure now seems like a good idea before we add another.
|
|
|
|
Some codepaths need to check if the system (in our case usually dpkg)
supports MultiArch or not. We had copy-pasted the check so far into
these paths, but having it as a system check is better for reusability.
|
|
The former is not thread-safe, whereas the latter is.
Gbp-Dch: ignore
|
|
This function only exists on a limited number of platforms, so
we add a configure check to make sure it exists.
Gbp-Dch: ignore
|
|
ctime() is not thread-safe, ctime_r() is.
Gbp-Dch: ignore
|
|
More safety, less writeable memory.
|
|
This changes the semantics of the option (which is renamed too) to be a
yes/no value with the special additional value "force" as this allows
by-hash to be disabled even if the repository indicates it would be
supported and is more in line with our other yes/no options like pdiff
which disable themselves if no support can be detected.
The feature wasn't documented so far and hasn't reached a (un)stable
release yet, so changing it without trying too hard to keep
compatibility seems okay.
|
|
Our error reporting is historically grown into some kind of mess.
A while ago I implemented stacking for the global error which is used in
this commit now to wrap calls to functions which do not report (all)
errors via return, so that only failures in those calls cause a failure
to propergate down the chain rather than failing if anything
(potentially totally unrelated) has failed at some point in the past.
This way we can avoid stopping the entire acquire process just because a
single source produced an error for example. It also means that after
the acquire process the cache is generated – even if the acquire
process had failures – as we still have the old good data around we can and
should generate a cache for (again).
There are probably more instances of this hiding, but all these looked
like the easiest to work with and fix with reasonable (aka net-positive)
effects.
|
|
How the Multi-Arch field and pkg:<arch> dependencies interact was
discussed at DebConf15 in the "MultiArch BoF". dpkg and apt (among other
tools like dose) had a different interpretation in certain scenarios
which we resolved by agreeing on dpkg view – and this commit realizes
this agreement in code.
As was the case so far libapt sticks to the idea of trying to hide
MultiArch as much as possible from individual frontends and instead
translates it to good old SingleArch. There are certainly situations
which can be improved in frontends if they know that MultiArch is upon
them, but these are improvements – not necessary changes needed
to unbreak a frontend.
The implementation idea is simple: If we parse a dependency on foo:amd64
the dependency is formed on a package 'foo:amd64' of arch 'any'. This
package is provided by package 'foo' of arch 'amd64', but not by 'foo'
of arch 'i386'. Both of those foo packages provide each other through
(assuming foo is M-A:foreign) to allow a dependency on 'foo' to be
satisfied by either foo of amd64 or i386. Packages can also declare to
provide 'foo:amd64' which is translated to providing 'foo:amd64:any' as
well.
This indirection over provides was chosen as the alternative would be to
teach dependency resolvers how to deal with architecture specific
dependencies – which violates the design idea of avoiding resolver
changes, especially as architecture-specific dependencies are a
cornercase with quite a few subtil rules. Handling it all over versioned
provides as we already did for M-A in general seems much simpler as it
just works for them.
This switch to :any has actually a "surprising" benefit as well: Even
frontends showing a package name via .Name() [which doesn't show the
architecture] will display the "architecture" for dependencies in which
it was explicitely requested, while we will not show the 'strange' :any
arch in FullName(true) [= pretty-print] either. Before you had to
specialcase these and by default you wouldn't get these details shown.
The only identifiable disadvantage is that this complicates error
reporting and handling. apt-get's ShowBroken has existing problems with
virtual packages [it just shows the name without any reason], so that
has to be worked on eventually. The other case is that detecting if a
package is completely unknown or if it was at least referenced somewhere
needs to acount for this "split" – not that it makes a practical
difference which error is shown… but its one of the improvements
possible.
|
|
We parse all architectures we encounter recently, which means we also
parse packages from architectures which are neither native nor foreign,
but still came onto the system somehow (usually via heavy force).
|
|
Previously we had python:any:amd64, python:any:i386, … in the cache and
the dependencies of an amd64 package would be on python:any:amd64, of an
i386 on python:any:i386 and so on. That seems like a relatively
pointless endeavor given that they will all be provided by the same
packages and therefore also a waste of space.
Git-Dch: Ignore
|
|
Reported-By: gcc
Git-Dch: Ignore
|
|
Multiple targets downloading the same file is bad™ as it leads us to all
sorts of problems like the acquire system breaking or simply a problem
of which settings to use for them. Beside that this is most likely a
mistake and silently ignoring it doesn't help the user realizing his
mistake…
On the other hand, we have 'duplicates' which are 'created' by how we
create indextargets, so we have to prevent those from being created to
but do not emit a warning for them as this is an implementation detail.
And then, there is the absolute and most likely user mistake: Having the
same target(s) activated in multiple entries.
|
|
|
|
Some targets like Contents-udeb are special-needs targets. Shipping the
configuration snippet for them is okay, but they shouldn't be downloaded
by default. Forcing the user to enable targets by uncommenting targets
is wrong and this would still not really solve the problem completely as
even if you want to download some -udebs it will probably not be for all
sources you have enabled, so having the possibility of disabling a
target by default, but giving the user the option to enable it on a
per-source entry basis is better.
|
|
Nobody knows what makes the 'unable to optimize loop' warning to appear
in the sourceslist minus-options parsing, especially if we use a foreach
loop, but we can replace it with some nice c++11 algorithm+lambda usage,
which also helps in making even clearer what happens here.
And as this would be a lonely change, lets do it for a few more loops as
well where I might or might not have seen the warning at some point in
time, too.
Git-Dch: Ignore
|
|
Some additional files like 'Contents' are very big and should therefore
kept compressed on the disk, which apt-file did in the past. It also
implemented pdiff patching of these files by un- and recompressing these
files on-the-fly, with this commit we can do the same – but we can do
this in both pdiff patching styles (client and server merging) and
secured by hashes.
Hashes are in so far slightly complicated as we can't compare the hashes
of the compressed files as we might compress them differently than the
server would (different compressor versions, options, …), so we must
compare the hashes of the uncompressed content.
While this commit has changes in public headers, the classes it changes
are marked as hidden, so nobody can use them directly, which means the
ABI break is internal only.
|
|
Disabling pdiffs can be useful occasionally, like if you have a fast
local mirror where the download doesn't matter, but still want to use it
for non-local mirrors. Also, some users might prefer it to only use it
for very big indextargets like Contents.
|
|
This could allow an attacker to mark a package as installed in a
remote package index, as long as the package was not listed in
the dpkg status file.
This way, an attacker could force the installation of a package
during a dist-upgrade, by providing two packages in an index,
an older marked as installed, and a newer - apt would "upgrade"
to the newer version.
|
|
While Target{,-Add,-Remove} is available for configuring IndexTargets
already, allow Targets to be mentioned explicitely as yes/no options as
well, so that the Target 'Contents' can be disabled via 'Contents: no'
as well as 'Target-Remove: Contents'.
|
|
Sometimes too much refactoring can have bad effects.
Thanks: Niels Thykier for reporting on IRC
Git-Dch: Ignore
|
