summaryrefslogtreecommitdiff
path: root/cmdline
AgeCommit message (Collapse)Author
2016-01-08keep compressed indexes in a low-cost formatDavid Kalnischkies
Downloading and storing are two different operations were different compression types can be preferred. For downloading we provide the choice via Acquire::CompressionTypes::Order as there is a choice to be made between download size and speed – and limited by whats available in the repository. Storage on the other hand has all compressions currently supported by apt available and to reduce runtime of tools accessing these files the compression type should be a low-cost format in terms of decompression. apt traditionally stores its indexes uncompressed on disk, but has options to keep them compressed. Now that apt downloads additional files we also deal with files which simply can't be stored uncompressed as they are just too big (like Contents for apt-file). Traditionally they are downloaded in a low-cost format (gz) as repositories do not provide other formats, but there might be even lower-cost formats and for download we could introduce higher-cost in the repositories. Downloading an entire index potentially requires recompression to another format, so an update takes potentially longer – but big files are usually updated via pdiffs which has to de- and re-compress anyhow and does it on the fly anyhow, so there is no extra time needed and in general it seems to be benefitial to invest the time in update to save time later on file access.
2016-01-07apt-helper: cat-file: Add -C/--compress optionJulian Andres Klode
This allows passing compressing the output. The compressor must be a compressor name, extension, or an extension without the leading dot.
2016-01-03apt-cache: stats: Show a table utilization as percentageJulian Andres Klode
Gbp-Dch: ignore
2016-01-03apt-cache: stats: Average is over used, not all, bucketsJulian Andres Klode
It does not make sense to consider empty buckets in the average, as they do not affect the lookup performance.
2015-12-28apt-helper: Use CopyFile() for concatenating the filesJulian Andres Klode
There's no point in keeping using yet another read-then-write loop. Gbp-Dch: ignore
2015-12-26apt-helper: Check that we can open stdoutJulian Andres Klode
Gbp-Dch: ignore
2015-12-26apt-helper: Add a cat-file command for concatening filesJulian Andres Klode
This can automatically handle compressed files and is useful for stuff like apt-file.
2015-12-19avoid triggering gpg2 migration in apt-keyDavid Kalnischkies
The presents (even of an empty) secring.gpg is indication enough for gpg2 to tigger the migration code which not only produces a bunch of output on each apt-key call, but also takes a while to complete as an agent needs to be started and all that. We workaround the first part by forcing the migration to happen always in a call we forced into silence, but that leaves us with an agent to start all the time – with a bit of reordering we can make it so that we do not explicitly create the secring, but let gpg create it if needed, which prevents the migration from being triggered and we have at least a bit less of a need for an agent. Changes - even to public only keyrings - still require one, but such actions are infrequent in comparison to verification calls, so that should be a net improvement.
2015-12-19avoid evaluating shell in paths used in apt-keyDavid Kalnischkies
apt-key creates internally a script (since ~1.1) which it will call to avoid dealing with an array of different options in the code itself, but while writing this script it wraps the values in "", which will cause the shell to evaluate its content upon execution. To make 'use' of this either set a absolute gpg command or TMPDIR to something as interesting as: "/tmp/This is fü\$\$ing cràzy, \$(man man | head -n1 | cut -d' ' -f1)\$!" If such paths can be encountered in reality is a different question…
2015-12-11apt-internal-solver: Make ShowHelp() and GetCommands() staticJulian Andres Klode
This fixes a warning reported by gcc. Gbp-Dch: ignore
2015-12-07part revert, part redo 'which' replacementDavid Kalnischkies
In e75e5879 'replace "which" with "command -v" for portability' I missed that command -v isn't actually required to be available in debian, so for the 5 files we are using it: Two (abicheck/run_abi_test & test/integration/framework) are called in environments were I believe sh is at least dash or 'better' as the first one is "interactive" for apt developers and the later is sourced by ~200 tests in the same directory run by hand and ci-services – for the later we have pulled some uglier hacks for worser things already, so if there should actually end up needing something more compatible we will notice eventually (and the later actually had a command -v call for some time already and nobody came running). debian/rules and debian/apt.cron.daily I switched back to which as that is more or less debian-specific or at least highly non-critical. That leaves cmdline/apt-key.in with a bunch of calls where I will implement that functionality in shell as this is relatively short-lived as it is used to detect wget (for net-update, which Michael wants to revive and in that process will properly use apt-helper instead of wget) and to detect gpg vs. gpg2 systems, where the earlier is supposed to go away in the longrun (or the later, but by replacing the earlier…). [and this gpg/gpg2 detection is new in sid, so I have some sympathy for that being a problem now.] Thanks: Jakub Wilk for pointing out #747320
2015-12-06replace run-parts with find|sort to avoid debianutils usageDavid Kalnischkies
After e75e5879 the reason for an implicit dependency on debianutils (which is essential for debian, but likely not on other systems) was just two uses of run-parts, which can be replaced with the a lot more portable find-piped-into-sort duo.
2015-12-06replace "which" with "command -v" for portabilityDavid Kalnischkies
which is a debian specific tool packaged in debianutils (essential) while command is a shell builtin defined by POSIX. Closes: 807144 Thanks: Mingye Wang for the suggestion.
2015-12-06apt-helper.cc: include <stdlib.h> for atoiFredrik Fornwall
Include <stdlib.h> to ensure that atoi(3) is defined to improve general portability and fix a specific build failure on Android. Closes: 807031
2015-11-29drop some needlessly public declarations in libapt-privateDavid Kalnischkies
Git-Dch: Ignore
2015-11-29move 'unmet' handling into libapt-privateDavid Kalnischkies
Git-Dch: Ignore
2015-11-29use function pointers instead of weak symbols for cmdline parsingDavid Kalnischkies
Passing function pointers around while working on this was very icky, but if weak symbols are too much to ask for… Reverts "do not use "-Wl,-Bsymbolic-functions" during the build to avoid breakage" aka a5fc9be36211a290a7abc3ca2a8bf98943bc1f57.
2015-11-21review of new/changed translatable program stringsJustin B Rye
Reference mail: https://lists.debian.org/debian-l10n-english/2015/11/msg00006.html
2015-11-04wrap every unlink call to check for != /dev/nullDavid Kalnischkies
Unlinking /dev/null is bad, we shouldn't do that. Also, we should print at least a warning if we tried to unlink a file but didn't manage to pull it of (ignoring the case were the file is /dev/null or doesn't exist in the first place). This got triggered by a relatively unlikely to cause problem in pkgAcquire::Worker::PrepareFiles which would while temporary uncompressed files (which are set to keep compressed) figure out that to files are the same and prepare for sharing by deleting them. Bad move. That also shows why not printing a warning is a bad idea as this hide the error for in non-root test runs. Git-Dch: Ignore
2015-11-04revamp all tools help messagesDavid Kalnischkies
The general idea is: A small paragraph on the tool itself as a description, a list of the most used (!= all) commands available in the tool, a remark where to find more information on the tool and its commands (aka: in the manpage) and finally a common block referring to even more manpages. In exchange options are completely omitted from the output as well as deprecated or obscure commands. (Better) Information about them is available in the manpages anyway and the few options which were listed before were also the least interesting ones (-o -c -q and co are hardly of interest for someone totally new looking to find info by asking for help and anyone with a bit of experience doesn't need this short list. Those would need a list of options applying to the command they call, but they are too numerous and command specific to list them sanely in this context.
2015-11-04hidden support more apt-get/apt-cache commands in aptDavid Kalnischkies
apt is supposed to be a user-friendly interface, so while these commands are usually poweruser material and therefore do not need to be shown in general introduction manpages/help messages its of no use to not allow users to use them. This includes clean, autoclean, build-dep, source, download, changelog, depends, rdepends and showsrc – it doesn't include more non-interactive commands like dump or xvcg as those are usually used by scripts if at all. Closes: 778234, 780700, 781237
2015-11-04deal with --version more centrallyDavid Kalnischkies
Git-Dch: Ignore
2015-11-04move apts cmdline helper type into -privateDavid Kalnischkies
Its not as simple as I initially thought to abstract this enough to make it globally usable, so lets not pollute global namespace with this for now. Git-Dch: Ignore
2015-11-04generate commands array after config is loadedDavid Kalnischkies
This ensures that location strings loaded from a location specified via configuration (Dir::Locale) effect the help messages for commands. Git-Dch: Ignore
2015-11-04new quiet level -qq for apt to hide progress outputDavid Kalnischkies
-q is for logging and -qqq (old -qq) basically kills every output expect errors, so there should be a way of declaring a middleground in which the output of e.g. 'update' isn't as verbose, but still shows some things. The test framework was actually making use of by accident as it ignored the quiet level in output setup for apt before. Eventually we should figure out some better quiet levels for all tools…
2015-11-04deduplicate main methodsDavid Kalnischkies
All mains pretty much do the same thing, so lets try a little harder to move the common parts into -private to have the real differences more visible. Git-Dch: Ignore
2015-11-04split up help messages for simpler reuseDavid Kalnischkies
That is one huge commit with busy work only: Help messages used to be one big translateable string, which is a pain for translators and hard to reuse for us. This change there 'explodes' this single string into new string for each documented string trying hard to split up the translated messages as well. This actually restores many translations as previously adding a single command made all of the bug message fuzzy. The splitup also highlighted that its easy to forget a line, duplicate one and similar stuff. Git-Dch: Ignore
2015-11-04move 'search' implementations as wellDavid Kalnischkies
Git-Dch: Ignore
2015-11-04centralize 'show' implementation of apt and apt-cacheDavid Kalnischkies
The show commands have different styles in both binaries as the audience is potentially very different, but that doesn't mean we need to separate the implementation especially as they are slightly similar. This also allows us to switch between the different show versions at runtime via an option. Git-Dch: Ignore
2015-11-04allow all dpkg selections to be set via apt-mark and libaptDavid Kalnischkies
As we have support for 'hold', we need support for undoing a hold which in effect means that we implemented most other states as well, just that they weren't exposed in the interface directly so far.
2015-11-04provide public interface to hold/unhold packagesDavid Kalnischkies
We had this code lying around in apt-mark for a while now, but other frontends need this (and similar) functionality as well, so its high time that we provide a public interface in libapt for this stuff.
2015-11-04switch 'apt-mark hold' from Pkg to Ver based operationDavid Kalnischkies
Users hold a package foo (at version X) or try to prevent the installation of foo (usually based on the information they know about version X), even if we say that we "hold a package". Conceptionally we also need to know about which architecture we are talking and that is an information bound to a version (as a package can change architecture over time). We internally did this lookup from Pkg to Ver already, we just move this to a central place where the user has a change to influence it now. Git-Dch: Ignore
2015-11-04add cacheset push_back wrapping for std::back_inserterDavid Kalnischkies
As usual by now, not all containers wrapped by the cacheset containers support all methods, like push_back now, but they fail on use of these unusable methods only. Would be nice to not expose these methods for unsupporting containers at all, but that means either a lot of classes or a lot of std::enable_if magic, which seems like too big work for this small wrapper for now. Git-Dch: Ignore
2015-11-04implement operator* for cacheset iteratorsDavid Kalnischkies
Technically an abi-break as we change a template parameter to std::iterator for this, but this class is empty in all instances and just causes the right typedefs to be set – which were incorrect as detected by std::stable_partition as its implementation uses ::pointer and needs also a operator* implementation. In practice CacheSets have no external users (yet) and the difference is visible only at compile time (which was an error before and now works), not while linking. The changes to apt-mark are functionally identical to the code before, just that we use a std:: algorithm now instead of trying hard on our own.
2015-11-04implement a public pkgSystem::MultiArchSupportedDavid Kalnischkies
Some codepaths need to check if the system (in our case usually dpkg) supports MultiArch or not. We had copy-pasted the check so far into these paths, but having it as a system check is better for reusability.
2015-09-14deal with spaces in path, command and filepaths in apt-keyDavid Kalnischkies
Filenames we get could include spaces, but also the tmpdir we work in and the failures we print in return a very generic and unhelpful… Properly supporting spaces is a bit painful as we constructed gpg command before, which is now moved to (multilevel) calls to temporary scripts instead.
2015-09-14various changes to increase test-coverageDavid Kalnischkies
And of course, testing obscure things ends up showing obscure 'bugs' or better shortcomings/inconsitencies, so lets fix them with the tests. Git-Dch: Ignore
2015-09-14implement apt-get source msg 'Please use: $vcs' for gitDavid Kalnischkies
A bit unfair that only Bzr had this message. Lets at least print it for git as well with the option of adding more later without string changes.
2015-09-14use a less generic special trigger filename for stdinDavid Kalnischkies
Git-Dch: Ignore
2015-09-14fix insecure use of /tmp in EDSP solver 'dump'David Kalnischkies
As said in the bugreport, this is hardly a serious problem on a security front, but it was always on the list to have the filename configurable somehow and the stable filename is a problem for parallel executions. Using an environment variable (APT_EDSP_DUMP_FILENAME) for this is more or less the best we can do here as solvers do not get told about our configuration and such. Closes: 795600
2015-09-14implement dpkgs vision of interpreting pkg:<arch> dependenciesDavid Kalnischkies
How the Multi-Arch field and pkg:<arch> dependencies interact was discussed at DebConf15 in the "MultiArch BoF". dpkg and apt (among other tools like dose) had a different interpretation in certain scenarios which we resolved by agreeing on dpkg view – and this commit realizes this agreement in code. As was the case so far libapt sticks to the idea of trying to hide MultiArch as much as possible from individual frontends and instead translates it to good old SingleArch. There are certainly situations which can be improved in frontends if they know that MultiArch is upon them, but these are improvements – not necessary changes needed to unbreak a frontend. The implementation idea is simple: If we parse a dependency on foo:amd64 the dependency is formed on a package 'foo:amd64' of arch 'any'. This package is provided by package 'foo' of arch 'amd64', but not by 'foo' of arch 'i386'. Both of those foo packages provide each other through (assuming foo is M-A:foreign) to allow a dependency on 'foo' to be satisfied by either foo of amd64 or i386. Packages can also declare to provide 'foo:amd64' which is translated to providing 'foo:amd64:any' as well. This indirection over provides was chosen as the alternative would be to teach dependency resolvers how to deal with architecture specific dependencies – which violates the design idea of avoiding resolver changes, especially as architecture-specific dependencies are a cornercase with quite a few subtil rules. Handling it all over versioned provides as we already did for M-A in general seems much simpler as it just works for them. This switch to :any has actually a "surprising" benefit as well: Even frontends showing a package name via .Name() [which doesn't show the architecture] will display the "architecture" for dependencies in which it was explicitely requested, while we will not show the 'strange' :any arch in FullName(true) [= pretty-print] either. Before you had to specialcase these and by default you wouldn't get these details shown. The only identifiable disadvantage is that this complicates error reporting and handling. apt-get's ShowBroken has existing problems with virtual packages [it just shows the name without any reason], so that has to be worked on eventually. The other case is that detecting if a package is completely unknown or if it was at least referenced somewhere needs to acount for this "split" – not that it makes a practical difference which error is shown… but its one of the improvements possible.
2015-09-01use clock() as source for SRV randomnessDavid Kalnischkies
Initializing a random number generator with the time since epoch could be good enough, but reaches its limits in test code as the 100 iterations might very well happen in the same second and hence the seed number is always the same… clock() has a way lower resolution so it changes more often and not unimportant: If many users start the update at the same time it isn't to unlikely the SRV record will be ordered in the same second choosing the same for them all, but it seems less likely that the exact same clock() time has passed for them. And if I have to touch this, lets change a few other things as well to make me and/or compilers a bit happier (clang complained about the usage of a GNU extension in the testcase for example).
2015-08-31if file is inaccessible for _apt, disable privilege drop in acquireDavid Kalnischkies
We had a very similar method previously for our own private usage, but with some generalisation we can move this check into the acquire system proper so that all frontends profit from this compatibility change. As we are disabling a security feature here a warning is issued and frontends are advised to consider reworking their download logic if possible. Note that this is implemented as an all or nothing situation: We can't just (not) drop privileges for a subset of the files in a fetcher, so in case you have to download some files with and some without you need to use two fetchers.
2015-08-29implement indextargets option 'DefaultEnabled'David Kalnischkies
Some targets like Contents-udeb are special-needs targets. Shipping the configuration snippet for them is okay, but they shouldn't be downloaded by default. Forcing the user to enable targets by uncommenting targets is wrong and this would still not really solve the problem completely as even if you want to download some -udebs it will probably not be for all sources you have enabled, so having the possibility of disabling a target by default, but giving the user the option to enable it on a per-source entry basis is better.
2015-08-28implement PDiff patching for compressed filesDavid Kalnischkies
Some additional files like 'Contents' are very big and should therefore kept compressed on the disk, which apt-file did in the past. It also implemented pdiff patching of these files by un- and recompressing these files on-the-fly, with this commit we can do the same – but we can do this in both pdiff patching styles (client and server merging) and secured by hashes. Hashes are in so far slightly complicated as we can't compare the hashes of the compressed files as we might compress them differently than the server would (different compressor versions, options, …), so we must compare the hashes of the uncompressed content. While this commit has changes in public headers, the classes it changes are marked as hidden, so nobody can use them directly, which means the ABI break is internal only.
2015-08-27sources.list and indextargets option for pdiffsDavid Kalnischkies
Disabling pdiffs can be useful occasionally, like if you have a fast local mirror where the download doesn't matter, but still want to use it for non-local mirrors. Also, some users might prefer it to only use it for very big indextargets like Contents.
2015-08-27use always priv-dropping for changelog download as rootDavid Kalnischkies
First of, the temporary directory we download the changelog to needs to be owned by _apt, but that also means that we don't need to check if we could/should drop privs as the download happens to a dedicated tempdir and only after that it is moved to its final location by a privileged user.
2015-08-27fix various typos reported by codespellDavid Kalnischkies
Reported-By: codespell
2015-08-20Merge remote-tracking branch 'mvo/feature/srv-records' into debian/experimentalMichael Vogt
2015-08-18Add support for "apt-cache showsrc --only-source srcpkgname"Michael Vogt
Thanks: Steve Slangasek for the suggestion Closes: 695633