summaryrefslogtreecommitdiff
path: root/test/integration/test-compressed-indexes
AgeCommit message (Collapse)Author
2016-05-16show final solution in --no-download --fix-missing modeDavid Kalnischkies
This commit moves the creation of the fetcher and with it the calculation of the filenames before the code generation the various lists detailing the solution. This means that simulation comes even so slightly closer to a real run as it will require and parse the package indexes for filenames and queuing of URIs, so that a simulation "using" an unavailable download method actually fails now. The real benefit of this change is through that the rather special but nontheless handy --no-download --fix-missing mode now actually shows what the solution is it will apply to the system rather than the solution it would if it could download all not-downloaded packages.
2016-04-14silently skip acquire of empty index filesDavid Kalnischkies
There is just no point in taking the time to acquire empty files – especially as it will be tiny non-empty compressed files usually.
2016-01-08keep compressed indexes in a low-cost formatDavid Kalnischkies
Downloading and storing are two different operations were different compression types can be preferred. For downloading we provide the choice via Acquire::CompressionTypes::Order as there is a choice to be made between download size and speed – and limited by whats available in the repository. Storage on the other hand has all compressions currently supported by apt available and to reduce runtime of tools accessing these files the compression type should be a low-cost format in terms of decompression. apt traditionally stores its indexes uncompressed on disk, but has options to keep them compressed. Now that apt downloads additional files we also deal with files which simply can't be stored uncompressed as they are just too big (like Contents for apt-file). Traditionally they are downloaded in a low-cost format (gz) as repositories do not provide other formats, but there might be even lower-cost formats and for download we could introduce higher-cost in the repositories. Downloading an entire index potentially requires recompression to another format, so an update takes potentially longer – but big files are usually updated via pdiffs which has to de- and re-compress anyhow and does it on the fly anyhow, so there is no extra time needed and in general it seems to be benefitial to invest the time in update to save time later on file access.
2016-01-08tests: try to pick up compressors from config automaticallyDavid Kalnischkies
Less hardcoding should help while introducing new compressors. Git-Dch: Ignore
2015-12-19tests: support spaces in path and TMPDIRDavid Kalnischkies
This doesn't allow all tests to run cleanly, but it at least allows to write tests which could run successfully in such environments. Git-Dch: Ignore
2015-11-04support arch:all data e.g. in separate Packages fileDavid Kalnischkies
Based on a discussion with Niels Thykier who asked for Contents-all this implements apt trying for all architecture dependent files to get a file for the architecture all, which is treated internally now as an official architecture which is always around (like native). This way arch:all data can be shared instead of duplicated for each architecture requiring the user to download the same information again and again. There is one problem however: In Debian there is already a binary-all/ Packages file, but the binary-any files still include arch:all packages, so that downloading this file now would be a waste of time, bandwidth and diskspace. We therefore need a way to decide if it makes sense to download the all file for Packages in Debian or not. The obvious answer would be a special flag in the Release file indicating this, which would need to default to 'no' and every reasonable repository would override it to 'yes' in a few years time, but the flag would be there "forever". Looking closer at a Release file we see the field "Architectures", which doesn't include 'all' at the moment. With the idea outlined above that 'all' is a "proper" architecture now, we interpret this field as being authoritative in declaring which architectures are supported by this repository. If it says 'all', apt will try to get all, if not it will be skipped. This gives us another interesting feature: If I configure a source to download armel and mips, but it declares it supports only armel apt will now print a notice saying as much. Previously this was a very cryptic failure. If on the other hand the repository supports mips, too, but for some reason doesn't ship mips packages at the moment, this 'missing' file is silently ignored (= that is the same as the repository including an empty file). The Architectures field isn't mandatory through, so if it isn't there, we assume that every architecture is supported by this repository, which skips the arch:all if not listed in the release file.
2015-11-04unbreak the copy-method claiming hashsum mismatch since ~exp9David Kalnischkies
Commit 653ef26c70dc9c0e2cbfdd4e79117876bb63e87d broke the camels back in sofar that everything works in terms of our internal use of copy:/, but external use is completely destroyed. This is kinda the reverse of what happened in "parallel" in the sid branch, where external use was mostly fine, internal and external exploded on the GzipIndexes option. We fix this now by rewriting our internal use by letting copy:/ only do what the name suggests it does: Copy files and not uncompress them on-the-fly. Then we teach copy and the uncompressors how to deal with /dev/null and use it as destination file in case we don't want to store the uncompressed files on disk. Closes: 799158
2015-08-10allow individual targets to be kept compressedDavid Kalnischkies
There is an option to keep all targets (Packages, Sources, …) compressed for a while now, but the all-or-nothing approach is a bit limited for our purposes with additional targets as some of them are very big (Contents) and rarely used in comparison, so keeping them compressed by default can make sense, while others are still unpacked. Most interesting is the copy-change maybe: Copy is used by the acquire system as an uncompressor and it is hence expected that it returns the hashes for the "output", not the input. Now, in the case of keeping a file compressed, the output is never written to disk, but generated in memory and we should still validated it, so for compressed files copy is expected to return the hashes of the uncompressed file. We used to use the config option to enable on-the-fly decompress in the method, but in reality copy is never used in a way where it shouldn't decompress a compressed file to get its hashes, so we can save us the trouble of sending this information to the method and just do it always.
2015-04-19send Alt-* info for uncompressed based on any compressionsDavid Kalnischkies
file sends information about the uncompressed file if it can find it as well as for the compressed file. This was done only for gzip so far, but we support more compression types. That this information isn't used a lot is a different story. Git-Dch: Ignore
2015-03-16test exitcode as well as string equalityDavid Kalnischkies
We use test{success,failure} now all over the place in the framework, so its only consequencial to do this in the situations in which we test for a specific output as well. Git-Dch: Ignore
2014-10-23chown finished partial files earlierDavid Kalnischkies
partial files are chowned by the Item baseclass to let the methods work with them. Now, this baseclass is also responsible for chowning the files back to root instead of having various deeper levels do this. The consequence is that all overloaded Failed() methods now call the Item::Failed base as their first step. The same is done for Done(). The effect is that even in partial files usually don't belong to _apt anymore, helping sneakernets and reducing possibilities of a bad method modifying files not belonging to them. The change is supported by the framework not only supporting being run as root, but with proper permission management, too, so that privilege dropping can be tested with them.
2014-10-15ignore Acquire::GzipIndexes for cdrom sourcesDavid Kalnischkies
We do not support compressed indexes for cdrom sources as we rewrite some of them, so supporting it correctly could be hard. What we do instead in the meantime is probably disabling it for cdrom sources.
2014-10-13do not load filesize in pkgAcqIndexTrans explicitlyDavid Kalnischkies
The constructor is calling the baseclass pkgAcqIndex which does this already – and also does it correctly for compressed files which would overwise lead to the size of uncompressed files to be expected. Git-Dch: Ignore
2014-10-07Merge remote-tracking branch 'upstream/debian/experimental' into ↵Michael Vogt
feature/acq-trans Conflicts: apt-pkg/acquire-item.cc
2014-09-25rewrite compressed indexes test to check with all compressorsDavid Kalnischkies
Git-Dch: Ignore
2014-09-21generalize Acquire::GzipIndexMichael Vogt
2014-06-10support Acquire::GzipIndexes in dumpavailDavid Kalnischkies
Closes: 742835
2014-03-13ensure that gz compression test is run with gzDavid Kalnischkies
The framework can be configured to use different compression algorithms to test different ones, but a testcase testing for gz support should always be run with gz, regardless of what compressions are configured otherwise. Git-Dch: Ignore
2013-08-12add chronic-like testsuccess/testfailure helpersDavid Kalnischkies
For many commands the output isn't stable (like then dpkg is called) but the exitcode is, so this helper enhances the common && msgpass || msgfail by generating automatically a msgtest and showing the output of the command in case of failure instead of discarding it unconditionally, the later being chronic-like behaviour Git-Dch: Ignore
2011-09-09fix bashism (local outside function) and releasefile creation in testcasesDavid Kalnischkies
2011-08-22remove the caches in 'apt-get update', too, as they will beDavid Kalnischkies
invalid in most cases anyway
2011-01-21avoid building simple packages with debhelper to speed it up a bitDavid Kalnischkies
2010-11-15* test/integration/test-compressed-indexes, test/test-indexes.sh:Martin Pitt
- Explicitly disable compressed indexes at the start. This ensures that we will actually test uncompressed indexes regardless of the internal default value of Acquire::GzipIndexes.
2010-10-13tests/integration/test-*: remove a bunch of "local" that are used outside ↵Michael Vogt
funtions (bash complains)
2010-08-21* apt-pkg/acquire-item.cc:David Kalnischkies
- don't use ReadOnlyGzip mode for PDiffs as this mode doesn't work in combination with the AddFd methods of our hashclasses Add also 2 testcases: one to test pdiffs in general and one to test the handling of compressed indexes.