From 895417ef99bb1371d8970da1afe87c6d64382f67 Mon Sep 17 00:00:00 2001 From: David Kalnischkies Date: Mon, 28 Apr 2014 10:02:27 +0200 Subject: reenable pipelining via hashsum reordering support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Now that methods have the expected hashes available they can check if the response from the server is what they expected. Pipelining is one of those areas in which servers can mess up by not supporting it properly, which forced us to disable it for the time being. Now, we check if we got a response out of order, which we can not only use to disable pipelining automatically for the next requests, but we can fix it up just like the server responded in proper order for the current requests. To ensure that this little trick works pipelining is only attempt if we have hashsums for all the files in the chain which in theory reduces the use of pipelining usage even on the many servers which work properly, but in practice only the InRelease file (or similar such) will be requested without a hashsum – and as it is the only file requested in that stage it can't be pipelined even if we wanted to. Some minor annoyances remain: The display of the progress we have doesn't reflect this change, so it looks like the same package gets downloaded multiple times while others aren't at all. Further more, partial files are not supported in this recovery as the received data was appended to the wrong file, so the hashsum doesn't match. Both seem to be minor enough to reenable pipelining by default until further notice through to test if it really solves the problem. This therefore reverts commit 8221431757c775ee875a061b184b5f6f2330f928. --- test/integration/test-http-pipeline-messup | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 test/integration/test-http-pipeline-messup (limited to 'test/integration/test-http-pipeline-messup') diff --git a/test/integration/test-http-pipeline-messup b/test/integration/test-http-pipeline-messup new file mode 100755 index 000000000..9c59e1825 --- /dev/null +++ b/test/integration/test-http-pipeline-messup @@ -0,0 +1,43 @@ +#!/bin/sh +set -e + +TESTDIR=$(readlink -f $(dirname $0)) +. $TESTDIR/framework + +setupenvironment +configarchitecture "i386" + +buildsimplenativepackage 'pkga' 'all' '1.0' 'stable' +buildsimplenativepackage 'pkgb' 'all' '1.0' 'stable' +buildsimplenativepackage 'pkgc' 'all' '1.0' 'stable' +buildsimplenativepackage 'pkgd' 'all' '1.0' 'stable' + +setupaptarchive --no-update + +# simulate (and be a predictable) pipeline mess-up by the server/proxy +changetowebserver \ + -o 'aptwebserver::overwrite::.*pkga.*::filename=/pool/pkgd_1.0_all.deb' \ + -o 'aptwebserver::overwrite::.*pkgc.*::filename=/pool/pkgb_1.0_all.deb' \ + -o 'aptwebserver::overwrite::.*pkgb.*::filename=/pool/pkgc_1.0_all.deb' \ + -o 'aptwebserver::overwrite::.*pkgd.*::filename=/pool/pkga_1.0_all.deb' + +testsuccess aptget update -o Debug::Acquire::http=1 -o Debug::pkgAcquire::Worker=1 + +# messup is bigger than pipeline: checks if fixup isn't trying to hard +testfailure aptget download pkga pkgb pkgc pkgd "$@" -o Acquire::http::Pipeline-Depth=2 +testfailure test -f pkga_1.0_all.deb + +# ensure that pipeling is enabled for rest of this test +echo 'Acquire::http::Pipeline-Depth 10;' > rootdir/etc/apt/apt.conf.d/99enable-pipeline + +# the output is a bit strange: it looks like it has downloaded pkga 4 times +testsuccess aptget download pkga pkgb pkgc pkgd -o Debug::Acquire::http=1 -o Debug::pkgAcquire::Worker=1 +for pkg in 'pkga' 'pkgb' 'pkgc' 'pkgd'; do + testsuccess test -f ${pkg}_1.0_all.deb + testsuccess cmp incoming/${pkg}_1.0_all.deb ${pkg}_1.0_all.deb + rm -f ${pkg}_1.0_all.deb +done + +# while hashes will pass (as none are available), sizes will not match, so failure +# checks that no hashes means that pipeline depth is ignored as we can't fixup +testfailure aptget download pkga pkgb pkgc pkgd "$@" --allow-unauthenticated -o Acquire::ForceHash=ROT26 -- cgit v1.2.3 From f9a3c4bde867e70e8c89b6ed5924ab9fab517096 Mon Sep 17 00:00:00 2001 From: David Kalnischkies Date: Wed, 8 Oct 2014 00:37:32 +0200 Subject: fix http-pipeline-messup testcase The test generates failures if the created deb files have the same size, so we try a little harder to avoid having the same size for them. Git-Dch: Ignore --- test/integration/test-http-pipeline-messup | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) (limited to 'test/integration/test-http-pipeline-messup') diff --git a/test/integration/test-http-pipeline-messup b/test/integration/test-http-pipeline-messup index 9c59e1825..405574e8a 100755 --- a/test/integration/test-http-pipeline-messup +++ b/test/integration/test-http-pipeline-messup @@ -7,10 +7,11 @@ TESTDIR=$(readlink -f $(dirname $0)) setupenvironment configarchitecture "i386" -buildsimplenativepackage 'pkga' 'all' '1.0' 'stable' -buildsimplenativepackage 'pkgb' 'all' '1.0' 'stable' -buildsimplenativepackage 'pkgc' 'all' '1.0' 'stable' -buildsimplenativepackage 'pkgd' 'all' '1.0' 'stable' +# try a little harder to create a size mismatch +buildsimplenativepackage 'pkga' 'all' '1.0' 'stable' "Depends: foo" '' '' '' '' 'none' +buildsimplenativepackage 'pkgb' 'all' '1.0' 'stable' "Depends: foo" '' '' '' '' 'none' +buildsimplenativepackage 'pkgc' 'all' '1.0' 'stable' "Depends: f$(for i in $(seq 0 1000); do printf 'o'; done)" '' '' '' '' 'none' +buildsimplenativepackage 'pkgd' 'all' '1.0' 'stable' "Depends: f$(for i in $(seq 0 1000); do printf 'o'; done)" '' '' '' '' 'none' setupaptarchive --no-update @@ -21,17 +22,20 @@ changetowebserver \ -o 'aptwebserver::overwrite::.*pkgb.*::filename=/pool/pkgc_1.0_all.deb' \ -o 'aptwebserver::overwrite::.*pkgd.*::filename=/pool/pkga_1.0_all.deb' -testsuccess aptget update -o Debug::Acquire::http=1 -o Debug::pkgAcquire::Worker=1 +echo 'Debug::Acquire::http "true"; +Debug::pkgAcquire::Worker "true";' > rootdir/etc/apt/apt.conf.d/99debug + +testsuccess aptget update # messup is bigger than pipeline: checks if fixup isn't trying to hard -testfailure aptget download pkga pkgb pkgc pkgd "$@" -o Acquire::http::Pipeline-Depth=2 +testfailure aptget download pkga pkgb pkgc pkgd -o Acquire::http::Pipeline-Depth=2 testfailure test -f pkga_1.0_all.deb # ensure that pipeling is enabled for rest of this test echo 'Acquire::http::Pipeline-Depth 10;' > rootdir/etc/apt/apt.conf.d/99enable-pipeline # the output is a bit strange: it looks like it has downloaded pkga 4 times -testsuccess aptget download pkga pkgb pkgc pkgd -o Debug::Acquire::http=1 -o Debug::pkgAcquire::Worker=1 +testsuccess aptget download pkga pkgb pkgc pkgd for pkg in 'pkga' 'pkgb' 'pkgc' 'pkgd'; do testsuccess test -f ${pkg}_1.0_all.deb testsuccess cmp incoming/${pkg}_1.0_all.deb ${pkg}_1.0_all.deb @@ -40,4 +44,4 @@ done # while hashes will pass (as none are available), sizes will not match, so failure # checks that no hashes means that pipeline depth is ignored as we can't fixup -testfailure aptget download pkga pkgb pkgc pkgd "$@" --allow-unauthenticated -o Acquire::ForceHash=ROT26 +testfailure aptget download pkga pkgb pkgc pkgd --allow-unauthenticated -o Acquire::ForceHash=ROT26 -- cgit v1.2.3 From 4fa34122cbe347d21b3a162ff2fa75dd2e73c3a8 Mon Sep 17 00:00:00 2001 From: David Kalnischkies Date: Mon, 20 Oct 2014 10:23:41 +0200 Subject: testcases: do not allow warnings in testsuccess Adds a new testwarning which tests for zero exit and the presents of a warning in the output, failing if either is not the case or if an error is found, too. This allows us to change testsuccess to accept only totally successful executions (= without warnings) which should help finding regressions. Git-Dch: Ignore --- test/integration/test-http-pipeline-messup | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'test/integration/test-http-pipeline-messup') diff --git a/test/integration/test-http-pipeline-messup b/test/integration/test-http-pipeline-messup index 405574e8a..dda8ef7eb 100755 --- a/test/integration/test-http-pipeline-messup +++ b/test/integration/test-http-pipeline-messup @@ -35,7 +35,7 @@ testfailure test -f pkga_1.0_all.deb echo 'Acquire::http::Pipeline-Depth 10;' > rootdir/etc/apt/apt.conf.d/99enable-pipeline # the output is a bit strange: it looks like it has downloaded pkga 4 times -testsuccess aptget download pkga pkgb pkgc pkgd +testwarning aptget download pkga pkgb pkgc pkgd for pkg in 'pkga' 'pkgb' 'pkgc' 'pkgd'; do testsuccess test -f ${pkg}_1.0_all.deb testsuccess cmp incoming/${pkg}_1.0_all.deb ${pkg}_1.0_all.deb -- cgit v1.2.3