| Age | Commit message (Collapse) | Author |
|---|
|
Providing the benefits of both without the downsides :)
(ABI breaks or external dependencies)
For this Anthonys rred is equipped with:
- magic-filename-pickup of patches rather than explicit messages
- use of FileFd instead of FILE* to get on-the-fly uncompress
of the gzip compressed pdiff patches
The acquire code in turn stops checking for apt-file's helper
as our own rred is now clever enough for our needs.
|
|
Based on the idea presented in:
https://lists.debian.org/deity/2009/08/msg00169.html and
https://lists.debian.org/debian-devel/2014/01/msg00081.html
It reads all patches one by one and merges them in-memory before
applying the merged changes to the index.
Beware: This commit by David Kalnischkies rips out the rred binary
rewrite unchanged (expect minor format issue corrections) from the
proposed changes, so this commit alone BREAKS pdiff completely.
The integration into the acquire system as it was prepared in the
previous POC will be done in the next commit to have proper 'blame'.
|
|
The idea of pdiffs is to avoid downloading the hole file by patching the
existing index. This works very well, but becomes slow if a lot of
patches needs to be applied to reconstruct an up-to-date index and in
recent years more and more dinstall (or similar) runs are executed
creating more and more pdiffs in the same amount of time, so pdiffs
became less useful.
The solution is simple: Reduce the amount of patches (which are very
small) which need to be applied on top of the index we have available
(which is usually pretty big).
This can be done in two ways: Either merge the patches on the
server-side so that the client has to download only one patch or the
patches are all downloaded and merged on the client-side.
The first needs a client who is doing one step at a time who can also
skip patches if it needs (APT supports this for a long time now).
The later is implemented by this commit, but depends on the server NOT
merging the patches and the patches being in a strict order in which no
patch is skipped.
This is traditionally the case for dak, but other repository creators
support merging – e.g. reprepro (which helpfully adds a flag indicating
that the patches are merged). To support both or even mixes a client
needs more information which isn't available for now.
This POC uses the external diffindex-rred included in apt-file to
do the heavy lifting of merging & applying all patches in one pass,
hence to test this feature apt-file needs to be installed.
|
|
|
|
Servers might respond with a complete file either because they don't
support Ranges at all or the If-Range condition isn't statisfied, so we
have to parse the headers curl gets ourself to seek or truncate the file
we have so far.
This also finially adds the testcase testing a bunch of partial
situations for both, http and https - which is now all green.
Closes: 617643, 667699
LP: 1157943
|
|
As lengthy discussed in lp:1157943 partial https support was utterly
broken as a 206 response was handled as an (unhandled) error. This is
the first part of fixing it by supporting a 206 response and starting to
deal with 416.
|
|
No effective behavior change, just shuffling big junks of code between
methods and classes to split them into those strongly related to our
client implementation and those implementing HTTP.
The idea is to get HTTPS to a point in which most of the implementation
can be shared even though the client implementations itself is
completely different. This isn't anywhere near yet though, but it should
beenough to reuse at least a few lines from http in https now.
Git-Dch: Ignore
|
|
Our http client requests the "filesize - 1" for the small edgecase of
handling a file which was completely downloaded, but not yet moved to
the correct place as we get 416 errors in that case, but as we can
handle 416 returns now we just special-case the situation of requesting
the exact filesize and handle it as a 200 without content instead.
|
|
If we get a 416 from the server it means the Range we asked for is above
the real filesize of the file on the server. Mostly this happens if the
server isn't supporting If-Range, but regardless of how we end up with
the partial data, the data is invalid so we discard it and retry with a
fresh plate and hope for the best.
Old behavior was to consider 416 an error and retry with a different
compression until we ran out of compression and requested the
uncompressed file (which doesn't exist on most mirrors) with an accept
line which server answered with "406 Not Acceptable".
Closes: 710924
|
|
|
|
|
|
Commit 2b9c9b7f28b18f6ae3e422020e8934872b06c9f3 not only removes
keep-alive, but also changes the request URI send to proxies which are
required to be absolute URIs rather than the usual absolute paths.
Closes: 717891
|
|
|
|
|
|
|
|
With the selfgrown splitting we got the problem of not recovering
from networks which just reply with invalid data like those sending
us login pages to authenticate with the network (e.g. hotels) back.
The good thing about the InRelease file is that we know that it must
be clearsigned (a Release file might or might not have a detached sig)
so if we get a file but are unable to split it something is seriously
wrong, so there is not much point in trying further.
The Acquire system already looks out for a NODATA error from gpgv,
so this adds a new error message sent to the acquire system in case
the splitting we do now ourselves failed including this magic word.
Closes: #712486
|
|
Building src:apt shows:
dpkg-shlibdeps: warning: package could avoid a useless dependency if
debian/apt/usr/lib/apt/methods/cdrom was not linked against libdl.so.2
(it uses none of the library's symbols)
dpkg-shlibdeps: warning: package could avoid a useless dependency if
debian/apt/usr/bin/apt-get was not linked against libutil.so.1 (it uses
none of the library's symbols)
|
|
* Fix double free (closes: #711045)
* Fix crash when the "mirror" method does not find any entry
(closes: #699303)
|
|
|
|
|
|
(closes: #705648)
|
|
translations. Closes: #705087
|
|
|
|
|
|
- use Errno() instead of strerror(), thanks to David Kalnischk
|
|
"Acquire::ForceIPv6" to allow focing one or the other
(closes: #611891)
|
|
|
|
- ExecGPGV is a method which should never return, so mark it as such
and fix the inconsistency of returning in error cases
|
|
- rename RunGPGV to ExecGPGV and move it to apt-pkg/contrib/gpgv.cc
|
|
|
|
(LP: #1003633)
|
|
- quote spaces in filenames to ensure as the http method is also
(potentially) used for non deb,dsc content that may contain
spaces, thanks to Daniel Hartwig and Thomas Bushnell
(LP: #1086997)
|
|
|
|
- reuse connection in https, thanks to Thomas Bushnell, BSG for the
patch. LP: #1087543, Closes: #695359
|
|
|
|
- after many years of pointless discussions disable http/1.1 pipelining
by default as many webservers and proxies seem to be unable to conform
to specification must's (rfc2616 section 8.1.2.2) (LP: #996151)
|
|
- generate an equal sign also for the first arch (Closes: #669142)
|
|
- handle redirections in the worker with the right method instead of
in the method the redirection occured in (Closes: #668111)
* methods/http.cc:
- forbid redirects to change protocol
|
|
|
|
with g++-4.1 it complains about this so lets be extra clear
|
|
- remove it as the functionality for all compressors can be
provided by gzip.cc now with the usage of FileFD
|
|
|
|
- ship the ftparchive, apt-extractemplates and apt-sortpkgs locales
in the apt-utils package instead of the apt package
|
|
to the more standard PACKAGE_VERSION and make it work in every file
|
|
- check return of writev() as gcc recommends
* methods/mirror.cc:
- check return of chdir() as gcc recommends
* apt-pkg/deb/dpkgpm.cc:
- check return of write() a gcc recommends
* apt-inst/deb/debfile.cc:
- check return of chdir() as gcc recommends
* apt-inst/deb/dpkgdb.cc:
- check return of chdir() as gcc recommends
|
|
- do not link rred against libz anymore as FileFd handles all
this transparently now
|
|
non-primitive types."
|
|
|
|
|
|
commit lifted the Line-length limit
|
