| Age | Commit message (Collapse) | Author |
|---|
|
|
|
The filename can be overridden, but sometimes it is useful to do it only
for the directory-part of the filename – e.g. if you want to let a flat
archive directory (like /var/cache/apt/archives) serve a pool-based
request like /pool/a/apt_version.deb.
Gbp-Dch: Ignore
|
|
We sometimes autogenerate HTML pages e.g. for listing files in a
directory or for various error codes. If this would be a serious
webserver this would be a security problem (althrough a bit hard to
exploit), but as it is not shipped and intended to be used by our
testcases only the world hasn't ended & we can ignore it for
changelog and fix it for brownie points.
Gbp-Dch: Ignore
|
|
Browsing pages served via aptwebserver is working better if we tell the
browser the Content-Type which for this simple usecase we can just do by
guessing based on the file extension – and because hardcoding a list
would be boring we just reuse the mime.types data from mime-support if
available and allow it to be overridden by files and config.
Gbp-Dch: Ignore
|
|
If multiple threads act on requests (like if connection comes from a
webbrowser) a thread might request the supported compressors while
another thread is still working on creating the list to be stored in the
static cache variable.
As the price to pay for atomic and co seems to high for the fringe
usecase of manual usage of aptwebserver the patch just makes a call to
generate the list while still single threaded.
Gbp-Dch: Ignore
|
|
Moving the Retry-implementation from individual items to the worker
implementation not only gives every file retry capability instead of
just a selected few but also avoids needing to implement it in each item
(incorrectly).
|
|
This makes the code easier to read.
|
|
This makes it easier to see which headers includes what.
The changes were done by running
git grep -l '#\s*include' \
| grep -E '.(cc|h)$' \
| xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/'
To modify all include lines by adding a space, and then running
./git-clang-format.sh.
|
|
An unknown code should be handled the same as the x00 code of this
group, but for redirections we used to treat 300 (and a few others)
as an error while unknown codes were considered redirections.
Instead we check now explicitly for the redirection codes we support for
redirecting (and add the 308 defined in RFC 7538) to avoid future
problems if new 3xx codes are added expecting certain behaviours.
Potentially strange would have been e.g. "305 Use Proxy" sending a
Location for the proxy to use – which wouldn't have worked and resulted
in an error anyhow, but probably confused users in the process.
|
|
Gbp-Dch: Ignore
|
|
The test test-handle-redirect-as-used-mirror-change serves multiple
clients at the same time, so the order of the output is undefined and
once in a while the two clients will intermix their lines causing the
grep we perform on it later to fail making our tests fail.
Solved by introducing client-specific logfiles which we all grep and
sort the result to have the results more stable.
Git-Dch: Ignore
|
|
If the server told us in a previous request that it isn't supporting
Ranges with bytes via an Accept-Ranges header missing bytes, we don't
try to formulate requests using Ranges.
|
|
It seems completely pointless from a server-POV to sent empty header
fields, so most of them don't do it (simply proven by this limitation
existing since day one) – but it is technically allowed by the RFC as
the surounding whitespaces are optional and Github seems to like sending
"X-Geo-Block-List:\r\n" since recently (bug reports in other http
clients indicate July) at least sometimes as the reporter claims to have
seen it on https only even through it can happen with both.
Closes: 834048
|
|
All apt versions support numeric as well as 3-character timezones just
fine and its actually hard to write code which doesn't "accidently"
accepts it. So why change? Documenting the Date/Valid-Until fields in
the Release file is easy to do in terms of referencing the
datetime format used e.g. in the Debian changelogs (policy §4.4). This
format specifies only the numeric timezones through, not the nowadays
obsolete 3-character ones, so in the interest of least surprise we should
use the same format even through it carries a small risk of regression
in other clients (which encounter repositories created with
apt-ftparchive).
In case it is really regressing in practice, the hidden option
-o APT::FTPArchive::Release::NumericTimezone=0
can be used to go back to good old UTC as timezone.
The EDSP and EIPP protocols use this 'new' format, the text interface
used to communicate with the acquire methods does not for compatibility
reasons even if none of our methods would be effected and I doubt any
other would (in these instances the timezone is 'GMT' as that is what
HTTP/1.1 requires). Note that this is only true for apt talking to
methods, (libapt-based) methods talking to apt will respond with the
'new' format. It is therefore strongly adviced to support both also in
method input.
|
|
Breaking here lets our handler die which a client will fix by
reconnecting… but that eats time needlessly and is simple the wrong
handling, too.
Git-Dch: Ignore
|
|
Iceweasel^WFirefox complains about the missing encoding in its console
which can be a bit annoying in interactive sessions, so fixing these
issues has no effect on apt itself, but on the testers.
Git-Dch: Ignore
|
|
Git-Dch: Ignore
|
|
This allows running tests in parallel.
Git-Dch: Ignore
|
|
Conflicts:
apt-pkg/pkgcache.h
debian/changelog
methods/https.cc
methods/server.cc
test/integration/test-apt-download-progress
|
|
Add a regression test that reproduced the hang of apt when a
partial file is present.
Git-Dch: ignore
|
|
The apt http code parses Content-Length and Content-Range. For
both requests the variable "Size" is used and the semantic for
this Size is the total file size. However Content-Length is not
the entire file size for partital file requests. For servers that
send the Content-Range header first and then the Content-Length
header this can lead to globbing of Size so that its less than
the real file size. This may lead to a subsequent passing of a
negative number into the CircleBuf which leads to a endless
loop that writes data.
Thanks to Anton Blanchard for the analysis and initial patch.
LP: #1445239
|
|
If we have the expected hashes we can check with them if the file we
have in partial we got a 416 for is the expected file. We detected this
with same-size before, but not every server sends a good Content-Range
header with a 416 response.
|
|
If we get a IMSHit for the Transaction-Manager (= the InRelease file or
as its still supported fallback Release + Release.gpg combo) we can
assume that every file we would queue based on this manager, but already
have locally is current and hence would get an IMSHit, too. We therefore
save us and the server the trouble and skip the queuing in this case.
Beside speeding up repetative executions of 'apt-get update' this way we
also avoid hitting hashsum errors if the indexes are in fact already
updated, but the Release file isn't yet as it is the case on well
behaving mirrors as Release files is updated last.
The implementation is a bit harder than the theory makes it sound as we
still have to keep reverifying the Release files (e.g. to detect now expired
once to avoid an attacker being able to silently stale us) and have to
handle cases in which the Release file hits, but some indexes aren't
present (e.g. user added a new foreign architecture).
|
|
Servers who advertise that they close the connection get the 'Closes'
encoding flag, but this conflicts with servers who response with a
transfer-encoding (e.g. encoding) as it is saved in the same flag.
We have a better flag for the keep-alive (or not) of the connection
anyway, so we check this instead of the encoding.
This is in practice not much of a problem as real servers we talk to are
HTTP1.1 servers (with keep-alive) and there isn't much point in doing
chunked encoding if you are going to close anyway, but our simple
testserver stumbles over this if pressed and its a bit cleaner, too.
Git-Dch: Ignore
|
|
Bug #778375 uncovered that https wasn't properly integrated in the class
family tree of http as it was supposed to be leading to a NULL pointer
dereference. Fixing this 'properly' was deemed to much diff for
practically no gain that late in the release, so commit
0c2dc43d4fe1d026650b5e2920a021557f9534a6 just fixed the synptom, while
this commit here is fixing the cause plus adding a test.
|
|
|
|
Real webservers (like apache) actually send an error page with a 416
response, but our client didn't expect it leaving the page on the socket
to be parsed as response for the next request (http) or as file content
(https), which isn't what we want at all… Symptom is a "Bad header line"
as html usually doesn't parse that well to an http-header.
This manifests itself e.g. if we have a complete file (or larger) in
partial/ which isn't discarded by If-Range as the server doesn't support
it (or it is just newer, think: mirror rotation).
It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a,
which removed the filesize - 1 trick, but this had its own problems…
To properly test this our webserver gains the ability to reply with
transfer-encoding: chunked as most real webservers will use it to send
the dynamically generated error pages.
(The tests and their binary helpers had to be slightly modified to
apply, but the patch to fix the issue itself is unchanged.)
Closes: 768797
|
|
Real webservers (like apache) actually send an error page with a 416
response, but our client didn't expect it leaving the page on the socket
to be parsed as response for the next request (http) or as file content
(https), which isn't what we want at all… Symptom is a "Bad header line"
as html usually doesn't parse that well to an http-header.
This manifests itself e.g. if we have a complete file (or larger) in
partial/ which isn't discarded by If-Range as the server doesn't support
it (or it is just newer, think: mirror rotation).
It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a,
which removed the filesize - 1 trick, but this had its own problems…
To properly test this our webserver gains the ability to reply with
transfer-encoding: chunked as most real webservers will use it to send
the dynamically generated error pages.
Closes: 768797
|
|
It is a very simple hashstring, which is why it isn't contributing to
the usability of a list of them, but it is also trivial to check and
calculate, so it doesn't hurt checking it either as it can combined even
with the simplest other hashes greatly complicate attacks on them as you
suddenly need a same-size hash collision, which is usually a lot harder
to achieve.
|
|
Git-Dch: Ignore
|
|
Beside being a bit cleaner it hopefully also resolves oddball problems
I have with high levels of parallel jobs.
Git-Dch: Ignore
Reported-By: iwyu (include-what-you-use)
|
|
Reported-By: gcc -Wignored-qualifiers
Git-Dch: Ignore
|
|
Git-Dch: Ignore
Reported-By: gcc -Wmissing-declarations
|
|
Reporting it via progress means that e.g. a redirect will trigger it,
too, so you get a Get & Hit while http only reports a Hit as it should
be.
|
|
The URI to use to set a config option is a bit arcane to write/remember
and checking if the setting was successful doubly so.
Git-Dch: Ignore
|
|
Clients like browsers prefer to open many connections and keep them open
for a while, so that pages with lot of subelements would take a while to
load (if at all), by using threads as all servers do some way or another
we can resolve this. libapt is not intended to be pthread-safe and stuff
like the storage of the last return code doesn't make too much sense if
multiple clients interact with us, but it is good enough for now and an
other interesting (mis)use of libapt in general.
Git-Dch: Ignore
|
|
Again, not (currently) used by the tests itself, but in interactive
usage of the webserver itself.
Git-Dch: Ignore
|
|
Git-Dch: Ignore
|
|
pretty much useless for the testcases, but handy to test the webserver
itself in 'real world' environments
Git-Dch: Ignore
|
|
Webserver wrongly sends an additional newline after the data which
causes curl to believe that the next request on this socket has no
header data and so includes all headers in the data output.
Git-Dch: Ignore
|
|
Git-Dch: Ignore
|
|
Git-Dch: Ignore
|
|
Commit 2b9c9b7f28b18f6ae3e422020e8934872b06c9f3 not only removes
keep-alive, but also changes the request URI send to proxies which are
required to be absolute URIs rather than the usual absolute paths.
Closes: 717891
|
|
Forking only after being ready to accept clients avoids running races
with the tests which sometimes failed on the first 'apt-get update'
(or similar) with the previous background-start and hope for the best…
The commit fixes also some oversight output-order changes in regards to
Description-md5 and (I-M-S) race conditions in various tests.
Git-Dch: Ignore
|
|
With the selfgrown splitting we got the problem of not recovering
from networks which just reply with invalid data like those sending
us login pages to authenticate with the network (e.g. hotels) back.
The good thing about the InRelease file is that we know that it must
be clearsigned (a Release file might or might not have a detached sig)
so if we get a file but are unable to split it something is seriously
wrong, so there is not much point in trying further.
The Acquire system already looks out for a NODATA error from gpgv,
so this adds a new error message sent to the acquire system in case
the splitting we do now ourselves failed including this magic word.
Closes: #712486
|
|
we have a test which required traditionally lighttpd to be executed
as it requires a webserver supporting some kind of URI rewriting.
Now with some lines of code our own webserver can do this and the
testcase can be enabled by default. This test hinted at the bug fixed
in the previous commit, so having more tests which can easily be run
is a good thing.
Git-Dch: Ignore
|
|
Git-Dch: Ignore
|
|
APT needs to acquire data in a secure fashion over an inherently
unsecure way, known as the internet, while communicating with
unreliable partners, known as webservers and proxies.
For your integration tests we so far relied on 'normal' webservers,
but all of them have certain quirks and none is able to provide us
with all quirks which can be observed in the wild and we therefore
have to test with, so this webserver isn't trying to be fast, secure
or feature complete, but to provide all the quirks we need in a
consistent way.
This webserver also makes the APT project self-contained, as it is now
able to generate, serve as well as acquire package indexes. ;)
Git-Dch: Ignore
|
