summaryrefslogtreecommitdiff
path: root/apt-pkg/contrib/fileutl.cc
AgeCommit message (Collapse)Author
2016-11-14Revert "if the FileFd failed already following calls should fail, too"Julian Andres Klode
This reverts commit 1b63558a39ee1eed7eb024cd0e164d73beb165b1. This commit caused a regression in the unit tests: The error was propagated to Close(), where we expected it to return true.
2016-10-05Do not read stderr from proxy autodetection scriptsJulian Andres Klode
This fixes a regression introduced in commit 8f858d560e3b7b475c623c4e242d1edce246025a don't leak FD in AutoProxyDetect command return parsing which accidentally made the proxy autodetection code also read the scripts output on stderr, not only on stdout when it switched the code from popen() to Popen(). Reported-By: Tim Small <tim@seoss.co.uk> (cherry picked from commit 0ecceb5bb9cc8727c117195945b7116aceb984fe)
2016-10-05fileutl: empty file support: Avoid fstat() on -1 fd and check resultJulian Andres Klode
When checking if a file is empty, we forget to check that fstat() actually worked. (cherry picked from commit 15fe8e62d37bc87114c59d385bed7ceefb72886b)
2016-10-05if the FileFd failed already following calls should fail, tooDavid Kalnischkies
There is no point in trying to perform Write/Read on a FileFd which already failed as they aren't going to work as expected, so we should make sure that they fail early on and hard. (cherry picked from commit 02c38073af51802c02bb104d4450e0e112d641ad)
2016-10-05(error) va_list 'args' was opened but not closed by va_end()David Kalnischkies
Reported-By: cppcheck Gbp-Dch: Ignore (cherry picked from commit 196d590a99e309764e07c9dc23ea98897eebf53a)
2016-08-31call flush on the wrapped writebuffered FileFdDavid Kalnischkies
The flush call is a no-op in most FileFd implementations so this isn't as critical as it might sound as the only non-trivial implementation is in the buffered writer, which tends not be used to buffer another buffer… (cherry picked from commit 8ca481e8419c19b6ef9074b68cc028177a507161)
2016-08-31don't change owner/perms/times through file:// symlinksDavid Kalnischkies
If we have files in partial/ from a previous invocation or similar such those could be symlinks created by file:// sources. The code is expecting only real files through and happily changes owner, modification times and permission on the file the symlink points to which tend to be files we have no business in touching in this way. Permissions of symlinks shouldn't be changed, changing owner is usually pointless to, but just to be sure we pick the easy way out and use lchown, check for symlinks before chmod/utimes. Reported-By: Mattia Rizzolo on IRC (cherry picked from commit 3465138575e1fd0d5892d9b6be1ae232eb873460)
2016-08-31don't do atomic overrides with failed filesDavid Kalnischkies
We deploy atomic renames for some files, but these renames also happen if something about the file failed which isn't really the point of the exercise… Closes: 828908 (cherry picked from commit fc5db01bb7d1546944200d197866b0b5c378f100)
2016-06-10don't leak an FD in lz4 (de)compressionDavid Kalnischkies
Seen first in #826783, but as this buglog also shows leaked uncompressed files as well we don't close it just yet. (cherry picked from commit 6f35be91c9e86e463bca7df6eadf05412c7b732c)
2016-06-10do not hang on piped input in PipedFileFdPrivateDavid Kalnischkies
This effects only compressors configured on the fly (rather then the inbuilt ones as they use a library). (cherry picked from commit bdc42211700ef0f6f40e4ef3f362e52d684d70fb)
2016-04-03consistently add APT_OVERRIDE as method markerDavid Kalnischkies
Some methods had it missing, some used the keyword directly, which isn't a problem as it is a cc file, but for consistency lets stick to our macro for now. Git-Dch: Ignore
2016-03-06Fix lzma write support to handle "try again" caseColin Watson
The liblzma-based write code needs the same tweaks that the read code already has to cope with the situation where lzma_code returns zero the first time through because avail_out is zero, but will do more work if called again. This ports the read tweaks to the write code as closely as possible (including matching comments etc.). Closes: #751688
2016-02-10Set seekpos to 0 after reopening files in Seek()Julian Andres Klode
If we just reopened the file, we also need to reset the current seek position when we reset the buffer, otherwise the code will not try to seek to the position given to Skip (from 0), but will try to seek to old offset + the position given to skip. Closes: #812994, #813000
2016-02-04Use some semantically more correct buffer operationsJulian Andres Klode
When writing into the buffer write to free() bytes starting at getend(), instead of buffersize_max bytes at get() -> get() is a read pointer. This makes no difference in practice though, as we reset the buffer before the call, so start = end = 0. Gbp-Dch: ignore
2016-02-04Correctly report write errors when flushing buffered writerJulian Andres Klode
We cannot just return false without setting an error, as InternalWrite does not set one itself.
2016-02-01Do not buffer writes larger than the buffer if possibleJulian Andres Klode
It makes no sense to split a large block into multiple small blocks, so when we have the chance to write them unbuffered, do so.
2016-02-01BufferedWriteFileFdPrivate: Simplify InternalWrite()Julian Andres Klode
We do not need the loop, FileFd::Private() handles this for us. Gbp-Dch: ignore
2016-02-01simple_buffer::write: Use free() instead of maxsize - size()Julian Andres Klode
We want to check whether the amount of free space is smaller than the requested write size. Checking maxsize - size() is incorrect for bufferstart >= 0, as size() = end - start. Gbp-Dch: ignore
2016-01-30fix "Mismatched free() / delete / delete []" in simple_bufferStefan Bühler
2016-01-07CopyFile: Use 64 * 1024 instead of 64000 as buffer sizeJulian Andres Klode
This is a multiple of the page size and thus results in less page faults, speeding up copying. Also, while we're at at, unify all uses of that size in a constant variable APT_BUFFER_SIZE.
2016-01-07FileFd: (native) LZ4 supportJulian Andres Klode
Implement native support for LZ4 compression, using the official lz4 library.
2016-01-03simple_compressor: Provide some accessors for end and freeJulian Andres Klode
This makes code easier to read, and somewhat more correct. Gbp-Dch: ignore
2016-01-03simple_buffer: Allow buffer size to changeJulian Andres Klode
Gbp-Dch: ignore
2015-12-28BufferedFileFdPrivate: Make InternalFlush() save against errorsJulian Andres Klode
Previously, if flush errored inside the loop, data could have already been written to the wrapped descriptor without having been removed from the buffer. Also try to work around EINTR here. A better solution might be to have the individual privates detect an interrupt and return 0 in such a case, instead of relying on errno being untouched in between the syscall and the return from InternalWrite.
2015-12-28BufferedWriter: flushing: Check for written < size instead of <=Julian Andres Klode
This avoids some issues with InternalWrite returning 0 because it just cannot write stuff at the moment.
2015-12-27FileFd: Add a buffered writing modeJulian Andres Klode
This is somewhat experimental right now, and might not work for everyone, so it is on an opt-in basis.
2015-12-27FildFd: Introduce a Flush() function and call it from Close()Julian Andres Klode
The flush function can be used for buffered writers.
2015-12-27FileFdPrivate: Add getter and setter for fieldsJulian Andres Klode
We will soon implement a buffered writing decorator and we will need to forward attribute changes to those.
2015-12-27fileutl: simple_buffer: Add write() and full() methodsJulian Andres Klode
These can be used to implement write buffering Gbp-Dch: ignore
2015-12-27fileutl: simple_buffer: Mark accessors as constJulian Andres Klode
Suggested by David. Gbp-Dch: ignore
2015-12-27FileFdPrivate: Extract SimpleBuffer and mark it as hiddenJulian Andres Klode
Gbp-Dch: ignore
2015-12-26Refactor InternalReadLine to not unroll Size == 0 caseJulian Andres Klode
There is not much point and this is more readable. Gbp-Dch: ignore
2015-12-26Change InternalReadLine to always use buffer.read() return valueJulian Andres Klode
This is mostly a documentation issue, as the size we want to read is always less than or equal to the size of the buffer, so the return value will be the same as the size argument. Nonetheless, people wondered about it, and it seems clearer to just always use the return value.
2015-12-26Get rid of memmove() in our read bufferingJulian Andres Klode
This further improves our performance, and rred on uncompressed files now spents 78% of its time in writing. Which means that we should really look at buffering those.
2015-12-26Use a hardcoded buffer size of 4096 to fix performanceJulian Andres Klode
The code uses memmove() to move parts of the buffer to the front when the buffer is only partially read. By simply reading one page at a time, the maximum size of bytes that must be moved has a hard limit, and performance improves: In one test case, consisting of a 430 MB Contents file, and a 75K PDiff, applying the PDiff previously took about 48 seconds and now completes in 2 seconds. Further speed up can be achieved by buffering writes, they account for about 60% of the run-time now.
2015-12-24Mark all FileFdPrivate classes as hidden1.1.6Julian Andres Klode
Gbp-Dch: ignore
2015-12-23fix new[] vs delete mismatch introduced by b3db9d81David Kalnischkies
And as we are at it lets fix the 'style' issue I introduced with the filefd changes as well. Reported-By: gcc -fsanitize's & cppcheck Git-Dch: Ignore
2015-12-23use a dynamic buffer for ReadLineDavid Kalnischkies
We don't need the buffer that often - only for ReadLine - as it is only occasionally used, so it is actually more efficient to allocate it if needed instead of statically by default. It also allows the caller to influence the buffer size instead of hardcoding it. Git-Dch: Ignore
2015-12-23implement a buffer system for FileFd::ReadLineDavid Kalnischkies
The default implementation of ReadLine was very naive by just reading each character one-by-one. That is kinda okay for libraries implementing compression as they have internal buffers (but still not great), but while working with files directly or via a pipe as there is no buffer there so all those reads are in fact system calls. This commit introduces an internal buffer in the FileFd implementation which is only used by ReadLine. The more low-level Read and all other actions remain unbuffered – they just changed to deal with potential "left-overs" in the buffer correctly. Closes: 808579
2015-12-22parse xz-compression level from configurationDavid Kalnischkies
If we use the library to compress xz, still try to understand and pick up the arguments we would have used to call xz to figure out which level the user wants us to use instead of defaulting to level 6 (which is the default level of xz).
2015-12-22follow dpkg and xz and use CRC64 for xz compressionDavid Kalnischkies
dpkg switched from CRC32 to CRC64 in 777915108d9d36d022dc4fc4151a615fc95e5032 with the message: | This is the default CRC used by the xz command-line tool, align with | it and switch from CRC32 to CRC64. It should provide slightly better | detection against damaged data, at a negligible speed difference.
2015-12-22shuffle compressor-specific code into private subclassesDavid Kalnischkies
This isn't implementing any new features, it is "just" moving code around from FileFd methods which decided on each call how to handle the request by including all logic for all possible compressor backends in the method body to a model in which backend-specifics are implemented in a FileFdPrivate subclass. This avoids a big chunk of #ifdef's and should make it a tiny bit more obvious which backend uses which code. The execution of the idea is slightly uglified by the need to preserve ABI and API which causes liberal befriending. Git-Dch: Ignore
2015-12-19Do not try to read in FileFd::Read() if Size is 0Julian Andres Klode
There's no point trying to read 0 bytes, so let's just not do this and switch to a while loop like in Write(). Gbp-Dch: ignore
2015-12-19Do nothing in FileFd::Write() if Size is 0Julian Andres Klode
Turn the do-while loop into while loops, so it simply does nothing if the Size is already 0. This reverts commit c0b271edc2f6d9e5dea5ac82fbc911f0e3adfa7a which introduced a fix for a specific instance of the issue in the CopyFile() function. Closes: #808381
2015-12-19CopyFile: avoid failing on EOF on some systemsPino Toscano
On EOF, ToRead will be 0, which might trigger on some systems (e.g. on the Hurd) an error due to the invalid byte count passed to write(). The whole loop already checks for ToRead != 0, so perform the writing step only when there was actual data read. Closes: #808381
2015-12-19CopyFile: fix BufSize to a sane valuePino Toscano
Commit e977b8b9234ac5db32f2f0ad7e183139b988340d tries to make BufSize calculated based on the size of the buffer; the problem is that std::unique_ptr::size() returns a pointer to the data, so sizeof() equals to the size of a pointer (later divided by sizeof(char), which is 1). The result is that the CopyFile copies in chunks of 8 bytes, which is not exactly ideal... As solution, declare BufSize in advance, and use its value to allocate the Buf array. Closes: #808381
2015-11-28disable privilege-drop verification by default as fakeroot trips over itDavid Kalnischkies
Dropping privileges is an involved process for code and system alike so ideally we want to verify that all the work wasn't in vain. Stuff designed to sidestep the usual privilege checks like fakeroot (and its many alternatives) have their problem with this through, partly through missing wrapping (#806521), partly as e.g. regaining root from an unprivileged user is in their design. This commit therefore disables most of these checks by default so that apt runs fine again in a fakeroot environment. Closes: 806475
2015-11-28show the group we failed to drop via setgroupsDavid Kalnischkies
This also deals with the unlikely case of groups being mentioned multiple times or if the effective group isn't mentioned at all. In practice, it is a debugging aid through like for #806475. Git-Dch: Ignore
2015-11-27add messages to our deprecation warnings in libaptDavid Kalnischkies
Git-Dch: Ignore
2015-11-19do not use _apt for file/copy sources if it isn't world-accessibleDavid Kalnischkies
In 0940230d we started dropping privileges for file (and a bit later for copy, too) with the intend of uniforming this for all methods. The commit message says that the source will likely fail based on the compressors already – and there isn't much secret in the repository content. After all, after apt has run the update everyone can access the content via apt anyway… There are sources through which worked before which are mostly single-deb (and those with the uncompressed files available). The first one being especially surprising for users maybe, so instead of failing, we make it so that apt detects that it can't access a source as _apt and if so doesn't drop (for all sources!) privileges – but we limit this to file/copy, so the uncompress which might be needed will still fail – but that failed before this regression. We display a notice about this, mostly so that if it still fails (e.g. compressed) the user has some idea what is wrong. Closes: 805069