Cache FileSystem layout and iterate over it #91
Description
A PR from July 2017 has been lurking unmerged in the old private dlsnode repo. Here's Nemanja's description of it:
This is a WIP patch, with lots of work still to be done in terms of tidying up code/fixing up commits (I want to start having a CI support for the final touches), so it's not yet ready for the general review.
The patch itself introduces four main things:
- B-Tree data structure, managed on the glibc's heap and the auxiliary tools for making that possible
- FileSystemCache used to build the initial view and track the changes of the file system. It uses the B-Tree for storing the fs' data
- FileSystemLayout which uses the range primitives and iterates over the files in the cache, for the given range.
- StorageEngine/StorageEngineStepIterator which now builds and uses the cache to do the iteration, instead of old directory iteration/stat method.
I downloaded the patch of the PR and tried to apply it to this repo, but it failed:
git apply 288.patch
error: patch failed: src/dlsnode/storage/BufferedBucketOutput.d:337
error: src/dlsnode/storage/BufferedBucketOutput.d: patch does not apply
error: patch failed: src/dlsnode/storage/StorageEngine.d:197
error: src/dlsnode/storage/StorageEngine.d: patch does not apply
error: patch failed: src/dlsnode/storage/BucketFile.d:18
error: src/dlsnode/storage/BucketFile.d: patch does not apply
error: patch failed: src/dlsnode/storage/iterator/StorageEngineStepIterator.d:41
error: src/dlsnode/storage/iterator/StorageEngineStepIterator.d: patch does not apply
I don't have time now to look into applying this properly, so will just upload the patch here for posterity:
288.patch.txt
Nemanja said that this is a useful PR that was tested but wasn't merged because we planned to install the DLS nodes on servers with SSDs (an alternative way of speeding up file access). He also mentioned that the BTree implementation in the patch was merged to ocean: sociomantic-tsunami/ocean#210.