Clear index state information of FormerIndexes based on the index name rather than subspace key#4024
Open
alecgrieser wants to merge 2 commits intoFoundationDB:mainfrom
Open
Conversation
…ame rather than subspace key There was a bug in the `removeFormerIndex` method which is called when a store is opened. It previously was clearing out the index state information (which states whether the index is built or not) based on whether the `FormerIndex`'s subspace key. However, this is incorrect, as the index state space is keyed by index name (see FoundationDB#514 about switching this to be by subspace key). This means that the old index state will persist in the database, and we won't properly clear it. This fixes things by using the former index name instead. Note that the former name was added after other fields in the `FormerIndex`, and so it may not be reliably set. Newer creations of the `FormerIndex` (including `RecordMetaDataBuilder::removeIndex`) should preserve that information, but that's not guaranteed. The change therefore needs to handle the fact that the subspace key may not be set. I've elected to update the logic so that we just give up on clearing out the index state if the name is not set, though that is technically a behavior change. In particular, if the user uses the default subspace key (which is the index name), we could attempt to guess at what the former name was by looking at the key. I decided this was probably a bad idea, though, as you could have a pair of indexes that were named `"index_1"` and `"index_2"` but had the index subpsace keys of `"index_2"` and `"index_1"`. It seems a little unlikely, but the cost of incorrectly leaving index state information is relatively low (the main cost being that we may not be able to mark an index on new types that re-uses the name as `READABLE`), whereas the danger of getting that wrong is higher (we could end up incorrectly marking an unbuilt index as `READABLE`). This fixes FoundationDB#515. It also probably suggests that we should look at FoundationDB#4019 with some skepticism, as we are now actively using the former index name data for something.
Collaborator
|
I somewhat prefer #514, though I am willing to accept arguments about that being a more significant change than is called for to fix all this. |
Collaborator
Author
|
The way I view it, we would probably want both. The current situation is dicey, in that we end up clearing out the wrong location in the index state space, and that seems like something we need to address. If we addressed #514, we'd only be able to do that on a new format version, and so I think we'd need this (or something like it) on at old format versions. We did try and address #514 with #662, and we got bogged down a bit in the details. I do think it would be a good change, though, if we could make it work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was a bug in the
removeFormerIndexmethod which is called when a store is opened. It previously was clearing out the index state information (which states whether the index is built or not) based on whether theFormerIndex's subspace key. However, this is incorrect, as the index state space is keyed by index name (see #514 about switching this to be by subspace key). This means that the old index state will persist in the database, and we won't properly clear it.This fixes things by using the former index name instead. Note that the former name was added after other fields in the
FormerIndex, and so it may not be reliably set. Newer creations of theFormerIndex(includingRecordMetaDataBuilder::removeIndex) should preserve that information, but that's not guaranteed. The change therefore needs to handle the fact that the subspace key may not be set. I've elected to update the logic so that we just give up on clearing out the index state if the name is not set, though that is technically a behavior change. In particular, if the user uses the default subspace key (which is the index name), we could attempt to guess at what the former name was by looking at the key. I decided this was probably a bad idea, though, as you could have a pair of indexes that were named"index_1"and"index_2"but had the index subpsace keys of"index_2"and"index_1". It seems a little unlikely, but the cost of incorrectly leaving index state information is relatively low (the main cost being that we may not be able to mark an index on new types that re-uses the name asREADABLE), whereas the danger of getting that wrong is higher (we could end up incorrectly marking an unbuilt index asREADABLE).This fixes #515. It also probably suggests that we should look at #4019 with some skepticism, as we are now actively using the former index name data for something.