New skip list

[marvin-j97]

Reopening of #131

Stuff to do:

0. Make it compile
1. Make buffer size configurable instead of being a const generic parameter
2. Insertions of duplicates need to be rejected (not panic), to make batch writes with duplicate keys work
3. Evaluate memory overhead vs crossbeam skiplist
4. Evaluate read and write latency vs crossbeam skiplist
5. Try in a full system (heavily cached) benchmark
6. Remove crossbeam-skiplist
   (7. Model test against crossbeam-skiplist(

[matthew-mcallister]

There is a quicker way to get an exponential distribution if you are OK with base 2 instead of base e.

const MASK: u32 = !((1 << (MAX_HEIGHT as u32 - 1)) - 1);
let x = random_u32 | MASK;
let height = x.trailing_zeros() + 1;

[matthew-mcallister]

This might not be worth addressing since user range queries already bounce around the skipmap in sorted order anyways...

... but, for what it's worth, you could solve this problem with minimal overhead by storing the height on each node in a clever way. Every node is aligned to 8 bytes, so the least significant 3 bits of each pointer in the tower are essentially free space. There are a few ways you could encode the height in those bits. Then you can iterate over the arena in linear order.

[matthew-mcallister]

I would be happy to contribute here if you want to share what you had in mind for benchmarking in points (3)-(5). Assuming you'll permit me to circle back to custom comparison functions after this PR is merged.

[marvin-j97]

I would be happy to contribute here if you want to share what you had in mind for benchmarking in points (3)-(5). Assuming you'll permit me to circle back to custom comparison functions after this PR is merged.

Benchmarking memory overhead should be pretty straightforward: just load tons of small values to see what the relative overhead of towers etc. is.

Benchmarking simple operations should be doable using criterion.

And full system benchmarks can be done using https://github.com/marvin-j97/rust-storage-bench/tree/v1, especially using the YCSB C workload with a small data set.