ublkpp

A high-performance C++23 library providing RAID0/1/10 support for Linux's userspace block (ublk) driver

🚀 Features

RAID Support: Full implementation of RAID0 (striping), RAID1 (mirroring), and RAID10 (stripe of mirrors)
RAID1 Resilient Bitmap: Memory-efficient dirty tracking (4 KiB page tracks 1 GiB data)
Thin-Aware, Resumable Resync: Per-scenario copy modes (blind / compare-skip / zero-detect) persisted in the superblock; a cleanly-stopped resync resumes where it left off
Hot Device Replacement: Swap devices in degraded RAID1 arrays without downtime
Lock-Free I/O Path: Read/write operations use lock-free algorithms (x86-64/ARM64)
Factory-Based API: File-backed disks and RAID compositions through supported factory functions
Coroutine I/O: Single-event-loop, CQE-driven coroutine pipeline
Comprehensive Testing: High test coverage with unit and functional (fio-driven) tests
Modern C++: Built with C++23, leveraging std::expected for error handling
Production Ready: Thread-safe, handles degraded modes

🏃 Quick Start

Prerequisites

Linux kernel with ublk support (5.19+)
Conan 2.0+
CMake 3.22+
C++23 compatible compiler (GCC 13+, Clang 17+)

Build Library

git clone https://github.com/szmyd/ublkpp
cd ublkpp
./prepare_v2.sh
conan build -s:h build_type=Debug --build missing .

Build Options

# Release build
conan build -s:h build_type=Release --build missing .

# With coverage
conan build -s:h build_type=Debug -o ublkpp/*:coverage=True --build missing .

# With sanitizers (address or thread)
conan build -s:h build_type=Debug -o ublkpp/*:sanitize=address --build missing .
conan build -s:h build_type=Debug -o ublkpp/*:sanitize=thread --build missing .

🏗️ Architecture

Project Structure

ublkpp/
├── include/ublkpp/       # Public headers
│   ├── drivers.hpp       # File-backed disk factory
│   ├── raid.hpp          # RAID factories and helpers
│   ├── target.hpp        # ublk target interface
│   └── lib/              # Base disk subclassing API
├── src/
│   ├── driver/           # File-backed backend implementation
│   ├── lib/              # Core ublk_disk base classes
│   ├── metrics/          # I/O and RAID metrics
│   ├── raid/             # RAID logic (bitmap, superblock)
│   └── target/           # ublkpp_tgt
└── example/              # Sample applications

Core Abstractions

ublk_disk: Base class for all block devices
disk_handle: Shared ownership handle for disks and RAID composites
make_fs_disk(): File/block-backed disk construction
make_raid0_disk() / make_raid1_disk(): RAID composition factories
raid0::* / raid1::*: Free-function helpers for topology and mirror management
ublkpp_tgt: Exposes devices to kernel via ublk

💾 RAID Features

RAID0 (Striping)

Configurable stripe size (default: 128 KiB)
Distributes data across devices for performance
Linear capacity aggregation

RAID1 (Mirroring)

Key Features:

Two-way mirroring with dirty bitmap tracking
Degraded mode operation (single device failure)
Hot device replacement via swap_device()
Read routing round-robins

Bitmap Efficiency:

4 KiB pages track 32 KiB chunks (default)
Memory footprint: ~0.4% of capacity (e.g., 8 MiB for 2 TB)
SuperBitmap optimization for fast initialization

Resync Features:

Background resync with per-region I/O coordination
Lock-free write tracking: resync yields only for chunks that conflict with an in-flight write
Two-phase conflict check with shadow completion log to close the mid-copy race window
Copy mode selected by recovery scenario, deciding per 4 KiB page:
- BLIND: full copy -- known-divergent dirty sets (a degraded leg's outage writes) and unverified fresh legs
- CHECK: read + memcmp the destination, rewrite only divergent pages -- power-loss self-heal, re-added legs
- ZERO_TEST: zero-detect the source, skip unallocated regions with no destination read -- fresh-leg rebuilds on thin devices (requires assume_clean)
Mode persists in the superblock: a cleanly-stopped resync resumes where it left off; an unclean stop falls back to a full CHECK pass
New arrays run an md-style initial sync unless constructed with assume_clean (see below)
Configurable delay intervals

assume_clean (per-device opt-in on make_raid1_disk() / swap_device(), --assume_clean on the example): asserts a genuinely-fresh leg (no superblock) reads back zero for never-written blocks, e.g. a newly-provisioned thin volume or sparse file. Enables ZERO_TEST rebuilds and skips the new-array initial sync (both legs already read identically), preserving thin provisioning. Leave unset for recycled/raw disks -- the initial sync then makes the mirrors read deterministically.

RAID10 (Stripe of Mirrors)

RAID0 striping across RAID1 pairs
Combines performance and redundancy
Requires even number of devices (min: 4)

🖥️ Example Application

The ublkpp_disk application demonstrates all RAID capabilities with a single target.

Build and Run

# Build release version
conan build -s:h build_type=Release --build missing .

# Load kernel module
sudo modprobe ublk_drv

# Create backing files
fallocate -l 2G file1.dat
fallocate -l 2G file2.dat
fallocate -l 2G file3.dat
fallocate -l 2G file4.dat

# Launch RAID10 device (sparse files read zero: --assume_clean skips the new-array initial sync)
sudo ublkpp/build/Release/example/ublkpp_disk --raid10 file1.dat,file2.dat,file3.dat,file4.dat --assume_clean

Usage Examples

# Single device (loop mode)
sudo ublkpp_disk --loop /dev/sdb

# RAID0 (striping)
sudo ublkpp_disk --raid0 /dev/sdc,/dev/sdd --stripe_size 262144

# RAID1 (mirroring; a brand-new array runs an initial sync to make the mirrors identical)
sudo ublkpp_disk --raid1 /dev/sde,/dev/sdf

# RAID10 (4+ devices; sparse/thin backing reads zero, so skip the initial sync)
sudo ublkpp_disk --raid10 file1.dat,file2.dat,file3.dat,file4.dat --assume_clean

# Recover existing device
sudo ublkpp_disk --device_id 0 --raid1 /dev/sde,/dev/sdf

Verify Device

$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
...
ublkb0      259:3    0    4G  0 disk

# Make Filesystem
$ sudo mkfs.xfs /dev/ublkb0
$ sudo mount /dev/ublkb0 /mnt

🛠️ Development

Code Style

Indentation: 4 spaces
Line Length: 120 characters
Pointers: Left alignment (Type* ptr)
Standard: C++23
Headers: #pragma once

Naming Conventions

Element	Convention	Example
Public API types (`include/ublkpp/`)	`lower_snake_case`	`ublk_disk`, `disk_handle`, `ublkpp_tgt`
Public API factories (free functions)	`make_<thing>`	`make_fs_disk()`, `make_raid1_disk()`
Internal classes (`src/`)	`PascalCase`	`SuperBlock`, `Bitmap`, `Raid1Disk` (impl), `MirrorDevice`
Functions / methods	`snake_case`	`async_iov()`, `prepare()`, `swap_device()`
Members	`_snake_case`	`_device`, `_dirty_bitmap`
Constants	`k_snake_case`	`k_page_size`
Macros / Enums	`SCREAMING_SNAKE_CASE`	`UBLK_IO_OP_WRITE`

Driver and RAID array implementations are not part of the public surface; consumers construct opaque disk_handles via make_*_disk() factories and compose them.

Workflow

# 1. Write code
# 2. Write tests (see Testing section)
# 3. Format code
./apply-clang-format.sh

# 4. Build and test
conan build -s:h build_type=Debug --build missing .

Error Handling

Uses std::expected<T, std::error_condition> pattern:

using io_result = std::expected<size_t, std::error_condition>;

io_result write_data(uint64_t addr, uint32_t len) {
    if (auto res = device->sync_iov(UBLK_IO_OP_WRITE, iov, 1, addr); !res) {
        DLOGE("Write failed at {:#x}: {}", addr, res.error().message());
        return res;
    }
    return len;
}

🧪 Testing

Test Organization

src/<component>/tests/
├── test_*_common.hpp      # Shared test utilities
├── simple/                # Basic functionality tests
├── failures/              # Error handling tests
├── bitmap/                # RAID1 bitmap tests
└── superblock/            # Superblock I/O tests

Running Tests

# Tests run automatically during build
conan build -s:h build_type=Debug --build missing .

# Coverage report
conan build -s:h build_type=Debug -o ublkpp/*:coverage=True --build missing .
# View: build/Debug/coverage_html/index.html

# Thread sanitizer
conan build -s:h build_type=Debug -o ublkpp/*:sanitize=thread --build missing .

# Address sanitizer
conan build -s:h build_type=Debug -o ublkpp/*:sanitize=address --build missing .

Writing Tests

Framework: Google Test (GTest) with GMock

#include "test_raid1_common.hpp"

TEST(Raid1, YourTestName) {
    auto device_a = CREATE_DISK_A(TestParams{.capacity = 2 * Gi});
    auto device_b = CREATE_DISK_B(TestParams{.capacity = 2 * Gi});

    EXPECT_TO_WRITE_SB(device_a);
    EXPECT_TO_WRITE_SB(device_b);

    auto raid = ublkpp::make_raid1_disk(uuid, device_a, device_b);

    // Test logic...
    EXPECT_EQ(expected, actual);
}

📦 Dependencies

Core Dependencies

sisl v14+: Logging, options, metrics, HTTP server
ublksrv: ublk driver interface
isa-l: RAID acceleration primitives
boost: UUID generation
liburing: io_uring support

Optional Dependencies

stdexec: C++ sender/receiver framework — provided transitively via the sisl conan package
fio: Functional I/O testing (optional; tests skip gracefully if absent)

Build Tools

Conan 2.0+
CMake 3.22+
clang-format (code formatting)
gcovr (coverage reporting)

📚 Documentation

Development & Contributing

CHANGELOG.md: Version history and release notes
CLAUDE.md: Development guidelines and workflows
docs/error_codes.md: RAID async_iov error code reference (EIO vs EAGAIN matrix)
docs/functional_testing.md: Functional test procedures
Linux ublk Documentation: Kernel driver details

🤝 Contributing

Contributions are welcome! Please:

Follow the code style (run ./apply-clang-format.sh)
Add tests for new functionality
Update CHANGELOG.md and version in conanfile.py
Ensure all tests pass with sanitizers
Submit pull requests against main

📄 License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

Primary Author: Brian Szmyd

Links:

Name		Name	Last commit message	Last commit date
Latest commit History 260 Commits
.claude		.claude
.github/workflows		.github/workflows
.jenkins		.jenkins
3rd_party		3rd_party
cmake		cmake
docs		docs
example		example
include/ublkpp		include/ublkpp
src		src
test_package		test_package
.clang-format		.clang-format
.codecov.yml		.codecov.yml
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
README.md		README.md
apply-clang-format.sh		apply-clang-format.sh
conanfile.py		conanfile.py
gcovr.cfg		gcovr.cfg
prepare_v2.sh		prepare_v2.sh
tsan.supp		tsan.supp

Folders and files

Latest commit

History

Repository files navigation