Skip to content

Latest commit

 

History

History
339 lines (270 loc) · 14.1 KB

File metadata and controls

339 lines (270 loc) · 14.1 KB

Instructions for LLMs

Vision

mountin mounts anything. Disk images, archives, filesystems from dead operating systems - if it ever existed, mountin should be able to open it.

The approach: spin up tiny VMs (guests) that use real kernels to read formats, expose the contents over 9P protocol to the host. Linux 6.12 for broad modern and late legacy coverage, Linux 2.6 for older compatibility, NetBSD for UFS/ZFS, eventually esoteric kernels for truly obscure formats.

Philosophy

Pure over pragmatic. We get things right rather than get things done. This project aims to be technically excellent - a showcase of what careful engineering looks like, not typical "make it work" LLM-assisted code.

Branches are the enemy. Special cases, divergent code paths, workarounds - these are technical debt. If something needs a special case, the abstraction is wrong. We simplify continuously without losing functionality.

Declarative source of truth. The markdown frontmatter defines everything: build dependencies, format detection rules, documentation, the website. One source, many outputs.

No root required. Neither at build time nor runtime. All build dependencies are isolated in containers (podman). The final tools run unprivileged.

Weed the garden. Continuous refinement. When something becomes inconsistent or awkward, fix it immediately rather than accumulating cruft.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Frontend Clients                         │
│  FUSE, GVFS, KIO, 7zip plugin, PeaZip, WASM, Windows driver...  │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                         libmountin                              │
│  - Recursive format detection (disk → partition → fs → archive) │
│  - Guest selection (best kernel for format + host arch)         │
│  - Transport orchestration (9P, future: NFS, etc.)              │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                          Guests                                 │
│  Linux 6.12 │ Linux 2.6 │ NetBSD 10 │ (future: AROS, Haiku...)  │
│  Minimal VMs with busybox + 9P server, run via QEMU             │
└─────────────────────────────────────────────────────────────────┘

libmountin is the brain. Clients ask it to "mount this and give me a transport endpoint". It detects the format, picks the right guest, starts QEMU, returns a socket. The client connects and does I/O.

Build System

The Python package mountin orchestrates everything via podman containers.

Concepts

Catalogue: All *.md files in src/mountin/ are parsed. YAML frontmatter defines build metadata, markdown body becomes documentation.

Paths: File paths map to logical catalogue paths. bin/linux/busybox/index.md becomes catalogue path bin/linux/busybox. Metadata inherits down the tree.

provides/requires: Each path can provide outputs and require inputs. The build system resolves the dependency graph and builds in order.

---
title: BusyBox
requires:
  - sources/busybox-1.36.1.tar.bz2
output_platforms:
  x86_64-linux-musl:
    provides:
      - bin/${MOUNTIN_TARGET_PLATFORM}/busybox
  aarch64-linux-musl:
    provides:
      - bin/${MOUNTIN_TARGET_PLATFORM}/busybox
---

env inheritance: Environment variables defined in parent index.md files cascade to children. Variables use ${VAR} syntax.

docker: prefix: provides: docker:builder/compiler/rust means the target builds a container image. requires: docker:... means it needs that image.

Commands

# Show what can be built
mountin-build outputs

# Show dependency graph for a target
mountin-build deps bin/x86_64-linux-musl/busybox

# Build a target (and all dependencies)
mountin-build build bin/x86_64-linux-musl/busybox

Build flow

  1. Dockerfile in the path's directory → build container image
  2. If provides includes files → run container with /host/build mounted
  3. Container writes outputs to /host/build/<provides-path>
  4. Build system verifies outputs exist

Directory Structure

src/mountin/                # Python package (the build system)
├── bin/                    # Binary build definitions
│   ├── detect/             # Format detection CLI tool
│   ├── linux/              # Linux-hosted binaries (busybox, etc.)
│   ├── netbsd/             # NetBSD-hosted binaries
│   └── qemu/               # Guest VM builds
│       ├── linux/          # Linux guests (2.6, 6.12)
│       └── netbsd/         # NetBSD guests
├── builder/                # Build infrastructure (compilers, disk tools)
├── data/                   # Test data generation (filesystem images)
├── docs/                   # Documentation (also defines formats)
│   └── format/             # Format detection rules live here
│       ├── fs/             # Filesystems (ext4, ntfs, etc.)
│       ├── pt/             # Partition tables
│       ├── arc/            # Archives (tar, zip, etc.)
│       └── disk/           # Disk image formats (qcow2, vdi, etc.)
├── lib/                    # Library builds
│   ├── format/             # Compiles detection rules → msgpack
│   └── mountin/            # Rust library (libmountin)
├── sources/                # Source tarball definitions
├── catalogue.py            # Loads markdown → catalogue dict
├── runner.py               # Executes builds via podman
└── main.py                 # CLI entry point

build/                      # Build outputs (gitignored)
├── bin/                    # Compiled binaries by target triple
├── guest/                  # Reusable guest kernels, roots and boot files
├── lib/                    # Compiled libraries by target triple
├── data/                   # Generated test filesystem images
├── sources/                # Downloaded source tarballs
└── catalogue.json          # Compiled catalogue snapshot

tests/                      # pytest tests
scripts/                    # Development scripts (venv, coverage, etc.)

Format Detection

Detection rules are defined in docs/format/ frontmatter:

---
title: ext4
detect:
  - offset: 0x438
    type: le16
    value: 0xef53
    then:
      - offset: 0x45c
        type: le32
        mask: 0x40
        op: "&"
        value: 0x40
---

The lib/format/compile.py script reads these from the catalogue and generates build/lib/format.bin (msgpack). The Rust library embeds this at compile time.

Detection is recursive: detect disk image format → detect partition table → detect filesystem → detect archive inside → etc.

Format Documentation vs Guest Capabilities

Separation of concerns: Format documentation and guest capabilities are distinct.

Format docs (docs/format/) describe what formats look like:

  • Detection rules (magic bytes, offsets, structures)
  • Feature flags (ext4 with encryption, NTFS with compression)
  • Container structure (what children a format can have)
  • Pure documentation - no mention of implementations or guests

Guest manifests declare what a guest can mount:

  • Supported formats and features
  • Resource requirements (RAM, disk)
  • Available transports (9P, SSH, NFS)
  • Architecture constraints

Selection engine (runtime) matches detected formats to available guests:

  • Input: detection tree, available guests, client constraints
  • Output: viable mounting options, ranked
  • Handles composite mounting (different guests for different subtrees)

This separation means:

  • Format docs stay pure and reusable
  • New guests just declare capabilities, don't modify format docs
  • Selection logic is centralized, not scattered

Recursive Detection and Container Readers

Detection returns a tree structure:

{
  format: "disk/vmdk",
  children: [
    {
      index: 0,
      format: "pt/gpt",
      children: [
        {index: 0, format: "fs/ext4", features: ["extent", "64bit"]},
        {index: 1, format: "fs/ntfs", features: []},
        {index: 0, format: "fs/swap"},  // same index = same partition, different detection
      ]
    }
  ]
}

Key concepts:

  • index identifies a child within its parent (partition number, track number)
  • Same index with different formats = multiple valid interpretations (ISO + Joliet + RockRidge)
  • Path through indices gives unique addressing: /0/gpt/1/ntfs
  • Features are detected properties that affect guest selection

Container readers enumerate children for container formats:

  • arc/gzip → single decompressed stream
  • pt/mbr, pt/gpt → partition spans (offset + length)
  • disk/qcow2 → virtual block device (needs translation)

Readers register by format path, matching the catalogue structure.

Target Naming

Paths follow Rust target triple order: {arch}-{os}[-{env}]

Examples:

  • bin/x86_64-linux-musl/busybox - Linux static binary
  • bin/x86_64-linux-gnu/detect - Linux dynamic binary
  • bin/x86_64-netbsd/9d - NetBSD (no env suffix)
  • lib/x86_64-darwin/libmountin.dylib - macOS
  • lib/x86_64-windows-gnu/mountin.dll - Windows

Environment variables:

  • MOUNTIN_BUILD_PLATFORM - Build machine platform (for example x86_64-linux)
  • MOUNTIN_BUILD_ARCH, MOUNTIN_BUILD_OS - Components of the build platform
  • MOUNTIN_BUILD_JOBS - Build parallelism; operational, not part of artefact identity
  • MOUNTIN_TARGET_PLATFORM - Platform targeted by a provider instance
  • MOUNTIN_TARGET_ARCH, MOUNTIN_TARGET_OS, MOUNTIN_TARGET_ENV - Output platform components
  • MOUNTIN_LIBC - libc environment (musl, gnu) - only for Linux

Frontmatter env contains semantic variables and participates in artefact identity. Use execution_env for operational controls such as job limits and cache locations; these are passed to the provider but deliberately excluded from its input hash. Injected MOUNTIN_BUILD_* variables and MOUNTIN_CACHE_DIR are likewise operational. Injected MOUNTIN_TARGET_* variables are semantic and are hashed. Catalogue-only context such as MOUNTIN_RELEASE_REF and MOUNTIN_SOURCE_KIND is available for resolution but is not exported to every provider.

Providers declare output_platforms independently from build_platforms. The former describes the artefacts they produce; the latter constrains which machines can run the build. One catalogue path may therefore have several provider instances without conflating host and target architecture. Default output selection follows architecture compatibility (x86_64 also selects i386); --output-arch and --output-platform provide explicit selection without changing provider identity or buildability.

Guest versions and build toolboxes

Guest paths identify the upstream operating-system generation, because that is what determines kernel, driver and filesystem compatibility. Use the upstream release or series version where one exists. For rolling projects without a release version, use the date of the newest upstream commit included when the fork last diverged or was synchronized, in YYYY-MM-DD form.

Compiler images follow the same separation:

  • builder/compiler/<os> is a reusable, source-independent host bootstrap.
  • builder/compiler/<os>/<version> is the toolbox for one upstream guest generation. It owns the matching compiler, SDK/sysroot and runtime closure.
  • Target-specific variants may live beneath that version when required.
  • Guest assembly consumes the toolbox; it does not publish build trees, sysroots or SDKs as runtime outputs.
  • guest/<platform>/<version> contains reusable, inspectable operating-system components shared by fixture builders and appliance assembly.
  • bin/qemu/... contains only files consumed by the emulator or launcher. Intermediate objects belong in the toolbox image or provider cache.

Current State

Mature build system with cut-down appliances. What works:

  • Build system resolves dependencies and builds in containers
  • Linux 2.6 and 6.12 guests boot, mount filesystems, serve 9P
  • NetBSD 10 guest builds (needs manual 9P init)
  • Format detection library compiles rules and detects many formats and gets better with each catalogue update.
  • 9pfuse client connects and mounts

What's next:

  • libmountin: guest selection and orchestration (not just detection)
  • Consistent target naming
  • More formats, more guests
  • Frontend clients (FUSE wrapper, then plugins, then everything else)

Guest Conventions

QEMU creates hardware. The -m flag passed to the runner script selects what runs inside the guest. sh gives a debug shell, 9p starts the 9P server.

Hardware config must be identical between modes or debugging becomes impossible. This is why we resist special cases - debug and production paths must match.

Testing

make test      # Run pytest
make coverage  # Generate coverage report

Tests are functional (pytest style, not unittest classes). Mocks indicate poor isolation - fix the code instead.

Build and run

The user will build the project, given the time required to run the build and the size of the outputs.