Extended-length and variable-length registers: design summary

[glopesdev]

TLDR

• We propose adding support for large variable-length payloads in Harp, via a single flag bit in the binary protocol that extends Length to 32-bit and the trailing Checksum to CRC-32, plus the device.yml schema and the operational conventions needed to make this work end-to-end.
• The proposal is split across four focused issues so each stakeholder can engage with the parts that touch their stack:
  • Add ExtendedLength flag to the binary protocol #218
  • Clarify Read request conventions in Device.md #219
  • Schema for variable-length registers #220
  • Device operation conventions for extended-length registers #221
• Most design decisions are locked as a working baseline; a smaller set of operational and logging questions is held open for SRM.

Why now

The need is concrete: DAC stimulus waveforms, firmware blobs, SD-card uploads. The 254-byte regular-format ceiling has been a friction point for years. The previous ExtendedLength v1 specification was unimplemented across every device and client and was removed earlier this year at #209. The successor "Blob" thread at #186 has been open for around six months without convergence, accumulating several incompatible counter-proposals.

This proposal is the result of design work that resolved the main points of contention by separating concerns that the original thread had conflated. The result is compact enough to land at the SRM rather than continue iterating in the original thread.

The proposal at a glance

The proposal introduces an ExtendedLength flag bit in the MessageType field of the 8-bit binary protocol. When the flag is set, the Length field is interpreted as a 32-bit little-endian integer and the trailing Checksum is interpreted as a CRC-32 with the IEEE 802.3 polynomial. Variable-length registers gain a maxLength attribute in device.yml. Whether a register's frames use regular or extended framing is derived mechanically from the schema, not declared. Device behaviour for extended-length registers, including the reply contract and a tiered core-support model, is specified in a separate Device Registers and Operation extension.

Four issues:

• Add ExtendedLength flag to the binary protocol #218: binary protocol diff. The flag bit, the 32-bit Length, the CRC-32.
• Clarify Read request conventions in Device.md #219: clarification that Read requests carry empty payloads and never set the flag.
• Schema for variable-length registers #220: device.yml schema for variable-length registers; folds with Support variable-length registers in device.yml register specifications #116.
• Device operation conventions for extended-length registers #221: Device operation conventions for extended-length registers; the largest scope.

The architectural reframing

The original "Blob" framing conflated three different concerns: a "Blob" message type at the wire level, a category of registers in the schema, and behavioural conventions for those registers. Every counter-proposal in the original thread had to touch all three at once, and the design space became combinatorially large.

The current proposal factors the work into three independent layers:

• Binary protocol: a flag bit, plus how it changes the Length and Checksum field encodings. Nothing about registers.
• Schema: a maxLength attribute, plus the rule that encoding strategy is derived from the size. Nothing about device behaviour.
• Device operation: the reply contract, polymorphism reformulation, tiered support, hub forwarding, DUMP and MUTE_RPL interactions, and recovery from incomplete writes. Nothing about wire format or schema attributes.

Concretely this means that the term "blob register" is no longer formally defined anywhere. Registers have a type, a length mode (length or maxLength), and a maximum size; whether a register requires extended framing is derived mechanically. Each issue can be reviewed independently of the others. Cross-references are captured in each issue's Related Issues section. The 32-bit binary protocol is explicitly out of scope since it is not yet ratified and will be handled separately if and when it is.

Key design decisions

Each item is a working baseline that downstream issues build on. The reasoning is in the relevant issue's Motivation and Alternatives sections.

• Flag bit, not new message type. Bit 4 of MessageType (mask 0x10). The flag preserves clean parser dispatch and keeps the existing Read / Write / Event semantics intact.
• 32-bit Length, not 24-bit. A clean contiguous field rather than reusing the removed ExtendedLength bytes. Aesthetic alignment toward a future 32-bit protocol.
• CRC-32 (IEEE 802.3 polynomial). Detects byte permutations and structured burst errors that a U8 or U32 byte sum misses. Available out of the box in the .NET System.IO.Hashing package, in the Pico SDK as DMA sniffer mode 0x0, and via standard table-based implementations elsewhere.
• No mandatory cryptographic digest. Software SHA-256 on ATxmega is on the order of 50 seconds per 16 MB, which fails the "implementable on small MCUs without hardware acceleration" constraint. CRC-32 is the realistic baseline.
• Receipt-CRC echo, not full payload echo. Write replies for extended-length writes carry the CRC of the request, not the written payload. Saves the upload-bandwidth doubling and removes the device-side buffering requirement, at the cost of one logging-stream property; see the trade-offs section.
• maxLength attribute, not a sentinel for length. A new attribute mutually exclusive with the existing integer length. Keeps JSON Schema validation clean and existing fixed-length declarations untouched.
• Encoding strategy derived from schema, not declared. The schema does not have an "is-blob" attribute. Codegen derives whether a register uses extended framing from length × sizeof(type) or maxLength × sizeof(type).
• Tiered core support. Every core MUST recognize the flag and reject frames it cannot handle. Only cores whose device.yml declares any register that needs extended framing implement the full parser, the 32-bit Length arithmetic, and CRC-32 computation.

Accepted trade-offs

The proposal accepts three trade-offs that downstream tooling will feel.

• Single-stream logging completeness is lost for write replies on extended-length registers. Today a device's reply stream contains the full payload of every write echoed back. Under the new contract, extended-length write replies carry only the CRC-32 of the request. Tools that need the full payload for offline analysis must log host-side requests separately and pair them with device replies by matching CRC values. Read replies and event messages on extended-length registers continue to carry full payloads, so the property is preserved for those message types.
• Variable-length per-register streams are not shape-aligned. The current Harp logging pattern demuxes messages by Address and bulk-loads each per-register stream into an N-dimensional matrix on read. Variable-length registers break the shape-alignment assumption. This is flagged as an open question with explicit decision points in Schema for variable-length registers #220 because the right resolution involves cross-stack alignment.
• Hub forwarding becomes constrained. Hubs cannot modify any field inside the CRC-32 scope, which includes the Port field today. The working position is that hubs are transparent for extended-length frames; this needs explicit confirmation since hub behaviour is currently underspecified across the rest of the Device Registers and Operation specification.

Engagement by stack

This proposal touches multiple stacks unevenly. Each section below indicates which issues are most relevant.

• harp-tech/core.atxmega (ATxmega-side firmware). Most directly affected by the resource constraints baked into the proposal: the no-mandatory-crypto rule and the tiered core-support split were chosen with ATxmega-class devices in mind. The Drawbacks section of Add ExtendedLength flag to the binary protocol #218 also calls out a specific ATxmega-side detail we could not verify during drafting, namely whether the AU variants' hardware CRC peripheral supports the IEEE 802.3 polynomial directly. Review of this detail by maintainers familiar with the AU CRC peripheral would be particularly valuable. The main reads are Add ExtendedLength flag to the binary protocol #218 for the binary protocol diff and Device operation conventions for extended-length registers #221 for the operational conventions.
• harp-tech/core.pico (Pico-side firmware). The Pico SDK DMA sniffer mode 0x0 is the IEEE 802.3 polynomial, so hardware-accelerated CRC is already available on both RP2040 and RP2350. The main read is Device operation conventions for extended-length registers #221 for the device-operation conventions, especially the reply contract and the overrun/timeout/cancellation/atomicity sub-section. Existing prototyping work that picked up the original Blob discussion can transition cleanly to the locked baseline.
• bonsai-rx/harp and adjacent Bonsai-side tooling. Most affected by the single-stream logging trade-off and the variable-length stream shape-alignment question. The main reads are Device operation conventions for extended-length registers #221 logging implications section and Schema for variable-length registers #220 logging compatibility unresolved question. Both raise concrete decisions for the Bonsai-side workflow.
• harp-tech/harp-python and other client codegen pipelines. The main reads are Schema for variable-length registers #220 for the schema additions and codegen impact, plus Add ExtendedLength flag to the binary protocol #218 for the framing rules. Each codegen pipeline already handles fixed length; the variable-length path is a small extension rather than a rewrite.

Open questions for SRM

The following items are held open for SRM. Each has a working position stated in the relevant issue.

• DUMP and MUTE_RPL interactions for extended-length registers. Working position: exclusion-by-default from DUMP with future per-register opt-in; full suppression of receipt-CRC echo under MUTE_RPL.
• Hub forwarding rules. Working position: hubs are transparent for extended-length frames. The broader question of hub behaviour for non-extended frames is currently underspecified and may need addressing in the same SRM session.
• Overrun, timeout, cancellation, and atomicity behaviour. Working positions stated in Device operation conventions for extended-length registers #221. The sub-questions on minimum acceptable timeout, MUST vs SHOULD on atomicity, and whether to specify cancellation now or defer are individually flagged.
• Variable-length logging strategy. The biggest open question because it touches downstream tooling assumptions directly. The expanded unresolved question in Schema for variable-length registers #220 describes three candidate paths: per-register stream format that retains random access, separate per-register index file, or explicit downstream-tooling responsibility without spec prescription.
• Version bundling with Use a single version number in device repositories #145 and Distribute Harp device GUI applications via .NET tool packages and a unified launcher #217. Both are breaking. The decision is whether to bundle them with this proposal into a single major version, or release them separately.

What to do with this discussion

This discussion is a navigator, not a substitute for the issue bodies. A stakeholder who reads only this discussion will be oriented to the design and able to identify which issues most concern their stack, but the actual review and approval require reading the relevant issue body. The four issues are short and focused for exactly this reason.

#186 will be closed and redirected to the new set of issues once the SRM-side conversation has started.

[glopesdev]

Feedback from SRM:

• Consider whether checksum / error-checking is even necessary given we rely on serial protocol / bulk transfers / TCP, etc. Another move could be to drop error-checking altogether.
• Clarify READ and EVENT messages for long registers (READ still returns the full register content as an extended-length message)
• Discussion around whether it would be possible to include content of long-registers in DUMP (and whether SHA / fingerprinting would be useful there)
• Error recovery from out-of-sequence bytes with very large extended messages: how long does it take to recover using CRC-32 and how best to drive the recover message to avoid. What are we really gaining from CRC-32 / checksum if we cannot guarantee recovery out-of-sequence?
• If nothing is gained then we should just drop all checksum / error-correcting codes and return empty payload to an extended-length write
• Consider the ability of "muting" the echo per-register (return an empty payload) on the device schema.