Enabling non-sequential writes via xrootd

[MoAly98]

Hi,

I'm working on a data processing pipeline in particle physics that runs on the coffea-casa analysis facility. The pipeline filters large datasets and writes the results as ROOT files to an xrootd storage using uproot.

I found that writes to endpoints running xrootd-multiuser fail with error 3013 ("operation not supported") when the writer seeks back to an earlier position in the file:

OSError: File did not write properly: [ERROR] Server responded with an error: [3013] Unable to write [/store/user/maly/write_test.root](https://coffea.casa/store/user/maly/write_test.root); operation not supported

The server logs show:

260212 16:32:38.523317 1741752 multiuser_Write: Out-of-order writes not supported while running checksum. /store/user/maly/fsspec_rw_test.bin
260212 16:32:38.523323 1741752 ofs_write: cms-jovy.2930:76@c2415.shor.hcc Unable to write /store/user/maly/fsspec_rw_test.bin; operation not supported
260212 16:32:38.523328 1741752 cms-jovy.2930:76@c2415.shor.hcc Xrootd_Response: sending err 3013: Unable to write /store/user/maly/fsspec_rw_test.bin; operation not supported

The issue is that the ROOT file format requires this write pattern, data is written first, then the writer seeks back to the beginning to patch the file header with final sizes and offsets that aren't known until all data is written. There's no way to produce a valid ROOT file with purely sequential writes. uproot (via fsspec-xrootd) opens files in read-write mode and performs these seeks, which xrootd-multiuser rejects because it's computing checksums inline and expects bytes in order.

I verified this by testing against the same endpoint (xrootd-local.unl.edu). Sequential writes work fine:

import XRootD.client as xrdclient

out_path = "root://xrootd-local.unl.edu:1094//store/user/maly/write_test.txt"

with xrdclient.File() as f:
    status, _ = f.open(out_path, xrdclient.flags.OpenFlags.DELETE | xrdclient.flags.OpenFlags.NEW)
    if not status.ok:
        raise RuntimeError(f"Open failed: {status.message}")
    status, _ = f.write(b"hello from xrootd\n")
    if not status.ok:
        raise RuntimeError(f"Write failed: {status.message}")

print(f"Wrote to {out_path}")

# Read back to verify
with xrdclient.File() as f:
    status, _ = f.open(out_path, xrdclient.flags.OpenFlags.READ)
    if not status.ok:
        raise RuntimeError(f"Open failed: {status.message}")
    status, data = f.read()
    print(f"Read back: {data.decode()}")

But writes with an offset fail with the same 3013 error:

import XRootD.client as xrdclient
out_path_offset = "root://xrootd-local.unl.edu:1094//store/user/maly/xrd_offset_test.bin"

with xrdclient.File() as f:
    status, _ = f.open(out_path_offset, xrdclient.flags.OpenFlags.DELETE | xrdclient.flags.OpenFlags.NEW)
    print(f"XRD open: {status.ok} {status.message}")
    status, _ = f.write(b"\x00" * 100)
    print(f"XRD sequential write: {status.ok}")
    status, _ = f.write(b"patched", offset=50)
    print(f"XRD offset write: {status.ok} {status.message}")

Same story with fsspec-xrootd — sequential wb mode works, but r+b with a seek does not:

import fsspec

# SUCCESS: fsspec_xrootd sequential write (no seek)
out_path_seq = "root://xrootd-local.unl.edu:1094//store/user/maly/fsspec_seq_test.txt"
with fsspec.open(out_path_seq, "wb") as f:
    f.write(b"sequential write test\n")
print(f"Sequential write OK: {out_path_seq}")

# FAIL: fsspec_xrootd with seek (r+b mode, like uproot uses)
out_path_rw = "root://xrootd-local.unl.edu:1094//store/user/maly/fsspec_rw_test.bin"
try:
    # First create the file
    with fsspec.open(out_path_rw, "wb") as f:
        f.write(b"\x00" * 100)
    # Then open for read-write and seek
    with fsspec.open(out_path_rw, "r+b") as f:
        f.seek(50)
        f.write(b"patched")
    print(f"Seek+write OK: {out_path_rw}")
except Exception as e:
    print(f"Seek+write FAILED: {e}")

Plain text files and xrdcp of locally-written ROOT files to the same endpoint work without issues. As discussed with @jthiltges, a possible path forward would be for xrootd-multiuser to skip or defer checksum computation when it encounters out-of-order writes. Can random access writes be allowed for these setups?

[MoAly98]

From discussions in IRIS-HEP slack:

• Relevant PR on this from @djw8605 Handle non-sequential writes in MultiuserFile #61
• Some criticism of writing and seeking over network from Jim in this dicussion, highlighting the performance loss form doing such operations. This was pointed out by Nick Smith.
• Nick also points out that it might be better to use simplecache:: in the redirectory to finalize the file locally and then upload it once complete

[MoAly98]

It is also worth noting we can write to EOS via xrootd :)