Skip to content

[Feature]: ARCBackend — submit + status over ARC 7 REST, token auth #22

Description

@aldbr

User Story

As the WMS consumer,
I want an ARCBackend that submits and monitors jobs on an ARC 7 CE via REST with token auth,
So that the primary grid-CE path (DIRAC's AREX CE) exists on the new contract, validated against
the arc-slurm stack.

Feature Description

Essential half of the AREX port (IC-ADR-001 migration map):

  • ARCBackend: implements Submitter + Monitor against the ARC REST
    API (/arex/rest/1.0) using pyarcrest (evaluate fitness; fall back to httpx against the REST
    spec if it can't do async/bulk cleanly — record the choice).
  • Job description generation from SubmissionSpec (XRSL or ADL — pick one, document why;
    DIRAC builds XRSL), incl. input-sandbox declaration and queue targeting.
  • Bulk semantics: ARC REST supports multi-job operations — submit count copies and poll many
    ids per call; per-id failures ride the outcome maps.
  • State normalisation: ARC state map → JobStatus (port DIRAC's STATES_MAP, incl. the
    auto-kill-on-Hold question — surface as policy, don't hide it in the backend).
  • Token auth per IC-ADR-003: bearer header, audience exposure. Proxy/delegation
    is issue-14.
  • OutputRetriever (destructive=False): list the ARC session dir, stream every member into
    dest file-by-file, return the JobOutput manifest (.stdout/.stderr/.files/.log).
    Bounded materialisation enforced inline: byte ceiling mid-stream, file-count ceiling,
    per-transfer timeout, path containment (../absolute members rejected) — IC-ADR-001 §4.
  • Cancellable.kill / Purgeable.purge (DIRAC killJob/cleanJob): bulk, per-id outcomes.
  • LoadReporter.counts() from the CE info endpoint.
  • SupportsLiveDiagnostics.diagnostics(): the diagnose/errors endpoint — the one genuinely
    independent, non-destructive pre-completion fetch.
  • Proxy/delegation auth: create delegation (CSR → sign with the
    proxy chain → upload), renew, AlwaysIncludeProxy parity where tokens+proxy coexist. Add the
    config/proxy/ configuration axis to the arc-slurm stack (ADR-002 §7) to test it.
  • Fetch-twice regression test proving re-fetchability (destructive=False) — the counterpart
    of HTCondor's one-shot test.
  • Phase 2 wiring: contract suite (submit/status subset) against the arc-slurm stack
    (markers remote and arc).

Definition of Done

  • Full contract suite (all capabilities) green against arc-slurm, both token and proxy
    configs
  • count > 1 submission bundles one Submission with per-copy failure reporting
  • Unknown/expired ids → JobStatus.UNKNOWN (never dropped)
  • State map unit-tested; Hold handling decided and documented
  • Fetch-twice test green; containment/ceiling tests green (hostile filenames, oversized
    output)
  • Delegation renewal covered by a test (short-lived test-CA proxy)
  • arc-slurm stack gains config/proxy/; manifest updated
  • Registered via intercede.backends as arc

Alternatives Considered

  • Wrapping the arc CLI tools — spawns processes, brittle parsing; REST is first-class in ARC 7
    and DIRAC's AREX already proves it.

Related Issues

Additional Context

ARC floor is 7.x (IC-ADR-002 §8). DIRAC reference: `AREXComputingElement.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions