Feat: Add Orchestration Layer for Evaluation

[ritaneves]

Goal:

Gym needs a shared way to launch evaluation across teams, otherwise risking having a messy non-sharable launch/deployment setups.

Reasoning:

Our customers (e.g Nemotron) are used to the concept of launching jobs in Slurm. Gym assumes local execution, no layer that allows for execution on Slurm. Right now, benchmarks are being executed via custom, often cluster-specific sbatch scripts which are not OSSable.
Beyond this, supporting the concept of jobs allows for easier caching and running on the background.
The ultimate goal is to continue supporting this by having this layer orchestrated in Gym, with support for local and Kubernetes in the near the future.

Action Items:

Defining ownership.
Architecture discussion, bearing in mind we want to keep evaluations running in the simplest manner possible (i.e. not have a gym eval and a gym launcher eval, which will confuse people). Options for integration:
NeMo-Skills
Nemo-run
Nemo-RL
NEL-Next
New tool based on NEL-Next / NEL (focused on orchestration only) -> Gym Launcher

[gchlebus]

Requirements to keep in mind:

1. Orchestration layer should ensure reproducibility. Output artifacts should contain full information required to reproduce a given run including the launch command.
2. Given output artifacts, it should be easy to tell if a given run used a validated config (e.g., one from EFB repo).