Add portable YARN resource cost tool

[gerashegalov]

Part of #264.

Summary

• add a provider-neutral YARN allocation ledger using authoritative ResourceManager allocation and terminal events
• support DefaultResourceCalculator and dominant shares across memory, vcores, GPUs, and custom resources
• add EMR, Dataproc, and on-prem input/node-classification adapters with offline, catalog, and optional live pricing
• preserve heterogeneous node-class seconds, completeness gates, pricing provenance, and baseline/test comparisons
• include sanitized fixtures, Python 3.10-3.12 CI, documentation, and a licensed standalone ZIP builder
• retain the original EMR command as a compatibility entry point while exposing a versioned portable CLI

Verification

• python3 -m unittest discover -s yarn-resource-cost -p "test*.py" (36 tests)
• Python 3.10.19, 3.11, and 3.12 compile and unit-test checks
• standalone ZIP build and extracted-bundle smoke tests under Python 3.10
• git diff --check
• private offline replay of historical EMR job 144 exactly matched the established calculator, container/resource seconds, node-equivalent seconds, expression, and completeness output; no private logs or identifiers are included

Follow-up staging

This PR establishes the common engine and initial documented adapters. Follow-up PRs under #264 can extend native Dataproc pricing/log export workflows and queue-specific FairScheduler configuration evidence without changing the accounting contract.

[greptile-apps]

Greptile Summary

The PR adds a provider-neutral YARN allocation and cost ledger with EMR, Dataproc, and on-premises adapters.

• Reconstructs authoritative container lifetimes and resource shares from YARN logs
• Adds catalog and live pricing, completeness gates, and baseline/test comparisons
• Includes portable CLI packaging, documentation, fixtures, tests, and Python CI

Confidence Score: 4/5

The PR should not merge until duplicate baseline comparison keys are detected or represented without silently dropping applications.

The default application-name comparison can collapse repeated baseline applications into one dictionary entry, producing an incomplete or incorrectly paired comparison report.

Files Needing Attention: yarn-resource-cost/yarn_resource_cost.py

Important Files Changed

Filename	Overview
yarn-resource-cost/yarn_resource_cost.py	Adds the portable CLI and comparison pipeline, but duplicate baseline comparison keys silently overwrite applications.
yarn-resource-cost/yarn_job_cost_core.py	Adds provider-neutral parsing, lifecycle reconstruction, calculator detection, and dominant-resource share accounting with conservative completeness handling.
yarn-resource-cost/yarn_job_cost_adapters.py	Adds provider input materialization, node-class mapping, archive safeguards, and catalog pricing gates.
yarn-resource-cost/yarn_job_cost_eventlog.py	Adds Spark rolling-event-log normalization and completeness metadata.
yarn-resource-cost/calculate_yarn_job_cost.py	Preserves the compatibility entry point while delegating ledger behavior to the portable core and retaining reporting and live-pricing logic.
yarn-resource-cost/package_yarn_job_cost.py	Builds and validates a licensed standalone distribution containing the CLI, support modules, tests, fixtures, and documentation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  E[Spark event logs] --> S[Application selection and metadata]
  Y[YARN RM/NM logs] --> L[Container allocation ledger]
  S --> A[Application accounting]
  L --> A
  N[Node classification] --> A
  A --> G{Ledger complete?}
  G -- No --> I[Evidence and warnings only]
  G -- Yes --> P[Apply catalog or live pricing]
  P --> O[JSON, CSV, and comparison output]
  I --> O

Loading

_{Reviews (1): Last reviewed commit: "Make EMR CLI test region-independent" | Re-trigger Greptile}