[SPARK-57710][YARN][TESTS][FOLLOW-UP] Reduce YarnClusterSuite flakiness from CI runner contention

[HyukjinKwon]

What changes were proposed in this pull request?

Follow-up to SPARK-57650, which fixed the deterministic "AM stuck in ACCEPTED" hang in BaseYarnClusterSuite. Two further test-only changes to reduce the remaining flakiness of YarnClusterSuite:

1. Give the single mini NodeManager 8GB (yarn.nodemanager.resource.memory-mb + yarn.scheduler.maximum-allocation-mb) so executor allocation is never starved once the ~1.4GB AM is running.
2. Raise the executor→driver connection retry budget for the launched apps (spark.rpc.io.maxRetries=10, spark.rpc.io.retryWait=2s) so a transient RPC-accept stall does not permanently fail an executor. These are defaults that individual tests can still override via extraConf.

Why are the changes needed?

Even after SPARK-57650, the scheduled Build / Java21 and Build / Java25 master lanes fail in the yarn module roughly 50% of runs (e.g. fork run 28151220075 PASS vs 28151247521 FAIL — same commit, 40s apart). All failures are the same six YarnClusterSuite tests timing out after 3 minutes (The code passed to eventually never returned normally ... handle.getState().isFinal() was false).

From the yarn-app-log / unit-tests-log artifacts, the AM/driver comes up, but the executor (and sometimes the AM) intermittently fail to connect back to the driver's RPC server on localhost (java.io.IOException: Failed to connect to localhost/127.0.0.1:<port>, connection refused). The in-JVM mini RM+NM, the driver subprocess and the AM/executor JVMs all contend for CPU on a single CI runner, so the driver's accept loop occasionally stalls; an executor that loses this race exits after the default 3 connection retries, and the application can then never reach a final state.

Does this PR introduce any user-facing change?

No. Test-only.

How was this patch tested?

YarnClusterSuite was previously failing ~50% of the time. With this change the yarn module job was run 6 times on the fork; all 6 passed, with YarnClusterSuite reporting tests=30, failures=0, skipped=0 (the 6 formerly-failing tests now pass):

• Before (master, failing): apache/spark run 28148781009 (Build / Java21) — 6 YarnClusterSuite timeouts.
• After (this branch): HyukjinKwon/spark runs 28162182834, 28162247111, 28162249819, 28175759262, 28175762257, 28175765871 — yarn job green in all six.

Was this patch authored or co-authored using generative AI tooling?

Yes, Generated-by: Claude Code.

[uros-b]

Note: the new spark.rpc.io.* defaults are set via setProperty BEFORE the loop that copies spark.* JVM system properties, so a -Dspark.rpc.io.maxRetries flag would silently override them; the comment claims only extraConf can override. Moving the two setProperty calls to just before extraConf.foreach removes the ambiguity.