Raise bag_replicator task memory to fit its 3G heap

[kenoir]

What does this change?

The bag_replicator runs a JVM configured for a 3G heap (-Xss6M -Xms2G -Xmx3G) inside a 2048 MB ECS task, so a large bag gets the container OOM-killed part way through replication with nothing written to the logs. That is what dead-lettered ingest 711eef99-8d63-4f3f-907a-6dcdc405d26d (bag b33765170, digitised, v1, 1.5 GB across 3,333 files) on the Azure replicator input queue after three attempts, while the primary and Glacier replicas succeeded. Task memory now matches the bag_verifier sitting next to it at 4096 MB, which covers all three replicators in prod and staging.

Getting that applied meant clearing something else first. stack_prod's lock file still pinned hashicorp/aws 5.13.0 from August 2023, under which any full plan proposed replacing around 150 inline IAM role policies, so applies had to be done resource by resource. Moving to 5.100.0 makes the whole stack safe to apply again.

Where the IAM churn came from and what the upgrade pulls in

The policies were created in 2020 and AWS recorded name_prefix = "terraform-" against them, which the config never sets. 5.13.0 read that as drift and wanted to recreate each one; 5.100.0 treats name and name_prefix as computed, and the churn disappears.

The provider is now pinned ~> 5.0 at the root, since nothing else constrained it, and the lock file carries hashes for both the local and CI platforms. transition_default_minimum_object_size is pinned to varies_by_storage_class on the two working-storage lifecycle rules, because 5.100's new default of all_storage_classes_128K would otherwise silently stop objects under 128K transitioning.

How to test

With terraform 1.6.5 and provider 5.100.0, a full terraform plan in stack_prod should contain only the intended changes: three task definition replacements, three service updates, and two cosmetic lifecycle normalisations. This has already been applied to prod, so a plan there should now be clean. The behavioural check is re-ingesting a bag of comparable size and watching the Azure replicator complete rather than time out and retry.

How can we measure success?

storage-prod_bag_replicator_azure_input_dlq_not_empty stays quiet, and replicator MemoryUtilization stops spiking to the high 80s and taking the task with it. The failed ingest above can be retried.

Have we considered potential risks?

The memory increase triggers a rolling replacement of three services per stack; in-flight messages go back to the queue and are picked up again, and the replicators are idempotent, so the window is not interesting. The provider upgrade is the larger risk, which is why the full plan was read before applying rather than trusting a targeted one.

stack_staging keeps its own lock file at the old provider version. It will want the same terraform init -upgrade when it is next touched, and anyone applying staging before that will still see the IAM policy churn.