AWS deployments should have a test deployment stage

[dra27]

Before pushing to AWS for opam.ocaml.org and ocaml.org, the container should be tested on a non-live service to ensure that it starts. If that stage fails, then the deployment itself doesn't proceed, and the website(s) remain at the previous deployment (with the issue picked up on the firehose).

[tmcgilchrist]

For AWS Fargate we should look into the health checks support for load balancers and use that to progressively update the containers. The deployer would notify a new image is available and the AWS machinery swaps in new containers running that image (testing using the health checks) before committing to using the new version. It does also handle rollback automatically on failing health checks.

If we choose not to use AWS, we could build this support into deployer itself.

[mtelvers]

In testing this afternoon, Fargate does not stop the existing containers until the new ones pass the load balancer health check. Therefore, when a faulty image is deployed, the original image remains running.