feat: objectstore drainer

[SimonRichardson]

Requires #20291 to land first.

Note

We require #20291 to land first as it re-writes the objectstore tests to better handle the test worker in the objectstore tests. Thus makes it a lot cleaner to apply the fixes to the tests on top of that.

---

This wires up the draining of models from the file backed objectstore to the s3 backed objectstore. This sits on top of all the work that took place at the sprint; the objectstore fortress, drainer, and facade. Once we spot a draining in process, we use the raw underlying types:

• The hash based filestore, with provides only raw access to the disk
• The s3 client which only pushes files directly from file to s3 compatible object stores.

There is complexity in ensuring the ordering of the process. Ensuring that each model is drained from the files to s3, then importantly flushing all the current existing objectstore workers, so that they pick up the new s3 configuration. Once that is done, we complete the process and unlock the guard.

If there is an error during the process, we report the error and require manual intervention.

It is required that all files are backed up before performing this process, as once a file has been drained, it is removed from the prior location.

Note

This currently doesn't drain the controller of agents, that'll follow on in another PR. Trying to moving into HA after the draining process, it will fail as there are no file backed object stores.

Warning

This doesn't allow s3 to s3 storage, we require changes to how we store the underlying data. This requires modifications to the controller charm.

---

This pull request introduces several updates to enhance object store management and expand functionality for retrieving model namespaces. Key changes include improvements to the object store drainer configuration, the addition of a new GetModelNamespaces method, and updates to object store metadata handling.

Object Store Enhancements:

• Updated objectstoredrainer.ManifoldConfig in manifolds.go to include additional dependencies and services, such as GetControllerService, GetGuardService, and NewDrainerWorker, to improve object store draining capabilities. (cmd/jujud-controller/agent/machine/manifolds.go, cmd/jujud-controller/agent/machine/manifolds.goR751-R765)
• Added a new ObjectStoreFlusher interface for flushing object store workers. (core/objectstore/objectstore.go, core/objectstore/objectstore.goR81-R86)
• Updated the SelectFileHash function to replace the private selectFileHash function, ensuring consistent hash selection across object store implementations. (internal/objectstore/base.go, [1]; internal/objectstore/fileobjectstore.go, [2]; internal/objectstore/s3objectstore.go, [3]

Model Namespace Retrieval:

• Added a GetModelNamespaces method to the State interface and implemented it in the State struct, allowing retrieval of all model namespaces from the database. (domain/controller/service/service.go, [1] [2]; domain/controller/state/state.go, [3]
• Extended unit tests to cover the GetModelNamespaces method, including scenarios where no namespaces are found. (domain/controller/service/service_test.go, [1]; domain/controller/state/state_test.go, [2]

Schema and Dependency Updates:

• Modified the database schema for object_store_drain_phase_type to correct phase type ordering and added a foreign key constraint for phase_type_id. (domain/schema/controller/sql/0014-objectstore-metadata.sql, domain/schema/controller/sql/0014-objectstore-metadata.sqlL43-R51)
• Updated dependencies in manifolds_test.go to reflect new requirements for the object-store-drainer manifold. (cmd/jujud-controller/agent/machine/manifolds_test.go, [1] [2]

These changes collectively improve the robustness and functionality of the system, particularly in managing object store operations and retrieving model-related data.

QA steps

Setup s3 (I'm using minio, but anything will suffice, even another Juju model with an s3 compatible application):

docker run -d \
   -p 9000:9000 \
   -p 9001:9001 \
   --user $(id -u):$(id -g) \
   --name minio1 \
   -e "MINIO_ROOT_USER=ROOTUSER" \
   -e "MINIO_ROOT_PASSWORD=CHANGEME123" \
   -v ${HOME}/.local/minio/data:/data \
   quay.io/minio/minio server /data --console-address ":9001"

minio is now community addition, so the browser version is basic.

$ docker exec -it <container> bash
$ mc alias set foo http://localhost:9000 ROOTUSER CHANGEME123
$ mc admin info foo
$ mc admin accesskey create foo
# Note down the access keys

Bootstrap:

Note

Don't use localhost for the container address, otherwise it can't access it. Your own address from ip a

$ juju bootstrap lxd test
$ juju add-model default
$ juju deploy ubuntu

Take note of the controller uuid, you can inspect that model in the minio CLI/dashboard.

juju controller-config "object-store-type=s3" "object-store-s3-endpoint=<endpoint>" "object-store-s3-static-key=<key>" "object-store-s3-static-secret=<secret>"

Watch the logs for object store draining completed successfully

Make sure that the blobs have been removed:

$ juju ssh -m controller 0
$ juju_object_store_contents

There should only be the controller left.

Also check that the objectstore workers were flushed:

  object-store:
    inputs:
    - agent
    - trace
    - object-store-services
    - lease-manager
    - object-store-s3-caller
    - api-remote-caller
    - upgrade-database-flag
    report:
      workers:
        77f5809c-41c1-4a5c-8b7d-19f3ba7cd9c2:
          report:
            namespace: 77f5809c-41c1-4a5c-8b7d-19f3ba7cd9c2
            path: /var/lib/juju/objectstore/77f5809c-41c1-4a5c-8b7d-19f3ba7cd9c2
            rootBucket: juju-dcbe59b4-2752-4bb3-8e7d-e54c7cf603a1
            type: s3-object-store
          started: "2025-07-18 16:01:35"
          state: started
        a262007b-ae56-4812-8f4a-e029f3afd3e9:
          report:
            namespace: a262007b-ae56-4812-8f4a-e029f3afd3e9
            path: /var/lib/juju/objectstore/a262007b-ae56-4812-8f4a-e029f3afd3e9
            rootBucket: juju-dcbe59b4-2752-4bb3-8e7d-e54c7cf603a1
            type: s3-object-store
          started: "2025-07-18 16:01:35"
          state: started
        controller:
          report:
            namespace: controller
            path: /var/lib/juju/objectstore/controller
            rootBucket: juju-dcbe59b4-2752-4bb3-8e7d-e54c7cf603a1
            type: s3-object-store
          started: "2025-07-18 16:01:35"
          state: started

[manadart]

Some nits, but this is very cogent, and works perfectly.

[manadart]

Quite long isn't it?

[SimonRichardson]

Yeah, probably, I really want to get juju/worker#41 in at some point, which gives us the best of both worlds.

[SimonRichardson]

All our workers restart after 10 seconds... I'm happy to revisit this as a whole.

[hpidcock]

Nice. Few little bits.

[hpidcock]

Feels like the object store model should compute and store both when a object is first stored.
Even though we only index by sha384, having the other values precomputed and stored in metadata can prove useful. Like here.

[SimonRichardson]

That's a big change, happy to make it. I'll do in a follow up.

[hpidcock]

Nice to have, but can we please get some statistics here please?
Objects Drained.
Objects To Be Drained.
Current Object Draining (hash, size).

This will help very much in seeing if the drainer is stuck or just slowly progressing.

[SimonRichardson]

Yes, good plan.

[SimonRichardson]

/merge