This infrastructure as code (IaC) project installs Harbor on a single node Kubernetes cluster. It uses Talos Linux as an operating system for running Kubernetes and Proxmox VE as hypervisor. The provisioning is done with OpenTofu.
Kubernetes cluster features:
- Kubernetes v1.36.3
- no kube-proxy
- Cilium v1.20.0 as Container Network Interface (CNI)
- without kube-proxy
- with L2 loadbalancer support
- with Gateway API support
- with Egress gateway support
This Kubernetes cluster is meant to be used in a test or home lab environment.
It's meant to be a standalone, turnkey solution: so after installing, you will have Harbor available and ready to use immediately. For making this happen, I had to do some design decisions:
- IaC: every piece of infrastructure is declarative
- Proxmox VE: installation of this hypervisor itself is a manual task, but everything else can be done fully declarative using APIs and a Terraform/OpenTofu provider
- Talos Linux/Kubernetes: both can be configured fully declarative using APIs and Terraform/OpenTofu providers
- Local storage for Kubernetes applications on the node: data storage needs to happen without other infrastructure dependencies like NFS or Ceph. Providing storage for a Kubernetes cluster can be rather complex if it needs to be highly available, and not everyone has a NFS share available or runs a Ceph cluster like me. So I choose to statically provision the volumes on the node with Talos Linux and configured local PersistentVolumes. This way it can be installed and run anywhere. Plus, I consider the data which will be stored here as ephemeral, as the container images can be easily pulled or reproduced again.
- Certificate authority (CA): bootstrapping and running a standalone CA is necessary to issue TLS certificates.
The whole PKI is generated by OpenTofu itself, so there is no manual bootstrap step and no CLI tooling involved:
a
tofu applyon a fresh clone produces a working CA. The root CA private key is only kept in the OpenTofu state and never published to the cluster, which only gets the intermediate CA it signs with.
- Proxmox VE with some resources available (default: 2 CPUs, 8GB RAM, 275GB disk space)
- OpenTofu installed locally
- A DNS entry for your Harbor domain, resolvable from inside the cluster (see Certificate Authority)
- Docker Hub account
First clone the repo. The provisioning with OpenTofu needs to be done in two steps:
- Create the VM on Proxmox hypervisor and install Kubernetes
- Install Harbor and all other applications in the Kubernetes cluster
Go to proxmox subdirectory and create a configuration.auto.tfvars file using the example:
$ cp configuration.auto.tfvars.example configuration.auto.tfvars Then add the configuration as it suits your needs to the new file.
Create the virtual machine, install and configure Talos Linux:
$ tofu init
$ tofu plan
$ tofu applyThen grab the kubeconfig and store it in some appropriate space (or merge with your already existing kubeconfig file):
$ tofu output -raw kubeconfig > ~/.kube/harbor-configIn the next step you will need to reference this kubeconfig file in your configuration.auto.tfvars of the OpenTofu
kubernetes module.
In kubernetes subdirectory create a configuration.auto.tfvars file using the example:
$ cp configuration.auto.tfvars.example configuration.auto.tfvars Then apply your configuration to the new file.
Install Harbor and all other applications into the Kubernetes cluster:
$ tofu init
$ tofu plan
$ tofu applyAfter everything was provisioned with OpenTofu, Harbor is available locally under the IP
address and domain which you configured earlier. You can now log in with username admin and your
harbor_admin_password which you specified in configuration.auto.tfvars.
The certificate authority is generated and owned by OpenTofu, see kubernetes/certificate_authority.tf:
- a root CA, whose private key stays in the OpenTofu state and is never published to the cluster
- an intermediate CA signed by it, which is handed to step-ca running in the cluster
- an ACME provisioner on step-ca, which cert-manager uses through a
ClusterIssuerto issue and renew the Harbor TLS certificate
Nothing here is interactive and there are no passwords to keep in sync, so the CA converges on every tofu apply.
The intermediate CA is renewed automatically by a later apply once it comes close to its expiry, which is transparent
to clients because only the root certificate needs to be trusted.
Back up your OpenTofu state. It holds the only copy of the root CA private key. Losing it means a new root CA, which has to be distributed to every trust store again. Consider enabling state encryption, as the state also holds the intermediate CA private key and the Harbor admin password.
Everything is published through a single Gateway in the network namespace, which is published under
cilium_load_balancer_ip_range_start. It terminates TLS for Harbor on port 443 and serves nothing but the ACME
HTTP-01 challenges on port 80.
The challenge is solved through that same Gateway, so your Harbor domain has to resolve to its address from inside the
cluster — usually by adding the record to the DNS server of your LAN, which you need anyway to use Harbor as a pull
through cache. Until that record exists, tofu apply still succeeds, but the Harbor certificate stays pending;
cert-manager issues it as soon as the name resolves.
You can check on it with:
$ kubectl -n network get gateway,certificate,order,challengeWrite out the root CA certificate:
$ cd kubernetes
$ tofu output -raw root_ca_crt > root_ca.crtAdd it to the trust store of your local machine, for example with Step CLI:
$ step certificate install root_ca.crtOther Kubernetes clusters that use this Harbor instance as a pull through cache need the same certificate in the trust store of their container runtime. On Talos Linux this is part of the machine configuration:
machine:
files:
- content: |
-----BEGIN CERTIFICATE-----
... contents of root_ca.crt ...
-----END CERTIFICATE-----
permissions: 0o644
path: /etc/ssl/certs/ca-certificates
op: appendThe ACME directory of step-ca is available in the cluster, so other workloads can get certificates from the same CA:
$ tofu output -raw step_ca_acme_directory_url
https://step-certificates.security.svc.cluster.local/acme/acme/directoryThe step-ca ClusterIssuer solves the HTTP-01 challenge on the HTTP listener of the Gateway, which only accepts
routes from its own namespace. Certificate resources using it therefore have to live in the network namespace, or
the challenge has to be solved with a solver of your own.
The objective is to have Harbor available as container image cache eventually. So the last step is to configure the image cache for your Kubernetes nodes. As this is specific to the container runtime and registry you are using, I need to exclude instructions here. For those using Talos Linux for running their cluster, this is straight forward and well documented.
- Talos Linux documentation
- Talos Linux Image Factory
- Terraform providers/modules
- Baremetal provisioning
- Kubernetes
- Applications
- Certificate Authority
- Helm charts
- CA configuration