Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform

[aicam]

Problem

Today bin/k8s on main ships a single Helm chart that targets a local / on-premise Kubernetes cluster only. Our AWS (EKS) deployment runs the same Helm chart, but everything underneath it — the EKS cluster, S3 bucket, IAM/IRSA roles, Elastic IP, DNS, node pools, cert-manager — is created by hand and lives nowhere in the repo. None of that substrate is captured as code.

The result: standing up Texera on a fresh AWS account is an undocumented, manual prerequisite checklist that drifts and is hard to reproduce. We'd like to unify the local and AWS deployment stories so there's one documented, reproducible path from "empty cluster (or empty AWS account)" to "running Texera," while keeping the Helm chart itself cloud-agnostic.

Scope of "substrate"

The Helm chart already covers the application. The open question is only how we provision/document the substrate it runs on:

• EKS cluster + managed node pools
• IAM roles / IRSA for service accounts
• S3 bucket (LakeFS backend), Elastic IP, DNS, cert-manager / Let's Encrypt

Options considered

1. Helm only

Deploy the chart into a cluster that already exists. No AWS substrate creation.

Pros

• Simplest — one helm install
• Identical for local and AWS — fully cloud-agnostic
• Nothing extra to learn or maintain in the repo
• Keeps the Apache PR scoped to just the chart

Cons

• Doesn't create the cluster, S3 bucket, IAM, EIP, node pools, or cert-manager — all manual on a new AWS account
• New-AWS-account UX = a long prerequisite checklist done by hand
• Substrate stays undocumented-as-code → drift, hard to reproduce

2. eksctl

AWS CLI that provisions an EKS cluster + node pools + IAM/IRSA + add-ons from a small YAML, via CloudFormation. Helm still deploys on top.

Pros

• Purpose-built for EKS; minimal config for a working cluster
• Handles IRSA, node pools, and add-ons natively
• "New AWS account → cluster" in ~one command
• Config file is committable and reproducible

Cons

• AWS-only — no use for local or other clouds
• Doesn't manage non-cluster resources well (S3 bucket, EIP, DNS) — you still script those
• CloudFormation-based: slower, less granular state control
• Adds an AWS-specific tool to a repo that's otherwise portable

3. Terraform

Declarative IaC that can provision everything: EKS cluster, S3, IAM, EIP, DNS, node pools — and across clouds. Helm deploys on top (or via the Helm provider).

Pros

• Most reproducible & reviewable (full state, plan/apply, diffs)
• One tool for all substrate, not just the cluster
• Cloud-agnostic — same tool for AWS / GCP / local
• Can even invoke Helm itself, so one apply does substrate + deploy

Cons

• Heaviest to set up and maintain (modules, remote state, locking)
• Steeper learning curve; more moving parts
• State management is its own operational burden
• Overkill if you only ever target one EKS cluster

Our current position: eksctl

We're currently leaning toward eksctl, because:

• GCP and Azure are low priority for now — the cloud-agnostic advantage of Terraform doesn't pay off yet, so we'd rather not take on its maintenance cost speculatively.
• We prefer easier maintenance over easier deploying. eksctl keeps the AWS-specific config small and declarative while letting the Helm chart stay fully portable, rather than introducing Terraform modules + remote state + locking that we'd have to operate.
• eksctl handles the genuinely fiddly EKS-specific parts (IRSA, node pools, add-ons) natively, and the cluster config is committable.

Regardless of the choice, a small amount of non-cluster substrate (S3 bucket, EIP, DNS) would still be scripted alongside.

We'd love your input

Which approach should we standardize on for upstreaming AWS deployment into bin/k8s? Please vote with a reaction on this post and share your reasoning in the comments — especially if you deploy Texera (or similar) on a cloud we haven't prioritized:

• 👍 — Helm only
• 🚀 — eksctl (our current lean)
• ❤️ — Terraform

[chenlica]

Thanks for starting the discussion. We want to support both cloud-based and on-prem deployments. Please confirm if eksctl supports both.

@zuozhiw @shengquan-ni @Yicong-Huang @sshiv012 @parshimers Any input from you?

[sshiv012]

I'm still getting familiar with the AWS side of the deployment, but here's what stood out reading through bin/k8s.

@chenlica standard eksctl does not support on-prem. It's the AWS-only CLI for provisioning Amazon EKS clusters as @aicam mentioned. What should be common for both flavors of deployment is Helm. Not sure if we use EKS ANYWHERE or just barebones k8s on prem, if its eks anywhere then we can maybe reuse it.

One thing I wasn't sure was fully captured in the framing: it might help to separate the application chart from the substrate, because as far as I can tell only the substrate is really in question.
On the chart side, texera-helm already looks closer to cloud-agnostic than I expected, the persistence templates only create hostPath PVs when the storage class is local-path (minio-persistence.yaml, postgresql-persistence.yaml), so other classes fall through to dynamic provisioning, and LakeFS already targets blockstore.type: s3 rather than MinIO-specific APIs. If I'm reading that right, moving to EBS + real S3 is mostly a values change, not a template change.

That leaves the substrate i.e cluster, node pools, IRSA, S3, EIP, DNS, cert-manager as the only layer the eksctl-vs-Terraform question is actually about.

On eksctl: I agree it's the right tool for the cluster layer (node groups, Pod Identity). My one hesitation is that it doesn't get us all the way to "one path to reproduce the deployment" — as @aicam noted, S3/EIP/DNS are out of scpe for it, so if those stay manual we may partly recreate the undocumented-setup problem we're trying to solve. That's where Terraform's plan is appealing: it warns before it deletes a data bucket or changes IAM, detects drift if someone clicks around in the console, and the code doubles as documentation.

But I think it really comes down to who owns the deployment:

• Project maintainers (keep-it-simple, rotating contributors): eksctl + Helm + a committed bootstrap script is probably the lighter path.
• General adopters (integrating into a real org/projects): Terraform is the better thing to offer — portable, composable into existing IaC, self-documenting, and usually the tool they already speak.

So maybe it isn't either/or. We can consider (eventually) shipping a thin, optional Terraform module (or even just documented .tf for S3/IAM/IRSA plus a reference eksctl cluster config), so maintainers stay on the simple path while adopters get something that drops into their stack.

[parshimers]

eksctl is just for Amazon's distribution of k8s. Which is fine, as stated GCP/Azure are way less popular, and on prem is always unique. The important thing is that there is no hard binding to AWS and that's already the case.

I think having a well documented set of steps for setting up everything on AWS is a very good idea. IAM in particular is a nightmare without knowing exactly what each service actually needs. People running on prem will have their own particular setups that will need interpretation anyway. For example they probably have some object store, but each one will have it's own configurations for buckets and so on. It's not possible for us as developers to say how that's done on each system a priori. There should be guidelines (e.g. what addresses and ports the various services need, what object stores are known to work, etc.) but not specifics. Maybe an example with some particular set of services would be a good intermediate step, because then the administrator would be able to draw analogies to their environment.

I have gone through Milvus's setup on AWS and that was about as straightforward as it could be. It really would have taken me a very long time to figure out things on my own without that walkthrough.