Skip to content

cloudandthings/terraform-aws-open-metadata

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Terraform AWS OpenMetadata Module

License: MIT Terraform OpenTofu AWS Provider pre-commit Tests

Description

Reusable Terraform / OpenTofu module for deploying OpenMetadata on AWS.

OpenMetadata is an open-source data catalog and metadata governance platform. This module provisions a production-ready OpenMetadata installation backed by managed AWS services:

AWS Service Role
Amazon EKS Kubernetes cluster running the OpenMetadata application
Amazon RDS (PostgreSQL) OpenMetadata metadata store
Amazon OpenSearch Full-text search and indexing
AWS Secrets Manager Credential storage for RDS and OpenSearch
AWS KMS Encryption at rest for all data-plane services
AWS ALB (via Load Balancer Controller) Ingress / HTTP(S) access to the UI
Amazon Route 53 Optional DNS record pointing to the ALB
AWS ACM (Private CA) Optional TLS certificate for the ingress

Submodules

Each concern is separated into an independently toggle-able submodule:

Submodule What it creates
cluster EKS cluster, managed node group, KMS encryption for K8s secrets
addons AWS Load Balancer Controller and External Secrets Operator (Helm releases + IRSA)
data RDS PostgreSQL instance, OpenSearch domain, and Secrets Manager secrets for both
app OpenMetadata Helm release, K8s namespace, IRSA role, ALB ingress, optional TLS
access EKS access entries for CI/CD pipelines or human operators
dns Route 53 CNAME record pointing to the ALB

Prerequisites

Before using this module you need:

  • VPC with at least two private subnets across different AZs (used by EKS, RDS, and OpenSearch).
  • KMS key — a customer-managed key used to encrypt EKS secrets, RDS, OpenSearch, and Secrets Manager secrets.
  • IAM permissions boundary ARN — required by the module for every IAM role it creates.
  • Approved EKS node AMI ID — the managed node group requires an explicit AMI ID. Use AL2023_ARM_64_STANDARD for Graviton (m7g.*) instances or AL2023_x86_64_STANDARD for x86.
  • AWS credentials configured locally (e.g. via AWS_PROFILE or environment variables).

Note: If you already have an EKS cluster, RDS instance, or OpenSearch domain, you can reuse them — see Bring Your Own Resources below.


Usage

module "openmetadata" {
  source  = "cloudandthings/open-metadata/aws"
  version = "~> 1.0"

  # Naming
  name        = "my-openmetadata"
  name_prefix = "myom"          # Short prefix — OpenSearch domain names are limited to 28 chars

  # AWS context
  region     = "us-east-1"
  account_id = "123456789012"
  tags       = { Environment = "production" }

  # Security
  iam_role_permissions_boundary = "arn:aws:iam::123456789012:policy/my-boundary"
  kms_key_id                    = "arn:aws:kms:us-east-1:123456789012:key/..."

  # Networking
  vpc_id             = "vpc-0abc1234"
  private_subnet_ids = ["subnet-0aaa", "subnet-0bbb", "subnet-0ccc"]

  # EKS
  kubernetes_version     = "1.32"
  eks_node_instance_type = "m7g.xlarge"   # Graviton — matches AL2023_ARM_64_STANDARD
  eks_node_ami_id        = "ami-0abc1234"
  eks_node_desired_size  = 2
  eks_node_min_size      = 1
  eks_node_max_size      = 4

  # Database (RDS PostgreSQL)
  database_name           = "openmetadata"
  database_username       = "openmetadata"
  rds_instance_class      = "db.t3.medium"
  rds_engine_version      = "16.3"
  rds_family              = "postgres16"
  rds_allocated_storage   = 50
  rds_multi_az            = true
  rds_deletion_protection = true
  rds_skip_final_snapshot = false

  # OpenSearch
  opensearch_engine_version  = "OpenSearch_2.17"
  opensearch_instance_type   = "m6g.large.search"
  opensearch_instance_count  = 1
  opensearch_ebs_volume_size = 50
  opensearch_master_username = "openmetadata"

  # Kubernetes namespace
  namespace = "openmetadata"
}

See examples/basic/ for a complete working example.

Bring Your Own Resources

Each major component can be disabled when you already have that resource. Supply the corresponding existing_* inputs in its place:

# Use an existing EKS cluster instead of creating one
create_cluster               = false
existing_cluster_name        = "my-existing-cluster"
existing_cluster_endpoint    = "https://..."
existing_cluster_ca_data     = "..."
existing_oidc_provider_arn   = "arn:aws:iam::..."
existing_cluster_security_group_id = "sg-..."
existing_node_security_group_id    = "sg-..."

# Use an existing PostgreSQL database
create_rds                   = false
existing_database_endpoint   = "my-db.cluster-xyz.us-east-1.rds.amazonaws.com"
existing_database_secret_arn = "arn:aws:secretsmanager:..."

# Use an existing OpenSearch domain
create_opensearch              = false
existing_opensearch_endpoint   = "search-my-domain-xyz.us-east-1.es.amazonaws.com"
existing_opensearch_secret_arn = "arn:aws:secretsmanager:..."

Optional DNS and TLS

# Create a Route 53 CNAME record
route53_zone_name = "internal.example.com"
subdomain         = "openmetadata"           # resolves to openmetadata.internal.example.com

# Enable TLS via AWS Private CA
enable_tls        = true
acm_private_ca_arn = "arn:aws:acm-pca:..."

Granting EKS Access

# Give a CI/CD role full cluster admin
cluster_access_principals = {
  ci = {
    principal_arn = "arn:aws:iam::123456789012:role/my-ci-role"
  }
}

# Give a team namespace-scoped access
namespace_access_principals = {
  data-team = {
    principal_arn = "arn:aws:iam::123456789012:role/data-team-role"
    namespaces    = ["openmetadata"]
  }
}

Notes

  • Credential flow: RDS and OpenSearch passwords are generated randomly by Terraform, stored in AWS Secrets Manager, and synced into Kubernetes as Secret objects via the External Secrets Operator. OpenMetadata reads them from there at startup.
  • IRSA: The OpenMetadata pod runs under a Kubernetes service account bound to an IAM role (IRSA) with permissions to read Glue and S3 — the minimum required for data discovery integrations.
  • ALB security group: The ALB security group is owned by the app submodule rather than the cluster submodule because its ingress rules are tightly coupled to the ingress_cidr_blocks variable.
  • Encryption: All data-plane services (EKS secrets, RDS, OpenSearch, Secrets Manager) are encrypted with the customer-managed KMS key passed via kms_key_id. A separate secrets_kms_key_id can be used for Secrets Manager if needed.
  • OpenTofu / Terraform: The module is compatible with both. The primary development binary is tofu (OpenTofu ≥ 1.6). Terraform ≥ 1.5 is also supported.

Known issues

None at this time.


Contributing

Direct contributions are welcome.

See CONTRIBUTING.md for further information.


Documentation


Inputs

Name Description Type Default Required
account_id AWS account ID for OpenSearch access policy rendering. string n/a yes
acm_private_ca_arn Private CA ARN used to issue the OpenMetadata certificate. string null no
aws_lb_controller_chart_version Helm chart version for the AWS Load Balancer Controller. string "3.1.0" no
cluster_access_principals Keyed map of principals that should get cluster-scoped access.
map(object({
principal_arn = string
policy_arn = optional(string, "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy")
}))
{} no
cluster_api_ingress_cidr_blocks CIDR ranges allowed to reach the private EKS API endpoint. list(string)
[
"10.0.0.0/8"
]
no
create Global create toggle for the module. bool true no
create_access Whether to manage EKS access entries. bool true no
create_addons Whether to create cluster-wide addon resources. bool true no
create_app Whether to create the OpenMetadata application resources. bool true no
create_aws_load_balancer_controller Whether to install the AWS Load Balancer Controller addon. bool true no
create_cluster Whether to create the EKS cluster resources. bool true no
create_data Whether to create OpenMetadata data-plane resources. bool true no
create_external_secrets_operator Whether to install the External Secrets Operator addon. bool true no
create_ingress Whether to create the OpenMetadata ingress resources. bool true no
create_namespace Whether to create the Kubernetes namespace. bool true no
create_node_group Whether to create the default EKS managed node group. bool true no
create_openmetadata_release Whether to install the OpenMetadata Helm release. bool true no
create_opensearch Whether to create the OpenSearch domain. bool true no
create_opensearch_secret Whether to create the OpenSearch credentials secret. bool true no
create_rds Whether to create the PostgreSQL database. When false, provide existing_database_endpoint and existing_database_secret_arn for any path that needs database connectivity. bool true no
create_rds_secret Whether to create the RDS credentials secret when create_rds is true. Managed RDS currently requires this to remain true. bool true no
database_name OpenMetadata database name. Used as the managed RDS database name when create_rds is true. When create_rds is false, this database must already exist on existing_database_endpoint. string "openmetadata" no
database_secret_property Optional JSON property to extract from the database secret value. Leave null for plain string secrets. Defaults to password only when this module creates the database secret (create_data, create_rds, and create_rds_secret are all true). string null no
database_username OpenMetadata database username. Used as the managed RDS username when create_rds is true. When create_rds is false, this user must already exist and have access to database_name. string "openmetadata" no
eks_node_ami_id Approved AMI ID for the EKS managed node group. string n/a yes
eks_node_ami_type AMI type for the EKS managed node group. string "AL2023_ARM_64_STANDARD" no
eks_node_desired_size Desired node count for the default EKS managed node group. number 2 no
eks_node_iam_role_policy_json Optional JSON IAM policy document to attach to the default node role. string null no
eks_node_instance_type Instance type for the default EKS managed node group. string "m7g.large" no
eks_node_max_size Maximum node count for the default EKS managed node group. number 4 no
eks_node_min_size Minimum node count for the default EKS managed node group. number 1 no
eks_node_startup_script Optional shell script to run on EKS node startup. string null no
enable_tls Whether to enable TLS for the OpenMetadata ingress. bool false no
existing_cluster_ca_data Existing cluster certificate authority data used when create_cluster is false. string null no
existing_cluster_endpoint Existing cluster API endpoint used when create_cluster is false. string null no
existing_cluster_name Existing cluster name used when create_cluster is false. string null no
existing_cluster_security_group_id Existing cluster security group ID used when create_cluster is false. string null no
existing_database_endpoint Endpoint (host or host:port) of an existing PostgreSQL database. Required when create_rds is false and managed data resources are used, and also required when create_data is false while create_app and create_openmetadata_release are true. string null no
existing_database_secret_arn Secrets Manager ARN for the existing database password (or JSON field selected by database_secret_property). Required in existing-database mode whenever the OpenMetadata release needs database credentials. string null no
existing_node_security_group_id Existing node security group ID used when create_cluster is false. string null no
existing_oidc_provider_arn Existing OIDC provider ARN used when create_cluster is false. string null no
existing_opensearch_endpoint Existing OpenSearch endpoint used when create_opensearch is false. Also required when create_data is false while create_app and create_openmetadata_release are true. string null no
existing_opensearch_secret_arn Secrets Manager ARN for the existing OpenSearch password (or JSON field selected by opensearch_secret_property). Required in existing-OpenSearch mode whenever the OpenMetadata release needs OpenSearch credentials. string null no
external_secrets_chart_version Helm chart version for the External Secrets Operator. string "2.2.0" no
iam_role_path Path of the IAM role. If not specified then the default of '/' is used. string "/" no
iam_role_permissions_boundary Permissions boundary ARN for IAM roles created by this module. string n/a yes
ingress_cidr_blocks CIDR ranges allowed inbound to the OpenMetadata ALB. list(string) [] no
kms_key_id KMS key ID or ARN used for data-plane encryption (EKS, RDS, OpenSearch). string n/a yes
kubernetes_version Kubernetes version for the EKS cluster. string "1.35" no
name Base name prefix used for named resources. string n/a yes
name_prefix Short prefix for resources with stricter name limits (e.g. OpenSearch 28-char domain limit). string n/a yes
namespace Kubernetes namespace for OpenMetadata. string "openmetadata" no
namespace_access_principals Keyed map of principals that should get namespace-scoped access.
map(object({
principal_arn = string
namespaces = list(string)
policy_arn = optional(string, "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy")
}))
{} no
oidc_thumbprints Custom OIDC root CA thumbprints for the EKS module. This module configures include_oidc_root_ca_thumbprint = false, so supply any required root CA thumbprints here. list(string) [] no
openmetadata_chart_version OpenMetadata Helm chart version. string "1.12.3" no
openmetadata_external_secret_kms_key_arns Optional list of KMS key ARNs to allow decrypt for synced Secrets Manager values when customer-managed KMS keys are used. list(string) [] no
openmetadata_external_secret_store_name SecretStore name used by ExternalSecret resources. string "aws-secrets" no
openmetadata_external_secrets Additional ExternalSecret entries keyed by Kubernetes Secret name. secret_arn is the AWS source secret, secret_key is the key written inside the Kubernetes Secret, and optional secret_property selects a JSON field from the AWS secret value.
map(object({
secret_arn = string
secret_key = optional(string, "value")
secret_property = optional(string, null)
}))
{} no
openmetadata_fqdn Precomputed OpenMetadata FQDN used for ACM and Route53 resources. string null no
openmetadata_heap_opts JVM heap options passed to OpenMetadata via OPENMETADATA_HEAP_OPTS. string "-Xmx2G -Xms2G" no
openmetadata_helm_set_sensitive_values Generic sensitive Helm set values applied directly to the OpenMetadata chart via set_sensitive. map(string) {} no
openmetadata_helm_set_values Generic Helm set values applied directly to the OpenMetadata chart. Key is Helm path (for example openmetadata.config.authentication.provider), value is converted to string. map(any) {} no
opensearch_ebs_volume_size OpenSearch EBS volume size in GB. number 20 no
opensearch_engine_version OpenSearch engine version. string "OpenSearch_3.3" no
opensearch_instance_count OpenSearch data node count. number 2 no
opensearch_instance_type OpenSearch node instance type. string "m6g.large.search" no
opensearch_master_username OpenSearch master username. string "openmetadata" no
opensearch_secret_property Optional JSON property to extract from the OpenSearch secret value. Leave null for plain string secrets. Defaults to password only when this module creates the OpenSearch secret (create_data, create_opensearch, and create_opensearch_secret are all true). string null no
private_subnet_ids Private subnet IDs used by EKS and data-plane services. list(string) n/a yes
rds_allocated_storage Allocated RDS storage in GB to create. number 20 no
rds_backup_retention_period Number of days to retain RDS automated backups for the created RDS. Set to 0 to disable backups. number 7 no
rds_deletion_protection Whether to enable RDS deletion protection on the created RDS. bool n/a yes
rds_engine_version PostgreSQL engine version to create. string "18.3" no
rds_family Parameter group family for PostgreSQL to create. string "postgres18" no
rds_ingress_cidr_blocks Additional CIDR blocks allowed to reach the RDS instance on port 5432. Useful for direct database access from a bastion or developer machine. list(string) [] no
rds_instance_class RDS instance class to create. string "db.t3.medium" no
rds_multi_az Whether to enable Multi-AZ on the created RDS. bool n/a yes
rds_skip_final_snapshot Whether to skip the final snapshot on RDS deletion. bool n/a yes
region AWS region for the deployment. string n/a yes
route53_zone_name Private Route53 hosted zone used for the OpenMetadata record. string null no
secrets_kms_key_id KMS key ID or ARN used to encrypt AWS Secrets Manager secrets. string null no
subdomain Subdomain used for the OpenMetadata DNS name. string "open-metadata" no
tags Tags to apply to supported resources. map(string) {} no
vpc_id VPC ID used by the cluster and data plane. string n/a yes

Modules

Name Source Version
access ./modules/access n/a
addons ./modules/addons n/a
app ./modules/app n/a
cluster ./modules/cluster n/a
data ./modules/data n/a
dns ./modules/dns n/a

Outputs

Name Description
access_entry_ids IDs of EKS access resources created for the cluster.
cluster_ca_data Base64-encoded certificate authority data for the EKS cluster.
cluster_endpoint Endpoint for the EKS cluster API server.
cluster_name Name of the EKS cluster.
cluster_security_group_id Cluster security group ID for the EKS cluster.
database_endpoint Endpoint for the PostgreSQL database instance.
node_security_group_id Node security group ID for the EKS managed node group.
oidc_provider_arn OIDC provider ARN used for IRSA.
openmetadata_alb_dns DNS name of the ALB provisioned for OpenMetadata.
openmetadata_fqdn Route53 record FQDN pointing to the OpenMetadata ALB.
openmetadata_irsa_role_arn IAM role ARN for the OpenMetadata service account.
opensearch_endpoint Endpoint for the OpenSearch domain.

Providers

Name Version
aws >= 5, < 7

Requirements

Name Version
terraform >= 1.5.7, < 2.0.0
aws >= 5, < 7
helm ~> 2.17
kubernetes ~> 2.35
random ~> 3.6

Resources

Name Type
aws_kms_key.secrets data source

About

Terraform module to deploy OpenMetadata (open-metadata.org) on AWS

Resources

License

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Packages

 
 
 

Contributors

Languages