Serialize Atmos library calls to prevent concurrent ReadDataSource crash (#523)

* Add local goreleaser config to exclude unsupported windows/arm target

Go 1.24+ dropped support for the windows/arm (32-bit) GOOS/GOARCH pair.
The shared org-wide goreleaser config includes it, causing release builds
to fail after the Go 1.23 -> 1.26 upgrade in #522.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update fix doc to reflect org-wide goreleaser fix

Document the broader scope: the windows/arm ignore rule is applied in
the shared config (cloudposse/.github), the provider's local override
(temporary), and needs to be added to atmos's local config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Serialize Atmos library calls to fix concurrent ReadDataSource crash

The Atmos library uses package-level mutable state (mergedConfigFiles in
pkg/config/load.go) that is explicitly not safe for concurrent use. When
Terraform executes multiple ReadDataSource calls in parallel (e.g., 5
concurrent utils_component_config reads), they race on the shared state,
corrupt it, and trigger CheckErrorPrintAndExit → os.Exit(1), killing the
gRPC plugin process ("Plugin did not respond").

Add a sync.Mutex to serialize all Atmos library calls across the five
data sources that use it: component_config, describe_stacks,
stack_config_yaml, spacelift_stack_config, and aws_eks_update_kubeconfig.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update fix doc: remove consumer-specific references

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address PR review: use defer for mutex unlock, update goreleaser for v2

- Refactor describe_stacks and stack_config_yaml to use anonymous functions
  with defer atmosMu.Unlock() instead of manual unlock on each error path
- Add version: 2 header to .goreleaser.yml for GoReleaser v2 compliance
- Replace deprecated .Commit with .FullCommit in goreleaser ldflags

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [autofix.ci] apply automated fixes

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

Author: 3 people

.goreleaser.yml

@@ -0,0 +1,59 @@
+version: 2
+
+# Local goreleaser config — overrides the org-wide config at cloudposse/.github/.github/goreleaser.yml
+# Needed because Go 1.24+ dropped support for windows/arm (32-bit), which the shared config includes.
+
+builds:
+- env:
+    - CGO_ENABLED=0
+  mod_timestamp: '{{ .CommitTimestamp }}'
+  flags:
+    - -trimpath
+  ldflags:
+    - '-s -w -X main.version={{.Version}} -X main.commit={{.FullCommit}}'
+  goos:
+    - freebsd
+    - windows
+    - linux
+    - darwin
+  goarch:
+    - amd64
+    - '386'
+    - arm
+    - arm64
+  ignore:
+    - goos: windows
+      goarch: arm
+  binary: '{{ .ProjectName }}'
+
+archives:
+- format: "{{ .Env.ARCHIVES_FORMAT }}"
+  name_template: '{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}'
+
+checksum:
+  name_template: '{{ .ProjectName }}_{{ .Version }}_SHA256SUMS'
+  algorithm: sha256
+
+signs:
+  - artifacts: checksum
+    args:
+      - "--batch"
+      - "--local-user"
+      - "{{ .Env.GPG_FINGERPRINT }}"
+      - "--output"
+      - "${signature}"
+      - "--detach-sign"
+      - "${artifact}"
+
+release:
+  draft: true
+  replace_existing_draft: true
+  replace_existing_artifacts: true
+  mode: keep-existing
+  make_latest: false
+  name_template: '{{.Tag}}'
+  target_commitish: '{{ if index .Env "GO_RELEASER_TARGET_COMMITISH"  }}{{ .Env.GO_RELEASER_TARGET_COMMITISH }}{{ else }}{{ .Branch }}{{ end }}'
+  prerelease: auto
+
+changelog:
+  skip: true

docs/fixes/2026-02-20-concurrent-readdatasource-crash.md

@@ -0,0 +1,143 @@
+# Provider `ReadDataSource` Crash — `os.Exit(1)` from Atmos Library
+
+**Affected Versions:** `cloudposse/utils` provider v2.0.0 (Atmos v1.207.0 embedded)
+
+**Severity:** Critical — provider process exits with code 1 during `ReadDataSource`
+calls, producing "Plugin did not respond" errors in Terraform
+
+## Symptoms
+
+Components with multiple `data "utils_component_config"` data sources crash the provider during
+`terraform plan`:
+
+```text
+Error: Plugin did not respond
+
+  with module.iam_roles.module.account_map.data.utils_component_config.config[0],
+  on .terraform/modules/iam_roles.account_map/modules/remote-state/main.tf line 1,
+  in data "utils_component_config" "config":
+   1: data "utils_component_config" "config" {
+```
+
+`TF_LOG=TRACE` reveals the provider exits with **exit status 1** (not a panic or signal):
+
+```text
+provider: plugin process exited: path=...terraform-provider-utils pid=3934 error="exit status 1"
+```
+
+## Root Causes
+
+Investigation revealed **two independent issues** that can each trigger `os.Exit(1)` inside the
+Atmos library, killing the provider's gRPC plugin process. Both are addressed in this fix.
+
+### Issue 1: `os.Exit(1)` on errors in the Atmos library
+
+The Atmos library uses `CheckErrorPrintAndExit()` in many code paths within `internal/exec`.
+This function is designed for CLI usage — it prints an error and exits the process. Inside a
+Terraform provider (gRPC plugin), calling `os.Exit(1)` terminates the plugin without returning
+a diagnostic error to Terraform.
+
+Any error that reaches `CheckErrorPrintAndExit` silently crashes the provider. Code paths that
+use it include:
+
+- `utils.go:664-673` — duplicate component config detection
+- `utils.go:753,765` — template processing errors
+- `yaml_func_store.go:29,89,99,107` — `!store` YAML tag errors
+- `yaml_func_store_get.go:52,90,101,116` — `!store.get` YAML tag errors
+- `describe_stacks.go:489,743,982` — describe stacks errors
+
+**Any** error reaching these paths will crash the provider via `os.Exit(1)`. The provider cannot
+intercept `os.Exit` — the Atmos library kills the process before the provider can return a
+diagnostic to Terraform.
+
+### Issue 2: Thread-unsafe global state (LATENT)
+
+The Atmos library was designed as a single-threaded CLI tool. It has **package-level mutable
+state** that is explicitly documented as not thread-safe:
+
+```go
+// pkg/config/load.go:51-54
+// NOTE: This package-level state assumes sequential (non-concurrent) calls to LoadConfig.
+// LoadConfig is NOT safe for concurrent use.
+var mergedConfigFiles []string
+```
+
+Additional thread-unsafe global state includes:
+
+- `errors/error_funcs.go:31` — `var atmosConfig *schema.AtmosConfiguration`
+- `errors/error_funcs.go:28` — `var render *markdown.Renderer`
+- `errors/error_funcs.go:34` — `var verboseFlag`
+
+Terraform invokes `ReadDataSource` concurrently (one goroutine per data source instance). Each
+call enters `ProcessComponentInStack` → `InitCliConfig` → `LoadConfig`, which resets and writes
+to the shared `mergedConfigFiles` slice. Concurrent goroutines can corrupt the slice through
+interleaved reads/writes, causing downstream errors that hit `CheckErrorPrintAndExit` →
+`os.Exit(1)`.
+
+This is a **latent** issue — it is a real data race that could cause unpredictable failures
+under concurrent load.
+
+### Debug log timeline (provider v2.0.0, pid=3934)
+
+| Timestamp      | Event                                                        |
+|----------------|--------------------------------------------------------------|
+| `19:24:15.825` | Provider starts, configures mTLS                             |
+| `19:24:15.941` | GetProviderSchema — success                                  |
+| `19:24:15.945` | Configure — success                                          |
+| `19:24:16.117` | ValidateDataSourceConfig — success                           |
+| `19:24:16.122` | **ReadDataSource #1** — "Calling downstream" (never returns) |
+| `19:24:16.152` | **ReadDataSource #2** — "Calling downstream" (never returns) |
+| `19:24:16.276` | **ReadDataSource #3** — "Calling downstream" (never returns) |
+| `19:24:16.817` | **Plugin process exited: exit status 1**                     |
+| `19:24:16.817` | gRPC: "connection reset by peer"                             |
+
+## Fix
+
+### Provider-side fix
+
+Add a package-level `sync.Mutex` in the provider to serialize all calls into the Atmos library.
+This addresses **Issue 2** by preventing concurrent goroutines from accessing the thread-unsafe
+global state simultaneously. It does not prevent `os.Exit(1)` from stack config errors (Issue 1),
+but it eliminates the data race as a potential trigger.
+
+**New file: `internal/provider/atmos_lock.go`**
+
+```go
+package provider
+
+import "sync"
+
+// atmosMu serializes all calls into the Atmos library.
+// The Atmos library uses package-level mutable state (e.g., mergedConfigFiles in
+// pkg/config/load.go) that is explicitly documented as not safe for concurrent use.
+// Terraform invokes ReadDataSource concurrently for independent data sources, so
+// without this mutex, concurrent calls corrupt shared state and trigger os.Exit(1)
+// via CheckErrorPrintAndExit, killing the gRPC plugin process.
+var atmosMu sync.Mutex
+```
+
+**Modified data source files:**
+
+Each `ReadContext` function wraps its Atmos library calls with `atmosMu.Lock()` /
+`atmosMu.Unlock()`:
+
+- `data_source_component_config.go` — wraps `ProcessComponentInStack` / `ProcessComponentFromContext`
+- `data_source_describe_stacks.go` — wraps `InitCliConfig` + `ExecuteDescribeStacks`
+- `data_source_stack_config_yaml.go` — wraps `InitCliConfig` + `ProcessYAMLConfigFiles`
+- `data_source_spacelift_stack_config.go` — wraps `CreateSpaceliftStacks`
+- `data_source_aws_eks_update_kubeconfig.go` — wraps `ExecuteAwsEksUpdateKubeconfig`
+
+### Long-term fix (Atmos library)
+
+The Atmos library should be refactored to:
+
+1. Replace `CheckErrorPrintAndExit` / `os.Exit` calls in library code paths with proper error
+   returns, so embedded consumers (like this provider) can handle errors gracefully — this is
+   the most critical fix, as it would convert silent crashes into visible Terraform diagnostics
+2. Eliminate package-level mutable state in `pkg/config` and `errors`
+3. Pass configuration through context or options structs instead of global variables
+
+## References
+
+- Atmos `LoadConfig` thread-safety comment: `pkg/config/load.go:51-54`
+- `CheckErrorPrintAndExit` implementation: `errors/error_funcs.go:324-366`

internal/provider/atmos_lock.go

@@ -0,0 +1,12 @@
+package provider
+
+import "sync"
+
+// atmosMu serializes all calls into the Atmos library.
+//
+// The Atmos library uses package-level mutable state (e.g., mergedConfigFiles in
+// pkg/config/load.go) that is explicitly documented as not safe for concurrent use.
+// Terraform invokes ReadDataSource concurrently for independent data sources, so
+// without this mutex, concurrent calls corrupt shared state and trigger os.Exit(1)
+// via CheckErrorPrintAndExit, killing the gRPC plugin process.
+var atmosMu sync.Mutex

internal/provider/data_source_aws_eks_update_kubeconfig.go

@@ -138,7 +138,10 @@ func dataSourceAwsEksUpdateKubeconfigRead(ctx context.Context, d *schema.Resourc
 		return diag.FromErr(err)
 	}
 
+	atmosMu.Lock()
 	err = a.ExecuteAwsEksUpdateKubeconfig(kubeconfigContext)
+	atmosMu.Unlock()
+
 	if err != nil {
 		return diag.FromErr(err)
 	}

internal/provider/data_source_component_config.go

@@ -114,11 +114,9 @@ func dataSourceComponentConfigRead(ctx context.Context, d *schema.ResourceData,
 		}
 	}
 
+	atmosMu.Lock()
 	if len(stack) > 0 {
 		result, err = p.ProcessComponentInStack(component, stack, atmosCliConfigPath, atmosBasePath)
-		if err != nil && !ignoreErrors {
-			return diag.FromErr(err)
-		}
 	} else {
 		result, err = p.ProcessComponentFromContext(&p.ComponentFromContextParams{
 			Component:          component,
@@ -129,9 +127,11 @@ func dataSourceComponentConfigRead(ctx context.Context, d *schema.ResourceData,
 			AtmosCliConfigPath: atmosCliConfigPath,
 			AtmosBasePath:      atmosBasePath,
 		})
-		if err != nil && !ignoreErrors {
-			return diag.FromErr(err)
-		}
+	}
+	atmosMu.Unlock()
+
+	if err != nil && !ignoreErrors {
+		return diag.FromErr(err)
 	}
 
 	if err != nil {

internal/provider/data_source_describe_stacks.go

@@ -151,31 +151,37 @@ func dataSourceDescribeStacksRead(ctx context.Context, d *schema.ResourceData, m
 		AtmosCliConfigPath: atmosCliConfigPath,
 	}
 
-	cliConfig, err := cfg.InitCliConfig(info, true)
-	if err != nil {
-		return diag.FromErr(err)
-	}
+	result, err = func() (map[string]any, error) {
+		atmosMu.Lock()
+		defer atmosMu.Unlock()
 
-	var filterByStack string
-
-	if stack != "" {
-		filterByStack = stack
-	} else if namespace != "" || tenant != "" || environment != "" || stage != "" {
-		filterByStack, err = cfg.GetStackNameFromContextAndStackNamePattern(namespace, tenant, environment, stage, cliConfig.Stacks.NamePattern)
+		cliConfig, err := cfg.InitCliConfig(info, true)
 		if err != nil {
-			return diag.FromErr(err)
+			return nil, err
 		}
-	}
 
-	result, err = describe.ExecuteDescribeStacks(
-		cliConfig,
-		filterByStack,
-		componentsList,
-		componentTypesList,
-		sectionsList,
-		false,
-		true,
-	)
+		var filterByStack string
+
+		if stack != "" {
+			filterByStack = stack
+		} else if namespace != "" || tenant != "" || environment != "" || stage != "" {
+			filterByStack, err = cfg.GetStackNameFromContextAndStackNamePattern(namespace, tenant, environment, stage, cliConfig.Stacks.NamePattern)
+			if err != nil {
+				return nil, err
+			}
+		}
+
+		return describe.ExecuteDescribeStacks(
+			cliConfig,
+			filterByStack,
+			componentsList,
+			componentTypesList,
+			sectionsList,
+			false,
+			true,
+		)
+	}()
+
 	if err != nil && !ignoreErrors {
 		return diag.FromErr(err)
 	}

internal/provider/data_source_spacelift_stack_config.go

@@ -93,6 +93,7 @@ func dataSourceSpaceliftStackConfigRead(ctx context.Context, d *schema.ResourceD
 		return diag.FromErr(err)
 	}
 
+	atmosMu.Lock()
 	spaceliftStacks, err := s.CreateSpaceliftStacks(
 		stacksBasePath.(string),
 		"",
@@ -104,6 +105,8 @@ func dataSourceSpaceliftStackConfigRead(ctx context.Context, d *schema.ResourceD
 		processComponentDeps.(bool),
 		processImports.(bool),
 		stackConfigPathTemplate.(string))
+	atmosMu.Unlock()
+
 	if err != nil {
 		return diag.FromErr(err)
 	}

internal/provider/data_source_stack_config_yaml.go

@@ -66,18 +66,13 @@ func dataSourceStackConfigYAML() *schema.Resource {
 }
 
 func dataSourceStackConfigYAMLRead(ctx context.Context, d *schema.ResourceData, meta any) diag.Diagnostics {
-	cliConfig, err := cfg.InitCliConfig(atmosSchema.ConfigAndStacksInfo{}, true)
-	if err != nil {
-		return diag.FromErr(err)
-	}
-
 	input := d.Get("input")
 	processStackDeps := d.Get("process_stack_deps")
 	processComponentDeps := d.Get("process_component_deps")
 	stacksBasePath := d.Get("base_path")
 	env := d.Get("env").(map[string]any)
 
-	err = setEnv(env)
+	err := setEnv(env)
 	if err != nil {
 		return diag.FromErr(err)
 	}
@@ -87,18 +82,29 @@ func dataSourceStackConfigYAMLRead(ctx context.Context, d *schema.ResourceData,
 		return diag.FromErr(err)
 	}
 
-	result, _, _, err := s.ProcessYAMLConfigFiles(
-		&cliConfig,
-		stacksBasePath.(string),
-		"",
-		"",
-		"",
-		"",
-		paths,
-		processStackDeps.(bool),
-		processComponentDeps.(bool),
-		false,
-	)
+	var result []string
+	result, _, _, err = func() ([]string, map[string]any, map[string]map[string]any, error) {
+		atmosMu.Lock()
+		defer atmosMu.Unlock()
+
+		cliConfig, err := cfg.InitCliConfig(atmosSchema.ConfigAndStacksInfo{}, true)
+		if err != nil {
+			return nil, nil, nil, err
+		}
+
+		return s.ProcessYAMLConfigFiles(
+			&cliConfig,
+			stacksBasePath.(string),
+			"",
+			"",
+			"",
+			"",
+			paths,
+			processStackDeps.(bool),
+			processComponentDeps.(bool),
+			false,
+		)
+	}()
 	if err != nil {
 		return diag.FromErr(err)
 	}