[9.4] (backport #15491) Write OTel collector logs to its own log file

[mergify]

What does this PR do?

The OTel runtime now writes to its own log file instead of sharing the Elastic Agent's own log file.

Before this change the collector wrote into the same file as the agent itself:

logs/
├── elastic-agent-20260713.ndjson
└── events/
    └── elastic-agent-event-log-20260713.ndjson

Example of the new log layout, running under the otel runtime:

logs/
├── elastic-agent-20260713.ndjson
├── elastic-otel-collector-20260713.ndjson
└── events/
    ├── elastic-agent-event-log-20260713.ndjson # only created if process runtime is also used
    └── elastic-otel-collector-event-log-20260713.ndjson

OTel collector now gets its own events log file. This is a consequence of how elastic-agent-libs typed loggers work, each logger instance creates its own event output file. Sharing a single events file across components would require a larger refactor of the logging framework.

Why is it important?

Before this change, the OTel collector wrote into the same log file as the Agent itself. This made it hard to find the logs that belong to a specific input or component and mixed event data together with regular operational logs.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

Anyone who wants to retrieve OTel collector logs should check the new file name under the logs and events directory. If they use the elastic-agent logs command nothing changes.

How to test this PR locally

This can be easily tested with the following integration test:

BUILD_AGENT=true EXTERNAL=true PACKAGES="tar.gz" TEST_PACKAGES="tar.gz" \
    mage -v integration:local TestEventLogFile

Related issues

• Closes Use separate log files per process (again) #15164

---

This is an automatic backport of pull request #15491 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 61463e6 has failed:

On branch mergify/bp/9.4/pr-15491
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 61463e62f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   NOTICE-fips.txt
	modified:   NOTICE.txt
	new file:   changelog/fragments/1783951984-otel-collector-dedicated-log-file.yaml
	modified:   go.mod
	modified:   go.sum
	modified:   internal/edot/go.mod
	modified:   internal/edot/go.sum
	modified:   internal/edot/testing/testing.go
	modified:   internal/pkg/agent/application/application.go
	modified:   internal/pkg/agent/application/application_test.go
	modified:   internal/pkg/agent/application/monitoring/component/v1_monitor.go
	modified:   internal/pkg/agent/application/monitoring/component/v1_monitor_test.go
	modified:   internal/pkg/agent/cmd/logs.go
	modified:   internal/pkg/agent/cmd/logs_test.go
	modified:   internal/pkg/agent/cmd/run.go
	modified:   internal/pkg/otel/manager/execution.go
	modified:   internal/pkg/otel/manager/execution_subprocess.go
	modified:   pkg/component/component.go
	modified:   pkg/core/logger/logger.go
	modified:   pkg/core/logger/logger_test.go
	modified:   pkg/testing/fixture.go
	modified:   testing/integration/ess/container_cmd_test.go
	modified:   testing/integration/ess/diagnostics_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   .golangci.yml
	both modified:   internal/pkg/otel/manager/manager.go
	both modified:   internal/pkg/otel/manager/manager_test.go
	both modified:   testing/integration/ess/event_logging_test.go
	both modified:   testing/integration/ess/logging_file_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[infra-vault-gh-plugin-prod]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 76aad28

Failed CI Steps

• check-ci
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 10
• Unit tests - Windows 11
• Unit tests - Windows 11
• Unit tests - Windows 11 arm64
• Unit tests - Windows 11 arm64
• :junit: Junit annotate
• 📦 amd64: zip,tar.gz,rpm,deb
• 📦 amd64: zip,tar.gz,rpm,deb
• 📦 amd64: FIPS tar.gz
• 📦 amd64: FIPS tar.gz
• 📦 arm64: zip,tar.gz
• 📦 arm64: zip,tar.gz
• 📦 arm64: FIPS tar.gz
• 📦 arm64: FIPS tar.gz
• 📦 amd64: Containers
• 📦 arm64: Containers
• 📦 amd64: FIPS Containers
• 📦 arm64: FIPS Containers
• Start ESS stack for integration tests
• Aggregate test reports
• ESS FIPS stack cleanup
• Aggregate test reports
• :pipeline: Run elastic-agent-package with agent core built locally

History

cc @samuelvl