feat: collect service override source by raphaelgavache · Pull Request #4500 · DataDog/dd-trace-go

raphaelgavache · 2026-03-05T04:18:00Z

This PR is a first of many that enriches service source to service overrides. In particalar the PR contains:

introduction of instrumentation.ServiceNameWithSource method to be used by all integrations
4 integrations covered: grpc, gin-gonic, go-redis, database/sql
inheritence of service source
coverage of service mapping configuration
encoding of source in span.Meta

See other similar PRs in dd-trace-java PR1 - integration services, PR2- client stats, PR3 - config cases, PR4 - manual source

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
New code is free of linting errors. You can check this by running make lint locally.
New code doesn't break existing tests. You can check this by running make test locally.
Add an appropriate team label so this PR gets put in the right place for the release notes.
All generated files are up to date. You can check this by running make generate locally.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running make fix-modules locally.

Unsure? Have a question? Request a review!

codecov · 2026-03-05T04:21:58Z

Codecov Report

❌ Patch coverage is 84.37500% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.17%. Comparing base (cb2ef86) to head (b9ab5b0).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
instrumentation/instrumentation.go	0.00%	3 Missing ⚠️
ddtrace/tracer/span.go	87.50%	1 Missing and 1 partial ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
ddtrace/tracer/tracer.go	`87.74% <100.00%> (ø)`
internal/tracer.go	`0.00% <ø> (ø)`
ddtrace/tracer/span.go	`85.85% <87.50%> (ø)`
instrumentation/instrumentation.go	`16.25% <0.00%> (ø)`

... and 427 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog-official · 2026-03-05T04:22:04Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 87.10%
• Overall Coverage: 59.29%

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: b9ab5b0 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

pr-commenter · 2026-03-10T01:02:49Z

Benchmarks

Benchmark execution time: 2026-03-12 18:11:15

Comparing candidate commit b9ab5b0 in PR branch raphael/srv_src with baseline commit cb2ef86 in branch main.

Found 0 performance improvements and 5 performance regressions! Performance is the same for 152 metrics, 7 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:BenchmarkPartialFlushing/Disabled-25

🟥 allocated_mem [+6.359MB; +6.364MB] or [+2.009%; +2.010%]

scenario:BenchmarkSingleSpanRetention/no-rules-25

🟥 allocated_mem [+3.233KB; +3.235KB] or [+2.119%; +2.120%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-all-25

🟥 allocated_mem [+3.185KB; +3.328KB] or [+2.053%; +2.145%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-half-25

🟥 allocated_mem [+3.210KB; +3.316KB] or [+2.069%; +2.137%]

scenario:BenchmarkStartSpan-25

🟥 allocated_mem [+32 bytes; +32 bytes] or [+2.123%; +2.123%]

hannahkm · 2026-03-10T14:36:06Z

contrib/database/sql/option.go

 	}
 	cfg.dbmPropagationMode = tracer.DBMPropagationMode(mode)
 	cfg.serviceName = defaultServiceName(driverName, rc)
+	cfg.serviceSource = "database/sql"


For each contrib, we already have something called componentName that is pulled from our instrumentation package. For this contrib, it's set in the sql.go file, but also accessible at instrumentation.PackageDatabaseSQL. Instead of creating a new string, can we reuse that variable?

This also goes for the other contribs + future contribs changes

hannahkm · 2026-03-10T14:38:49Z

contrib/gin-gonic/gin/gintrace_test.go

 		assert.Len(spans, 1)
 		span := spans[0]
 		assert.Equal("global-service", span.Tag(ext.ServiceName))
+		assert.Nil(span.Tag("_dd.svc_src"), "_dd.svc_src should not be set when service matches global service")


Instead of using _dd.svc_src, let's import ext (see one of my comments below)

hannahkm · 2026-03-10T14:42:20Z

internal/tracer.go

+// KeyServiceSource is the internal span meta key for tracking the origin of a
+// service name override. It is intercepted by the tracer and stored in an
+// internal field rather than written directly to span meta.
+const KeyServiceSource = "_dd.svc_src"


Can we move this to tag.go (under ddtrace/ext)? We have other span tags living there, and then we don't have to reflect this in span.go or hardcode the string from integrations, as I mentioned above (just import ext).

Of course, that's a public package, so if we want to keep this internal then maybe that's not the best idea. I'm open to discussing this! I just want to avoid hardcoding the strings in tests/from the contribs as we currently are doing.

hannahkm · 2026-03-10T14:45:39Z

contrib/gin-gonic/gin/option.go

 }

 func newConfig(serviceName string) *config {
+	serviceSource := "opt.middleware"


I wonder if it's worth having a file of potential serviceSource values stored somewhere in instrumentation or in ddtrace/ext for the commonly used values (ie opt.middleware, manual, opt.mapping). Up to you to decide if this is worthwhile (to avoid magic strings).

ddtrace/tracer/span.go

hannahkm

LGTM!

rarguelloF

Nice! This is a cool feature :) left some nit comments

rarguelloF · 2026-03-12T12:31:09Z

instrumentation/instrumentation.go

+
+	// ServiceSourceMiddleware is the service source value used when the service
+	// name is set via a middleware parameter (e.g. gin's Middleware(serviceName)).
+	ServiceSourceMiddleware = "opt.middleware"


hmm I'm not sure this would be a generic enough pattern to be considered here.

Basically because of legacy/historical reasons, we have a number of integrations where the main function that "installs" the integration requires the service name as a mandatory argument. In the case of gin, Middleware is the main entrypoint of the integration and sadly it requires the mandatory argument so we will see many service overrides here as a consequence.

Ideally we will deprecate these functions at some point and follow the same pattern as the rest of the integrations (the service name being optional through the opt.WithService option).

My suggestion here:

something like entrypointArgument to signal its just an integration where the main entrypoint has a mandatory service argument (maybe this will be confusing among some folks? not sure).

just an specific gin.Middleware or http.WrapHandler functions that are doing this (I don't think there are that many in the repo).

WDYT?

sounds good, I'll keep the prefix opt. as it's used to distinguish what is a user option, in our backend and is not user facing. For example here the customer even if forced to use the API could leave it empty string

We'll have to discuss plan to deprecate

rarguelloF · 2026-03-12T12:32:27Z

instrumentation/instrumentation.go

+
+	// ServiceSourceDriverName is the service source value used when the service
+	// name is derived from the driver name (e.g. database/sql Register driverName).
+	ServiceSourceDriverName = "opt.drivername"


I don't think this is generic enough to have it in the shared instrumentation package (I think database/sql is the only place where we do something like this). WDYT about sql.inferredFromDriver?

rarguelloF

LGTM!

mtoffl01

Mainly nits.

Also:

I'm a little concerned that the new ServiceNameWithSource API, which modifies both service name and service source, does not have a clear enough purpose, given we have a separate API for setting service name alone. Is it necessary to bundle them together, or can we separate APIs for service name and service source?
In the screenshot found in the PR description, the svc src tag values are in different formats (some are camel case while some are not, some have dot notation while others include a slash). I think it's important to standardize the format, both in terms of characters/casing as well as maybe defining prefixes, similar to how Datadog Metrics have standard prefixes

mtoffl01 · 2026-03-12T14:20:50Z

contrib/gin-gonic/gin/option.go


+// serviceSourceGinMiddleware is the service source value used when the service
+// name is set via gin's Middleware(serviceName) parameter.
+const serviceSourceGinMiddleware = "opt.gin_middleware"


This seems inconsistent with serviceSourceSQLDriver.

This one is set when a user has explicitly passed in a serviceName parameter to Middleware, but serviceSourceSQLDriver is set in the default case.

yes the behaviors are actually different between the integrations

on gin if the string is empty, the default services become the tracer global service

on database/sql if the string is empty, the service still is built from it and results in .db

See the following tests

gin

database.sql

One of my planned follow-up is to deprecate these APIs that pushes/forces users to have service overrides that are then confusing in the product. Replacing them with the options WithService for those who want to keep the behaviour

As far as I understand this new property the source is the place where the name comes from.

For SQL opt.sql_driver is used when the name is set by the package. It makes sense here as well because the middleware is the origin of the name and when no name is given we fallback to another source (default)

mtoffl01 · 2026-03-12T14:29:21Z

ddtrace/tracer/span.go

-		service:  s.service,
-		pprofCtx: s.pprofCtxActive,
+		service:       s.service,
+		serviceSource: s.serviceSource,


I'm concerned that this could be misleading.

Let's say we "inherit" context from a service we don't own -- like some third party client made the request and we're serving it. This client, let's say, had instrumentation configured using the WithService parameter.

On our server side, we'll see that as the service source without having any insight into it. Could be confusing.

yes, but it would map the service on the spans that were set with WithService. The service source targets service overrides (service set on spans) which is different from the base service source project that is set on process level tags

inheritedData serves as a default value for a span created as child of another span.

If the span has no WithService this means that we're still in the same service as the parent meaning that we can re-use the same value

If the span has WithService it will change the service and serviceSource values to what we want

mtoffl01 · 2026-03-12T14:34:43Z

ddtrace/tracer/span.go

+// or when no source was determined.
+// +checklocks:s.mu
+func (s *Span) enrichServiceSource() {
+	if s.serviceSource == "" || s.service == globalconfig.ServiceName() {


(For internal conversations)

@genesor see globalconfig.ServiceName() -- here is another example where global access to service name is blocked by config migration.

globalconfig was created to share tracer config data across packages; it works as the "source of truth" here only because globalconfig.SetServiceName is called after the tracer.config has evaluated its own service name.

Since we are within the tracer package here, we don't really have a need for it, other than it being a convenient API. It's probably safe to use in this case but it's not ideal.

Alternatives (other than completing service name migration) include...

get access to tracer.config's service name via GetGlobalTracer() and calling it's config field?

expand TracerConf() to share service name....

raphaelgavache · 2026-03-12T14:48:03Z

@mtoffl01 thanks for the review, regarding your questions:

new ServiceNameWithSource API, the goal is in follow-up PRs to enforce that all contribs only pass through that function to ensure that we always attribute the source of thier services. the span set service options will result in the m state, this is detailed in the RFC
on standardisation those values are hidden in internal fields _dd and if we ever make them user facing we will choose the string values in UI. The purpose right now is to use them for the service override migration tool and for troubleshooting purpose

genesor

nit about naming to have something a tad clearer otherwise this looks fine to me.

genesor · 2026-03-12T15:10:06Z

instrumentation/instrumentation.go

+const (
+	// ServiceSourceWithService is the service source value used when the service
+	// name is explicitly set via a WithService option.
+	ServiceSourceWithService = "opt.WithService"


nit: the name would be clearer if extended with `Option

Suggested change

ServiceSourceWithService = "opt.WithService"

ServiceSourceWithServiceOption = "opt.WithService"

genesor · 2026-03-12T15:14:57Z

contrib/database/sql/option.go

 	}
 	cfg.dbmPropagationMode = tracer.DBMPropagationMode(mode)
 	cfg.serviceName = defaultServiceName(driverName, rc)
+	cfg.serviceSource = serviceSourceSQLDriver


Would be nice to have the defaultServiceName func handling both name and source.

If it causes to much change having an helper func defaultSource could work.

genesor · 2026-03-12T16:15:39Z

contrib/gin-gonic/gin/option.go


+// serviceSourceGinMiddleware is the service source value used when the service
+// name is set via gin's Middleware(serviceName) parameter.
+const serviceSourceGinMiddleware = "opt.gin_middleware"


As far as I understand this new property the source is the place where the name comes from.

For SQL opt.sql_driver is used when the name is set by the package. It makes sense here as well because the middleware is the origin of the name and when no name is given we fallback to another source (default)

genesor · 2026-03-12T16:32:41Z

ddtrace/tracer/span.go

-		service:  s.service,
-		pprofCtx: s.pprofCtxActive,
+		service:       s.service,
+		serviceSource: s.serviceSource,


inheritedData serves as a default value for a span created as child of another span.

If the span has no WithService this means that we're still in the same service as the parent meaning that we can re-use the same value

If the span has WithService it will change the service and serviceSource values to what we want

github-actions bot added the apm:ecosystem contrib/* related feature requests or bugs label Mar 5, 2026

raphaelgavache force-pushed the raphael/srv_src branch from b2c209c to 4512e9e Compare March 5, 2026 04:23

DataDog deleted a comment from pr-commenter bot Mar 10, 2026

raphaelgavache force-pushed the raphael/srv_src branch from 5eb2d95 to 202b067 Compare March 10, 2026 00:41

raphaelgavache marked this pull request as ready for review March 10, 2026 01:09

raphaelgavache requested review from a team as code owners March 10, 2026 01:09

raphaelgavache added the changelog/no-changelog label Mar 10, 2026

introduce service source and migrate 4 integrations

3a4506d

raphaelgavache force-pushed the raphael/srv_src branch from 9004f07 to 3a4506d Compare March 10, 2026 01:32

hannahkm requested changes Mar 10, 2026

View reviewed changes

raphaelgavache added 2 commits March 10, 2026 11:36

update after review

18e175d

update - use drivername as service

94a9b75

hannahkm reviewed Mar 10, 2026

View reviewed changes

ddtrace/tracer/span.go Outdated Show resolved Hide resolved

remove dupe key

57f2c45

hannahkm approved these changes Mar 11, 2026

View reviewed changes

raphaelgavache mentioned this pull request Mar 11, 2026

feat: support service source for 19 integrations #4526

Open

9 tasks

na-ji approved these changes Mar 12, 2026

View reviewed changes

rarguelloF reviewed Mar 12, 2026

View reviewed changes

update after review

5928587

rarguelloF approved these changes Mar 12, 2026

View reviewed changes

mtoffl01 requested changes Mar 12, 2026

View reviewed changes

normalisation

6dc325a

raphaelgavache added 2 commits March 12, 2026 11:49

add case for empty driver and service source

cf7cddc

Merge branch 'main' into raphael/srv_src

e0fd07b

genesor reviewed Mar 12, 2026

View reviewed changes

raphaelgavache added 3 commits March 12, 2026 13:40

update tests

12e2c7c

restore unrelated file reodering

1973c77

lint remove trailing line

b9ab5b0

mtoffl01 approved these changes Mar 12, 2026

View reviewed changes

genesor approved these changes Mar 12, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: queued labels Mar 12, 2026

gh-worker-dd-mergequeue-cf854d bot merged commit 206b642 into main Mar 12, 2026
205 checks passed

gh-worker-dd-devflow-36fce6 bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Mar 12, 2026

gh-worker-dd-mergequeue-cf854d bot deleted the raphael/srv_src branch March 12, 2026 19:17

	ServiceSourceWithService = "opt.WithService"
	ServiceSourceWithServiceOption = "opt.WithService"

Conversation

raphaelgavache commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewer's Checklist

Uh oh!

codecov bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-official bot commented Mar 5, 2026 • edited by datadog-datadog-prod-us1 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:BenchmarkPartialFlushing/Disabled-25

scenario:BenchmarkSingleSpanRetention/no-rules-25

scenario:BenchmarkSingleSpanRetention/with-rules/match-all-25

scenario:BenchmarkSingleSpanRetention/with-rules/match-half-25

scenario:BenchmarkStartSpan-25

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hannahkm left a comment

Choose a reason for hiding this comment

Uh oh!

rarguelloF left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rarguelloF left a comment

Choose a reason for hiding this comment

Uh oh!

mtoffl01 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raphaelgavache commented Mar 12, 2026

Uh oh!

genesor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raphaelgavache commented Mar 5, 2026 •

edited

Loading

codecov bot commented Mar 5, 2026 •

edited

Loading

datadog-official bot commented Mar 5, 2026 •

edited by datadog-datadog-prod-us1 bot

Loading

pr-commenter bot commented Mar 10, 2026 •

edited

Loading