How to make a custom SamplingDecision for a started span?

[un000]

Hi!

I have a service that starts tracing with a given ratio:

	jaeger.WithSDK(&trace.Config{DefaultSampler: trace.TraceIDRatioBased(0.001)}),

So I want to override this Sampler with a trace.AwlaysSampler like I did it with opencensus.

		opts := []trace.SpanOption{trace.WithNewRoot(), trace.WithSpanKind(trace.SpanKindServer)}
		forceTrace := len(reqCtx.Request.Header.Peek("X-Force-Trace")) > 0
		if forceTrace {
			opts = append(opts, trace.WithRecord())
		}

		tr := otel.Tracer(tracer.DefaultInstrumentationName)
		ctxWithSpan, span := tr.Start(ctx, name, opts...)

trace.WithRecord() enables recording but doesn't affect the sampling decision.

How to achieve 100% sampling by a flag?

[MrAlias]

Currently the OpenTelemetry specification does not allow this type of sampling override option, it would provide the instrumentation author a mechanism that could drastically conflict with what an operator would want regarding sampling.

The Sampler interface can be implemented if you want to create custom sampling algorithms. This would allow you to implement the described algorithm if you wanted to write your own sampler.

[atombender]

Hijacking this old thread: Is this the officially recommended solution? I noticed that samplers get the attributes as an array of KeyValue structs. This means a linear search on every single span rather than a direct boolean flag lookup. Is there a more optimal approach?

I have also noticed that sampling appears to only affect spans and their children, not parents. So if I have a chain span1 -> span2 -> span3 and I flag span2 as "sampled", this only causes span2 and span3 to be recorded. (I'm just assuming span3 is included here, with the ParentBased sampler; I definitely get span2, but I don't have a scenario that creates child spans.) But this means that the root is lost. As far as I know, with OpenTracing you get the whole chain. Maybe I'm doing something wrong?

[Yamparala-Venkata-Gopi]

To force 100% sampling via a request header, you need to implement a custom Sampler and wrap it around your existing ratio-based sampler. Here is a concrete implementation:

type ForceTraceSampler struct {
    fallback trace.Sampler
}

func (s ForceTraceSampler) ShouldSample(p trace.SamplingParameters) trace.SamplingResult {
    // Check if the force-trace attribute was set on the span
    for _, attr := range p.Attributes {
        if attr.Key == "force.trace" && attr.Value.AsBool() {
            return trace.SamplingResult{
                Decision:   trace.RecordAndSample,
                Tracestate: p.ParentContext.TraceState(),
            }
        }
    }
    return s.fallback.ShouldSample(p)
}

func (s ForceTraceSampler) Description() string {
    return "ForceTraceSampler"
}

Then at the trace start point, inject the attribute based on your header:

forceTrace := len(reqCtx.Request.Header.Peek("X-Force-Trace")) > 0

attrs := []attribute.KeyValue{}
if forceTrace {
    attrs = append(attrs, attribute.Bool("force.trace", true))
}

ctxWithSpan, span := tr.Start(ctx, name,
    trace.WithSpanKind(trace.SpanKindServer),
    trace.WithAttributes(attrs...),
)

And wire it in when building your TracerProvider:

tp := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(
        ForceTraceSampler{
            fallback: sdktrace.TraceIDRatioBased(0.001),
        },
    ),
)

On the efficiency question from @atombender — the linear attribute scan is generally fine since ShouldSample is called once per span creation and the attribute list at that point is small (just what you pass to Start). If you want O(1) lookup, you can use a context key instead of attributes:

type forceTraceKey struct{}

// set in middleware
ctx = context.WithValue(ctx, forceTraceKey{}, true)

// check in sampler
func (s ForceTraceSampler) ShouldSample(p trace.SamplingParameters) trace.SamplingResult {
    if p.ParentContext.Value(forceTraceKey{}) == true {
        return trace.SamplingResult{Decision: trace.RecordAndSample}
    }
    return s.fallback.ShouldSample(p)
}

This avoids the attribute scan entirely and is safe across goroutines since context.Context is immutable.

[ManithSanvidu]

trace.WithRecord() only enables recording on the span if it is already sampled. It does not override the sampler or change the sampling decision.

In OpenTelemetry, the sampler is consulted only when a span is created. After that, the sampling decision cannot be changed for that span.

If you want to force 100% sampling based on a request header such as X-Force-Trace, you have two common options:

1. Implement a custom Sampler that checks the context (or baggage/attributes propagated from the request) and returns sdktrace.RecordAndSample when the flag is present. Otherwise, delegate to your default sampler (e.g., TraceIDRatioBased(0.001)).

2. Create a separate TracerProvider configured with sdktrace.AlwaysSample() and use it only for requests where X-Force-Trace is present.

There isn't a built-in span option that overrides the sampler after tr.Start() has been called. The sampling decision must be made by the configured sampler at span creation time.