mechanism for adding information to a span after completion

[hughlunnon]

Hi

At the moment we're using the go.opentelemetry.io/otel/sdk/trace trace provider, with a few options. I'd like to be able to add a little metadata (specifically, the amount of time the span spent on cpu, which I'm calculating elsewhere, but isn't available for maybe 1 second after the span finishes) before the go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc pushes it to our collector.

It looked like a SpanProcessor would be a nice way to do this, as it supports the OnEnd function, and is called in order, so I could have a processor that adds my data, then go into the normal stack - however it seems this is only called with a snapshot of the span, which is immutable (so "can't" be edited).

Now, the attributes on the snapshot are a normal slice (I think), and as slices are passed by pointer I think I could modify some placeholder value with the actual value, but it seems like generally editing the value of a ReadOnlySpan should not be done.

I could also copy-paste the code from within the exporter, and add some custom code hook to do what I'm after just before we ship the data, but this also seems like an un-necessarily messy method.

Am I missing another hook that I can use to inject an attribute after span finish?

Thanks

Hugh

[dmathieu]

The OnEnding operation would be a solution for you.
It's still unstable in the specification, and not implemented in this SDK yet though.

See #6593

[hughlunnon]

thanks for pointing me to that - it looks like just the sort of thing I'm after.

at the moment that API asks for code to called within the span.end() call and non-blocking, which makes sense - in our case (where the measurement isn't available for a second or so), we wouldn't want to hold the calling function that long. I can do my work in a goroutine, but I'm guessing that risks a tonne of race conditions where the span has already been processed/shipped before I have the timings.

I think a decent option would be for me to start a goroutine that

1. marks an end time
2. do my asyncronous work
3. set the attributes
4. call span.End(actualEndTimeAsAnOption) to kick off all the otel work.

that feels slightly bloated but reasonable - can you think of any issues with that?

[dmathieu]

Sure, does feel bloated, but may work.
If you can gather the data from an outside process (like a sidecar container), a collector instance can also do data enrichment.

[flc1125]

I’m not entirely sure, so take this for reference only:

I’m thinking that perhaps we only need to wrap one of the existing SpanProcessor implementations with a custom SpanProcessor—let’s call it CustomSpanProcessor. It would rely on the SDK-provided SpanProcessor underneath (either Simple or Batch), add some extra support there, and ultimately delegate to the underlying SDK.