Update opentelemetry_exporter headers at runtime.

[francois-caddet]

Descritption

I try to export my elixir app's traces to the official opentelemetry collector.

To configure headers to inject in the opentelemetry exporter http/grpc messages, what I understoud is that we must set it in a [{String.t(), String.t()}] list.

My issue is that the collector I use is protected by an OAuth2 bearer authentication method:

• The app authenticate on an IAM and get a short live jwt token.
• The collector is configured with the OIDC extension and so check the token authenticity on the standart oidc endpoint for that.
• Every application set the token given by the IAM in the header authorization: Bearer {TOKEN} when they emit an otlp event.
• This token expire in a short amount of time an so the apps need to renew it periodicaly.

All the process to authenticate on the IAM and to renew the token on time is not an issue. However, since I get the new token,
I don't know how I could update the token passed in the headers of the exporter.

Proposed options

This issue may be fixed with one or more of these updates:

• Add a method to update the exporter options at runtime or at least the headers. It may be in two steps (e.g.: :opentelemetry_exporter.set_opt(:otlp_headers, [{"authorization", "Bearer {MY_NEW_BEARER}") then :opentelemetry_exporter.refresh_config())
• Accepting a function as config value for the :otlp_headers which will be called for each message send.
• A maybe more flexible option could be to give a way to add http/grpc middlewares/interceptors at startup or in the config. I thing it can easyly be implemented for grpc with grpcbox but for the http export, I have no idea.

Other options explored

Setting the application env otlp_headers before the token expire an letting the exporter crash and restart.
Tryed it without success for now. NB: I'm a junior in elixir development.
Anyway, I feel it is not the nicer way to handle my usecase.

Any other options accepted ;)

[tsloughter]

Is there something between your app and the collector doing auth or is this built into the collector?

If it is builtin to the collector I'd assume there is prior work to specify how the exporter should negotiate a new token and we can build that. But if not this is likely to be tricky/messy.

[francois-caddet]

Hello, thanks, I was not expecting a answer that soon :)

I just edited the issue trying to clarify these details.

Shortly:

There is an authitication service in the stack. The apps authenticate on it and get a jwt token in exchange.

The collector is configured with the oidc extension.

I know the collector config is working as expected because I have apps written in rust or in python in the stack for which the authentification and the telemetry in general is working well.

In my edit I add a 3rd resolution option which may actualy be more idiomatic one. Simply be able to add a list of middlewares/interceptors in the config/setup option. A couple of extra fields like grpc-interceptors and http-middlewares can be great I think.

However as noted in the issue, my short experience with elixir does not let me the knoledges to find what could be the best resolution for that.

Feel free to share your proposals and to explain them. It will help me in my elixir/erlang learning probably :)

[tsloughter]

Ah, do you know if the rust/python options are setting middlewares on the grpc/http clients directly?

We should definitely expose such an option. It'll only exist for grpc tho since the http client, httpc, has no concept of middlewares.

[francois-caddet]

Yes, in both python and rust it's done by passing a callback function:

• When you construct the exporter in python you can pass an arg credentials as you can see in the opentelemetry python documentation e.g.: opentelemetry.exporter.otlp.proto.grpc.trace_exporter.OTLPSpanExporter(endpoint=None, insecure=None, credentials=None, headers=None, timeout=None, compression=None, channel_options=None). In our code, we pass a grpc.metadata_call_credential from the python grpc lib. It is more or less a callback which take a context and a function to call to update the headers. I don't know why to pass a function and not simply update the headers with what return the callback.
• • In rust, you build the tonic connection and give it already configured to the exporter. it also has a system of callbacks to update the metadata of the connection at call/send time.

The important thing to notice is that in both cases, the designed followed is to take an object from the grpc library in param of the builder of the exporter.

Both python and Rust grpc libraries accept to be configured with a callback which will be called every time we send a message/call a grpc endpoint.

I think the same approch for elixir/erlang could be the good dirrection. I don't know how grpcbox handle metadata settings but I hope it accept a system of callback too.

About the HTTP part, I don't know, we are not using them.