Upgrading EKS amazon-cloudwatch-observability add on to 4.4.0 causes init container crash

[ssanthanam-jha]

App: Is a .net 8 Asp.net core API
Issue:
AWS EKS Environment, upgrading the amazon-cloudwatch-observability to version v4.4.0-eksbuild.1 causes the init container to crash with the following

Error Message with stack trace:

Defaulted container "App-qa" out of: App-qa, opentelemetry-auto-instrumentation-dotnet (init)
Cannot load library libgssapi_krb5.so.2
Error: libgssapi_krb5.so.2: cannot open shared object file: No such file or directory
Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Uncaught exception during type initialization. (0x80131534)
File name: 'Microsoft.Extensions.Logging.Abstractions, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
 ---> System.TypeInitializationException: The type initializer for 'OpenTelemetry.AutoInstrumentation.Loader.Loader' threw an exception.
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Could not find or load a specific file. (0x80131621)
File name: 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
 ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
   at System.Runtime.Loader.AssemblyLoadContext.<LoadFromPath>g____PInvoke|5_0(IntPtr __ptrNativeAssemblyBinder_native, UInt16* __ilPath_native, UInt16* __niPath_native, ObjectHandleOnStack __retAssembly_native)
   at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyBinder, String ilPath, String niPath, ObjectHandleOnStack retAssembly)
   at System.Runtime.Loader.AssemblyLoadContext.InternalLoadFromPath(String assemblyPath, String nativeImagePath)
   at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
   at System.Reflection.Assembly.LoadFrom(String assemblyFile)
   at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
   at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
   at System.Runtime.Loader.AssemblyLoadContext.OnAssemblyResolve(RuntimeAssembly assembly, String assemblyFullName)
   at OpenTelemetry.Sdk.CreateTracerProviderBuilder()
   at OpenTelemetry.AutoInstrumentation.Instrumentation.Initialize() in /_/src/OpenTelemetry.AutoInstrumentation/Instrumentation.cs:line 139
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at OpenTelemetry.AutoInstrumentation.Loader.Loader.TryLoadManagedAssembly() in /_/src/OpenTelemetry.AutoInstrumentation.Loader/Loader.cs:line 59
   at OpenTelemetry.AutoInstrumentation.Loader.Loader..cctor() in /_/src/OpenTelemetry.AutoInstrumentation.Loader/Loader.cs:line 35
   --- End of inner exception stack trace ---
   at OpenTelemetry.AutoInstrumentation.Loader.Loader.AssemblyResolve_ManagedProfilerDependencies(Object sender, ResolveEventArgs args)
   at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
   at System.Runtime.Loader.AssemblyLoadContext.OnAssemblyResolve(RuntimeAssembly assembly, String assemblyFullName)
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.<>c__DisplayClass16_0.<AddDefaultServices>b__0(ILoggingBuilder logging)
   at Microsoft.Extensions.DependencyInjection.LoggingServiceCollectionExtensions.AddLogging(IServiceCollection services, Action`1 configure)
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.AddDefaultServices(HostBuilderContext hostingContext, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostApplicationBuilder..ctor(HostApplicationBuilderSettings settings)
   at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)
   at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder(String[] args)
   at App.Api.Program.Main(String[] args) in /src/App.Api/Program.cs:line 30
   at App.Api.Program.<Main>(String[] args)

Workaround: downgrade to v4.3.1-eksbuild.1 fixes the problem.
Init Container Version (v4.3.1-eksbuild.1) - 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/observability/adot-autoinstrumentation-dotnet:v1.7.0

Init Container Version (v4.4.0-eksbuild.1) - 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/observability/adot-autoinstrumentation-dotnet:v1.9.0

[mddavisjha]

Our current workaround is updating our dependencies . For example, we update
from 8.0.1 along with various other dependencies from 8 to 9.
But this is requiring us to recompile and redeploy code so that we can update our addon without causing our pods to crash on startup afterwards.

[ssanthanam-jha]

For .NET 8.0 Blazor Server apps, upgrading the libraries to 9.** did not work. Had to upgrade the actual app to use .net 9 which fixed the problem.

[bjrara]

Thank you. Will take a look into the issue.

[bjrara]

It's a typical dependency conflict issue when using DOTNET auto-instrumentation solution: https://opentelemetry.io/docs/zero-code/dotnet/troubleshooting/#assembly-version-conflicts.

But this is requiring us to recompile and redeploy code so that we can update our addon without causing our pods to crash on startup afterwards.

To address this issue, you can explicitly specify the ADOT .NET instrumentation version in the Instrumentation configuration. This prevents automatic updates of the .NET instrumentation when the add-on version is upgraded, giving you more control over version management.

[ssanthanam-jha]

Upgrading to 4.7.0 cause a crash with this error

Unhandled exception. System.MissingMethodException: Method not found: 'Amazon.Runtime.ImmutableCredentials Amazon.Runtime.IRequestContext.get_ImmutableCredentials()'.
at OpenTelemetry.Instrumentation.AWS.Implementation.AWSTracingPipelineHandler.ProcessEndRequest(IExecutionContext executionContext, Activity activity)
at OpenTelemetry.Instrumentation.AWS.Implementation.AWSTracingPipelineHandler.InvokeAsync[T](IExecutionContext executionContext) in /home/runner/work/aws-otel-dotnet-instrumentation/aws-otel-dotnet-instrumentation/src/OpenTelemetry.Instrumentation.AWS/Implementation/AWSTracingPipelineHandler.cs:line 83
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
at CustomCode.AmazonSecretsManager.GetSecretAsync(String secretName) in /src/CustomCode/AWS/AmazonSecretsManager.cs:line 37