I had misconfigured my Observatorium with an invalid client ID and OIDC secret in my K8s observatorium-xyz-observatorium-api Secret, while following https://github.com/observatorium/observatorium/blob/main/docs/usage/getting-started.md
token-refresher did not work, as expected, but it was very hard to troubleshoot. Hundreds of pages of token-refresher logs, mostly http: proxy error: dial tcp 127.0.0.1:8443: socket: too many open files.. No Observatorium-api logs.
I had no idea what was going on. It looks like net/http/httputil/reverseproxy.go closes files under defer. The incredible amount of output mislead me to suspect a recursion problem.
Perhaps more logging or rate limiting around rejection would help.
I had misconfigured my Observatorium with an invalid client ID and OIDC secret in my K8s observatorium-xyz-observatorium-api Secret, while following https://github.com/observatorium/observatorium/blob/main/docs/usage/getting-started.md
token-refresher did not work, as expected, but it was very hard to troubleshoot. Hundreds of pages of token-refresher logs, mostly
http: proxy error: dial tcp 127.0.0.1:8443: socket: too many open files.. No Observatorium-api logs.I had no idea what was going on. It looks like net/http/httputil/reverseproxy.go closes files under
defer. The incredible amount of output mislead me to suspect a recursion problem.Perhaps more logging or rate limiting around rejection would help.