Keycloak on Kubeflow: User identity is empty in the request header

[gioargyr]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.10.2

Detailed Description

When I had Kubeflow v1.8, I used to access Kubeflow pipelines programmatically. Here is an issue where I participated by answering my question: awslabs/kubeflow-manifests#493

Since I installed Kubeflow v1.10.2 and I put Keycloak on top of it according to this guide https://github.com/kubeflow/manifests/tree/master/common/dex#kubeflow-dex--keycloak-integration-guide I managed to have Keycloak on top of Kubeflow when I log in through GUI, but I have troubles whenI try to access the pipelines programmatically.

I do -more or less- whatever I describe in this issue awslabs/kubeflow-manifests#493 , which is:

• getting a token from Keycloak
• Using the token in my request
  but, I always get this error: User identity is empty in the request header

I understand that instead of going directly to Kubeflow -> Keycloak -> Kubeflow, the flow is now: Kubeflow -> Dex -> Keycloak -> Kubeflow, so Dex is still in the flow.
As a result, the token I get from Keycloak has an issuer that is not included in Istio.
Please, correct me if I am wrong.

Do you have any ideas on how can I get a token from Keycloak that is approved by Istio?
And generally, can you point me to any documentation that might help me?

Kind regards,
Giorgos

Steps to Reproduce

1 Install Kubeflow v1.10.2 according to the main documentation
2. Follow this documentation https://github.com/kubeflow/manifests/tree/master/common/dex#kubeflow-dex--keycloak-integration-guide to put Keycloak in front of Kubeflow.

Screenshots or Videos (Optional)

No response

[juliusvonkohout]

Hello, may you upgrade to Kubeflow 26.03+ first, 1.10.2 is too old.

[gioargyr]

Thank you very much for your answer @juliusvonkohout

I will surely go to this direction (upgrading).
In the meantime, as I was trying to understand all the authorization/authentication system, I read this documentation: [1] https://github.com/kubeflow/manifests/blob/master/common/oauth2-proxy/README.md

I understand that you can have an authentication system and ignore Dex (as I used to do in the past, when I put Keycloak in front of Kubeflow).
Is documentation [1] in contrast to this guide [2] https://github.com/kubeflow/manifests/tree/master/common/dex#kubeflow-dex--keycloak-integration-guide ?
Is documentation [1] for newer versions (e.g. after Kubeflow 1.10), or the guide [2] is still valid?

I'll come back to this issue if/when I have something solid.

[gioargyr]

[1] https://github.com/kubeflow/manifests/blob/master/common/oauth2-proxy/README.md
[2] https://github.com/kubeflow/manifests/tree/master/common/dex#kubeflow-dex--keycloak-integration-guide
[3] awslabs/kubeflow-manifests#493

I am mentioning the links above as [1], [2], and [3] so I can describe easier my experience.

While being in v1.10.2:

• I apply the guide [2] and I put Keycloak in front of Dex and Kubeflow. The Kubeflow GUI works, but the programmatic access as discussed in [3] doesn't work because the token provided by Kubeflow is not valid.
• I applied the documentation [1] (which is similar to [2] but we ignore Dex). The Kubeflow GUI does NOT work, but the programmatic access outside Kubeflow as discussed in [3] DOES work and the token provided by Kubeflow is valid.

I upgraded to v1.11.0:
I have the exact same result as in v1.10.2

I upgraded to 23.06:
This time I have bigger issues, as the Ingress that points to service istio-ingressgateway is not working at all and I cannot have access to Kubeflow. (it's the same Ingress that worked for v1.8.0, v1.9.1, v1.10.2, and v1.11.0 !!)
BUT! If port-forward the service I have access to it! So, first I need to fix this and then try the rest. (probably I will open a new ticket for this one).

So, my problem is not solved, but at least there is a reference (and some progress) here.
My actual question is: what is the difference between the guide [2] and the documentation [1]. Can we have a way that Keycloak is in front of Kubeflow (preferably without Dex) and both GUI and programmatic access work?

[juliusvonkohout]

Can we make it work technically? yes. But that guide is offered by volunteers on a best effort basis. If you need commercial consulting that would be another level. https://www.kubeflow.org/docs/started/support/#support-from-commercial-providers-in-the-kubeflow-ecosystem