SSO Authentication for Applications in Kubernetes

courtesy to unsplash.com

Introduction

This post is about securing applications and ingresses with SSO authentication and authorization. Ingresses can be protected with username and passwords. If many teams are working on a cluster with many applications, it would be better to use an SSO authentication mechanism with RBAC(Role bases access control). This makes life easier as the groups that are allowed / denied access can be configured easily.

Current setup

I have a Traefik ingress controller and there are lot of ingresses. And I want to use GitHub as an OIDC provider.

Prerequisite

To understand and follow this documentation, I assume the reader has a bit of knowledge on deploying apps with helm and when making changes to the values.yaml file for an app, it is assumed that the chart release is upgraded with new values.

DEX as OIDC Provider

There are many OIDC providers that are available, but I chose to go with Dex, because

• Its not tied up with Kubernetes, I can use Dex for protecting my applications or all ingresses.
• Its feature rich
• It has many plugins which can integrate with many OIDC providers and can act like a single hub for authentication and authorization in the cluster.

So instead of using GitHub directly for OIDC config, I will use Dex as the OIDC provider inside the cluster and configure GitHub plugin on Dex for OIDC.

For applications that can communicate directly with an OIDC provider, can be protected by adding the OIDC config in the application configuration itself. This is cool. But what is uncool, is about ingresses or applications that do not support OIDC configuration. So lets see how we can do this.

Creating a GitHub OAuth Application

First things first

Go to github.com > Your Profile > Settings > Developer Settings > OAuth Apps > New OAuth App

The Application Name , Homepage URL, Application Description can be anything. The important is Authorization callback URL . This should be the url to Dex that will be deployed.

When the App is registered, a ClientID and a ClientSecret is generated. Copy this in a safe place.

Now lets create a secret in Kubernetes to store this credentials that will be used by Dex

dex-github-secret.yaml

and apply it in cluster

kubectl apply -f dex-github-secret.yaml

Dex as OIDC provider for Applications that CAN talk OIDC

I have ArgoCD for GitOps in my cluster. ArgoCD can talk to OIDC providers directly. So I will configure ArgoCD to use Dex as OIDC. The auth workflow goes like this

DEX workflow for app with OIDC support

1. User requests the application ArgoCD
2. ArgoCD forwards auth request to the OIDC provider configured, which is Dex.
3. Dex will check for tokens already exists or not. Since there is not one, so it will send the request to the OIDC connector configured.
4. The GitHub OAuth app authenticates the user in the request and returns the feedback to the configured Authorization Callback URL in the OAuth app, which is the Dex url
5. Dex verifies the response and creates signed ID tokens to the client that requested via the RedirectURI configured in staticClients.
6. ArgoCD will now forward the requested resource as its authenticated with a valid token

This is the workflow explained in simple terms. In reality there is lot of stuff happening, like authentication and creation of signed ID tokens or JWT tokens and so on. For that I would advise to check more into how OIDC authentication works. Or wait for sometime, when I write one about it ;)

Now lets deploy Dex. Since I am using Kubernetes, I will prefer to deploy via helm chart. So my values.yaml will look like this:

The storage part mentions where to store the tokens, I use kubernetes cluster for storage. And use Github connector.

If loadAllGroups is true then anyone logged in GitHub will be able to pass the authorization. Here I am telling Dex to authorize users from only myorg and who belongs to the teams myteam1 and myteam2. If I dont provide the teams config, then anyone part of the org myteam will be able to be authorized.

One important thing to note is that, if this Oauth App is a personal user created, then this app should be authorized by the organization. When using for the first time, Github will ask you to send request to the organization to approve the request.

In the staticClients section, I have added one for ArgoCD. The id and secret will be used as clientID and clientSecret in the oidc.config respectively

For more configuration on Github connector, please check here

Lets deploy Dex

kubectl apply -f dex-github-secret.yaml
helm repo add dex https://charts.dexidp.io
helm install --generate-name --wait dex/dex -n dex -f values.yaml 

When Dex is ready, configure ArgoCD config to add OIDC provider. I am not explaining it here. A TL;DR; would be to edit the config map argocd-cm and add the OIDC config and try to login. If properly configured, then the OIDC config should work.

Dex as OIDC provider for Applications that CANNOT talk OIDC

Now lets look at using Dex for applications that do not have OIDC support. We have two options.

Option 1: Traefik Forward Auth

Traefik Forward Auth is a simple SSO/Oauth authentication tool for Traefik as ingress controller.

The request workflow will be like this:

1. User requests for App to Traefik Ingress controller
2. Traefik redirects request to Traefik Forward Auth, through a middleware that is configured on Traefik
3. Traefik Forward Auth forwards the request to Dex
4. Dex forwards redirects request to GitHub
5. GitHub accepts the login and sends the auth request to Dex
6. Dex redirects the request to Traefik Forward Auth
7. Traefik Forward Auth checks the response from Dex and sends 200/403 based on the response
8. Traefik then sends the user request to the requested resource or app.

Configure Traefik Forward Auth

Before creating Traefik Forward Auth, we need to add a static client section in Dex. We already did the same for ArgoCD. So append the following to staticClients section of the values.yaml file for Dex:

- id: traefik-fwd-auth
  name: Traefik Forward Auth OIDC Dex App
  redirectURIs:
  - https://auth.mydomain.com/_oauth
  secret: random-secret

The redirectURIs should be the ingress url for Traefik Forward Auth which will configure soon.

The values for the Traefik Forward Auth helm chart is like this:

So here I am setting

• default oidc provider
• oidc section contains OIDC information to pass through. The issuerUrl is the configured Dex application URL. The clientID and clientSecret are added here and this should be exactly the same one as you provide in the staticClients section
• cookie.insecure to true , so that the cookie is allowed from non https domains because I have ssl termination on traefik. A random secret can be provided here. This will be stored as a secret in the namespace where Traefik Forward Auth is installed. The domain should be the value of the domain
• authHost is important, as this is the hostname that Traefik Forward Auth configures for authentication
• middleware.enabled set to true will create a middleware automatically. This middleware contains information on the auth url and service to point to.
• An ingress for Traefik Forward Auth

The original helm chart may have issues when installing, since, I had some of them. So it would be better to clone this chart and modify the chart as your requirement. For me the issue was with the ingress config, because I was using the latest kubernetes version and the version was outdated.

Deploy Traefik Forward Auth

helm repo add itscontained https://charts.itscontained.io
helm install itscontained/traefik-forward-auth -n traefik -f values.yaml

After installing,modify an unauthenticated ingress by adding the annotation:

traefik.ingress.kubernetes.io/router.middlewares: traefik-traefik-forward-auth@kubernetescrd

The value of annotation will be in the format <namespace>-traefik-forward-auth@kubernetescrd . If namespace or kubernetescrd is omitted, then this auth will not work.

Try to login and if everything is good, then you should be redirected to GitHub via Dex and if authentication works, then you will be authorized to access the Ingress

Option 2: Oauth2-Proxy

Oauth2-Proxy is a reverse proxy that provides authentication using Providers (Google, GitHub, and others) to validate accounts by email, domain or group.

The authentication workflow for Oauth2-Proxy looks like this:

1. User requests the App to Traefik
2. Traefik forwards the request to requested ingress, which is the Oauth2-Proxy
3. Oauth2-Proxy redirects request to Dex, its OIDC provider
4. Dex redirects request to GitHub, its OIDC provider
5. GitHub accepts login and forwards request to Dex via callback URL
6. Dex forwards request to Oauth2-Proxy via Redirect URL
7. Oauth2-Proxy authorizes request and sends the request to the Upstream URL configured

The difference between Traefik Forward Auth and Oauth2-Proxy is that, The ingresses are separate for Traefik Forward Auth and the application ingress. In Oauth2-Proxy, the app ingress is pointing to Oauth2-Proxy as it is acting like a proxy for the application

Configure Oauth2-Proxy

First a static client on DEX. So append the DEX values.yaml with the following in staticClients section

- id: oauth2-proxy-oidc
  name: Oauth2-Proxy OIDC Dex App
  redirectURIs:
  - https://myapp.mydomain.com/oauth2/callback
  secret: random-secret

The values for Oauth2-Proxy for a sample app is:

so there is:

• config section for the OIDC config, provided in Dex staticClients section
• ingress with tls config. Note that, I am using a wildcard dns certificate, stored as a secret in all namespaces.
• extraArgs, where more config is added. Like the oidc-issuer-url, redirect-url, allowed-group, upstream. In upstream is where we configure where to send request if auth is successful. Here it is pointing to a kubernetes service named myapp running in default namespace at port 8080

Deploy Oauth2-Proxy

helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install my-release oauth2-proxy/oauth2-proxy -n default -f values.yaml

If everything is working good, then the url https://myapp.mydomain.com will show a screen for Oauth2-Proxy and then redirects to Dex, who will redirect to GitHub and then to the app.

The important question. Traefik Forward Auth or Oauth2-Proxy?

Lets take a look why Oauth2-Proxy is not ideal in this scenario. When there is multiple ingresses that needs to be protected, then you will need same number of Oauth2-Proxy helm charts, which is not a good idea because then it will be Oauth2-Proxies all over the place.

Although Oauth2-Proxy can be used as a central authentication system for all ingresses, which means one instance can be used for all auth. This will work perfectly fine, when you have NGINX as your ingress controller. Then you deploy the Oauth2-Proxy normally with setting reverse-proxy to false and then remove upstream parameter, and set the hostname something like oauth.mydomain.comfrom the above config. Also deploy it in a namespace like security

Then on the application ingress add the following annotation

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-signin: https://oauth.mydomain.com/oauth2/start
    nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.security.svc.cluster.local/oauth2/auth
...

• The auth-siginredirects any needed login to the Oauth2-Proxy Ingress.
• The auth-url annotation can access the Oauth2-Proxy internally via its service to verify a submitted token

Sadly this annotations will not work with Traefik Ingress Controller. Which is why Traefik Forward Auth is best in this case.

Can we use both?

Yes, its fine to have both as there will be use-cases for that. For example, you are part of a Platform Engineering team and you created Traefik Forward Auth so that customers use this middleware to protect their ingresses and access is allowed only for the customer provided groups. Now in this cluster, if you as part of a team need to deploy something, like Prometheus for example, and expose the ingress and protect it with your own groups(not customer groups), then an instance of Oauth2-Proxy will do the trick here without much complications

Also its unwise to have more than, lets say 3 or 5 Oauth2-Proxy instances to be running in the cluster, unless its fine for you. For me, that will look like an ugly solution

Conclusion

Dex as an OIDC provider, is a really cool tool. One can integrate all OIDC providers into one hub inside the cluster and use each provider specific config for authentication. And using Traefik Forward Auth or Oauth2-Proxy adds more security for ingresses or applications that do not support in-built OIDC config. The perfect choice here is Traefik Forward Auth, because in my setup I have Traefik as the ingress controller. It should also be noted that, if I use NGINX ingress controller, I will not be able to use Traefik Forward Auth at all, as it requires Traefik because of the Middleware configuration.

I hope you have liked this post and understood a bit of the concept of using this in your setup

Let me know your comments, if you have some other options available, I will be happy to try and document it as well!!