Single Sign-On with Keycloak on Kubernetes — Part 2

How to add Single Sign-On to applications without OIDC support using OpenResty and some Lua scripting.



In the [last blog post]({{< relref "" >}}), I described how to configure Keycloak on Kubernetes to enable Single Sign-On for multiple services. All these services had built in support for authentication and authorization using OpenID Connect. But what if a service doesn't?

In this part, I'll describe how to use an authentication reverse proxy in front of applications to secure them properly. A few advantages of OIDC will of course get lost: For example, fine-grained access control on the application level is not possible.

The reverse proxy

I was looking around for a good authenticating reverse proxy and came across Pomerium. But to me, this project seemed a bit too complex for my use case. I guess it's more apt for businesses with lots of users.

Then I stumbled upon lua-resty-openidc. It's a Lua based plugin to OpenResty, which is a web server built on Nginx with a lot nice features available.

The functionality is quite straighforward: You place lua-resty-openidc in front of your application. When a request comes in, the reverse proxy checks if the user is already authenticated via OpenID connect. If yes, it forwards the request to the backend application. If no, it redirects the user to the identity provider (Keycloak in my case) for authentication. Keycloak then redirects back after successful authentication. The backend application never sees anything of the authentication flow.

Building a Container

To deploy the reverse proxy in Kubernetes, I first had to package lua-resty-openidc as a Docker container. This is actually quite straightforward:

FROM openresty/openresty:

RUN ["/usr/local/openresty/luajit/bin/luarocks", "--global", "install", "--no-manifest", "lua-resty-session", "3.7"]

RUN ["/usr/local/openresty/luajit/bin/luarocks", "--global", "install", "--no-manifest", "lua-resty-http", "0.15"]

RUN ["/usr/local/openresty/luajit/bin/luarocks", "--global", "install", "--no-manifest", "lua-resty-jwt", "0.2.2"]

RUN ["/usr/local/openresty/luajit/bin/luarocks", "--global", "install", "--no-manifest", "lua-resty-openidc", "1.7.3"]

RUN : \
    && curl -s -L -o /usr/local/bin/dumb-init \
    && chmod +x /usr/local/bin/dumb-init

COPY ./ /start-nginx
COPY ./nginx.conf /usr/local/openresty/nginx/conf/nginx.conf

CMD ["/usr/local/bin/dumb-init", "--", "/start-nginx"]


The module versions are only a snapshot. Make sure to use the latest ones and to update them regularly (or automatically)!

The nginx configuration is very default-y:

# [...]

events {
    worker_connections  1024;

http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log logs/access.log combined;
    error_log  logs/error.log  info;

    # See Move default writable paths to a dedicated directory (#119)
    client_body_temp_path /var/run/openresty/nginx-client-body;
    proxy_temp_path       /var/run/openresty/nginx-proxy;
    fastcgi_temp_path     /var/run/openresty/nginx-fastcgi;
    uwsgi_temp_path       /var/run/openresty/nginx-uwsgi;
    scgi_temp_path        /var/run/openresty/nginx-scgi;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    # Required because of huge session cookies
    large_client_header_buffers 8 64k;
    client_header_buffer_size 64k;

    lua_shared_dict discovery 10m;

    lua_ssl_trusted_certificate /etc/pki/tls/certs/ca-bundle.crt;
    lua_ssl_verify_depth 2;

    # Include dynamically generated resolver config
    include /etc/nginx/resolver.conf;

    server {
        # Include dynamically generated listener config
        include /etc/nginx/listen.conf;
        server_name _;

        location /health {
            return 204;

        include /etc/nginx/conf.d/oidc.conf;

One interesting point is the large_client_header_buffers and client_header_buffer_size setting. More on that later.

As you can see, the configuration imports /etc/nginx/conf.d/oidc.conf, which is not part of the container. This file needs to be mounted into the container at runtime to configure the access rights.

The start script is straightforward as well:

#!/usr/bin/env bash

# We need to extract the DNS server from /etc/resolv.conf and put it into the
# `resolver` directive in nginx

nameservers="$(grep ^nameserver /etc/resolv.conf | cut -d ' ' -f 2 | paste -s -d ' ')"

echo "resolver ${nameservers};" > /etc/nginx/resolver.conf

echo "listen ${LISTEN_PORT:-80};" > /etc/nginx/listen.conf

exec /usr/bin/openresty -g "daemon off;"

Building the container and making it available to Kubernetes is left as an exercise to the reader ;)

Putting it into Kubernetes

Now we can use the container in Kubernetes. The idea is simple: Add it to a pod, expose only the port of the reverse proxy, and make the service use that port.

Here, I'm using Firefly III as an example, an absolutely awesome personal finance manager I use to manage all my income and expenses. It does not support OpenID connect by default. But it supports "remote user" authentication1

apiVersion: apps/v1
kind: Deployment
  name: firefly-iii
    app: firefly-iii
  replicas: 1
  revisionHistoryLimit: 0
    type: Recreate
      app: firefly-iii
        app: firefly-iii
        - name: firefly-iii-db
          image: ""
          imagePullPolicy: IfNotPresent
            - mountPath: '/var/lib/postgresql/data'
              name: database
          env: # snip
        - name: firefly-iii
          image: ""
          imagePullPolicy: IfNotPresent
            - mountPath: '/var/www/firefly-iii/storage'
              name: storage
          env: # [...]
            - name: AUTHENTICATION_GUARD
              value: remote_user_guard
              # So in PHP, header gets prefixed with HTTP_ and dashes
              # are replaced by underscores. In nginx, we set
              # X-AUTH-USERNAME and X-AUTH-EMAIL
              value: HTTP_X_AUTH_USERNAME
              value: HTTP_X_AUTH_EMAIL
            - name: CUSTOM_LOGOUT_URI
              value: /logout
        - name: oidc-proxy
          imagePullPolicy: IfNotPresent
            - containerPort: 80
              protocol: TCP
              name: http-oidc-proxy
            - name: nginx-oidc-config
              mountPath: /etc/nginx/conf.d/oidc.conf
              subPath: conf
              port: http-oidc-proxy
              scheme: HTTP
              path: /health
              port: http-oidc-proxy
              scheme: HTTP
              path: /health
        - name: database
            claimName: firefly-iii-db
        - name: storage
            claimName: firefly-iii-storage
        - name: nginx-oidc-config
            name: firefly-iii-nginx-oidc-config-v3

First, we need a configmap the contains the nginx configuration for /etc/nginx/conf.d/oidc.conf mentioned above:

kind: ConfigMap
apiVersion: v1
    app: firefly-iii
  name: firefly-iii-nginx-oidc-config-v3
  conf: |-
    set $session_storage cookie;
    set $session_cookie_persistent on;
    set $session_cookie_secure on;
    set $session_cookie_httponly on;
    set $session_cookie_samesite Strict;

    server_tokens off;

    location = /logout_done {
      return 200 'Logout done. <a href="/">Login again</a>';
      add_header Content-Type text/html;

    location / {
        access_by_lua_block {
            local opts = {
              redirect_uri = "",
              discovery = "https://{keycloak}/auth/realms/{realm}/.well-known/openid-configuration",

              client_id = "firefly-iii",
              client_secret = "{secret}",

              token_endpoint_auth_method = "client_secret_post",

              scope = "openid email profile firefly-iii",

              session_contents = {id_token=true, access_token=true},

              ssl_verify = "yes",
              accept_unsupported_alg = false,
              accept_none_alg = false,

              renew_access_token_on_expiry = true,

              logout_path = "/logout",
              post_logout_redirect_uri = "",

              revoke_tokens_on_logout = true,

            local oidc = require("resty.openidc")

            -- call authenticate for OpenID Connect user authentication
            local res, err = oidc.authenticate(opts)

            if err then
              ngx.log(ngx.CRIT, tostring(err))

            -- get acess token
            local parsed_token, token_err = oidc.jwt_verify(res.access_token, opts)
            if token_err then
              ngx.log(ngx.CRIT, tostring(token_err))

            -- get roles from access token
            local roles = (parsed_token["realm_access"] or {})["roles"] or {}

            local function has_role(role_name)
                for _, value in ipairs(roles) do
                    if value == role_name then
                        return true
                return false

            local allow = false

            -- all the setup is done. now we can check roles

            local email = parsed_token["email"]

            if has_role("firefly-iii:user") and email ~= nil then
                allow = true

            if allow == true then
                ngx.req.set_header("X-AUTH-USERNAME", email)
                ngx.req.set_header("X-AUTH-EMAIL", email)


You can see, all the authentication magic is done in the lua block. Actually, you're completely free to put any logic you want there. Here, I require the user to have the firefly-iii:user role in keycloak. In case of authentication success, the X-AUTH-* headers are set correspondingly. Firefly is configured to trust those headers.

The Cookie Problem

Now with 80% of the work done, I could expect to spend 80% of the time on the remaining 20%. I was not disappointed, because a very hard to debug problem showed up: Sometimes, the authentication would fail first with a 500 HTTP code, and then with a 502 on reload.

I checked the logs of nginx, and it complained that it could not decode the authentication cookie properly. It then started the authentication workflow again, leading to an infinite loop. This turned out to be quite hard to debug, mainly because there were so many components:

The first step was to reduce the number of components, or eliminate them from the list of potential sources of errors. The application itself could be the problem, because I saw the exact same error with other applications as well.

The next step was to strip down the setup to a minimal "proof-of-error". During that, I noticed that the error vanished as soon as I omitted access_token = true in the nginx lua configuration. I dug into the documentation of lua-resty-openidc and stumpled upon this FAQ entry:

Why does my browser get in to a redirect loop?

It may be that you are using the (default) cookie-only session storage of lua-resty-session library that lua-resty-openidc depends on and the size of the cookie becomes too large, typically >4096 bytes. See:

Solution: either make the size of the session cookie smaller by having the Provider include less information in the session/claims, or revert to server side session storage such as memcache, see:

That's it: The session cookie gets too large! But at which point are they dropped? I first suspected nginx to be the problem. So I set the following configuration in the openresty nginx config, which you already saw above:

large_client_header_buffers 8 64k;
client_header_buffer_size   64k;

This did not help though. As I saw later, it was one piece of the solution puzzle. So this setting is required, but it's not enough (yet).

So, to the next suspect: Kubernetes, or more specifically, the nginx ingress. To pinpoint the exact point, I set up SSH tunnels to different components from my local machine and checked authentication:

So the problem was also with the header settings of the Kubernetes ingress. I first checked the configuration. Effectively, the kubernetes nginx ingress is just a regular nginx with a dynamically generated configuration. So I used kubectl to take a look at that config:

kubectl -n ingress-nginx exec \
    $(kubectl get -n ingress-nginx pod \
        --field-selector=status.phase=Running \, \\
        -o jsonpath='{.items[*]}') \
    -- cat /etc/nginx/nginx.conf

I saw a few concerning defaults:

proxy_buffer_size 4k;
proxy_buffers     4 4k;

Note that the settings above (large_client_header_buffers and client_header_buffer_size) do not apply here, because nginx only acts as a proxy. Anyway, the values of the settings above are too small for the huge cookies that we need. So let's increase them!

Fortunately, the nginx ingress exposes these values as annotations2, which can be set on a per-ingress basis. So I updated the ingress manifest for the firefly application:

kind: Ingress
  name: firefly-iii
    app: firefly-iii
  annotations: / "nginx" "letsencrypt-production"

    # required for big session cookies in oidc authentication "8" "64k"

A kubectl apply later, authentication worked flawlessly!

Wrap up

I used the above solution to secure access to multiple other applications, as well. For example a Prometheus & Alertmanager setup that I use for some internal monitoring. Prometheus does not have any authentication story anyway, so this setup allowed me to easily secure access. But as already mentioned, there are a few drawbacks:

Due to that, I much prefer to use applications with proper OIDC integration. But it's still good as a stopgap measure!