# Upstream mTLS With Pomerium
Part of a complete zero trust security model is secure communication between your identity-aware access proxy (Pomerium) and the upstream service it provides access to. This means both Pomerium and the upstream service will authenticate each other.
Pomerium confirms the identity of an upstream service by the TLS certificate it serves. See
tls_server_name for more information on configuring Pomerium to accept an upstream's TLS certificate.
Generally, every organization has a preferred method of managing internal certificates. How an upstream service is configured to validate and accept client certificates is entirely up to that specific service and how it's configured. To provide a general explanation, in this guide we will use mkcert (opens new window) to provide the client certificate to Pomerium, and OpenSSL (opens new window) as the upstream service.
# Before You Begin
To simplify this proof-of-concept, we assume that Pomerium is running in all-in-one mode as a system service with the configuration file in the standard location, configured to communicate with an identity provider (IdP), and all command are run on the same host. You may have to move files and/or adjust paths and commands to match a different configuration.
See Install - Binaries for more information on initial setup.
This guide uses mkcert (opens new window) to provide certificates. mkcert is a development tool, and not designed for production environments. In this guide, mkcert takes the place of your trusted certificate tooling solution.
*.pomerium.localhost.ioas the placeholder for the Pomerium domain space. If you're testing on your local computer you can edit
/etc/hostsor use dnsmasq on macOS (opens new window) or Linux (opens new window) systems to route this domain space to your local computer.
# Create Certificates
After installing mkcert (opens new window), confirm the presence and names of your local CA files:
mkcert -install The local CA is already installed in the system trust store! 👍 The local CA is already installed in the Firefox and/or Chrome/Chromium trust store! 👍 ls "$(mkcert -CAROOT)" rootCA-key.pem rootCA.pem
The output of
mkcert -install may vary depending on your operating system.
Create a certificate and key for our example upstream service, OpenSSL:
mkcert openssl.localhost Created a new certificate valid for the following names 📜 - "openssl.localhost" The certificate is at "./openssl.localhost.pem" and the key at "./openssl.localhost-key.pem" ✅ It will expire on 10 March 2024 🗓
Create a client certificate and key for Pomerium to use:
mkcert -client 'pomerium@localhost' Created a new certificate valid for the following names 📜 - "pomerium@localhost" The certificate is at "./email@example.com" and the key at "./firstname.lastname@example.org" ✅ It will expire on 10 March 2024 🗓
Change ownership of the client certificate files and move them to Pomerium's configuration directory:
sudo chown pomerium:pomerium pomerium@localhost-client*pem /etc/pomerium sudo mv pomerium@localhost-client*pem /etc/pomerium/
# Configure OpenSSL Server
OpenSSL is installed or easily available for most 'nix-based operating systems like macOS and Linux. In this demo it will take the place of an upstream service with mTLS support.
In a terminal environment and the same directory where you created the certificate files, start an OpenSSL server process. Note that it will run in the foreground until stopped, so you will need another terminal environment to run additional commands:
openssl s_server -key ./openssl.localhost-key.pem -cert ./openssl.localhost.pem -accept 44330 -www
You can confirm that the server is responding using
curl -k https://localhost:44330 <HTML><BODY BGCOLOR="#ffffff"> <pre> s_server -key ./openssl.localhost-key.pem -cert ./openssl.localhost.pem -accept 44330 -www Secure Renegotiation IS supported Ciphers supported in s_server binary ...
Even though OpenSSL is providing a certificate signed by a CA in our trust store, we need the
-kflag because the domain name
openssl.localhostdefined in our certificate.
# Configure Pomerium
/etc/pomeriumwith a new route for this upstream service:
- from: https://openssl.localhost.pomerium.io to: https://localhost:44330 tls_server_name: openssl.localhost policy: - allow: or: - email: is: email@example.com
Pomerium should automatically process changes when the configuration file is updated.
Visit https://openssl.localhost.pomerium.io (opens new window) in your browser to confirm the route:
Note the last line:
no client certificate available
In the next section, we'll adjust the OpenSSL server to require a client certificate and edit our Pomerium configuration to provide one.
# Install Pomerium Client Certificate
Stop the OpenSSL server process (Ctrl+C) and start a new one with the additional flag
openssl s_server -Verify 1 -key ./openssl.localhost-key.pem -cert ./openssl.localhost.pem -accept 44330 -www
When you refresh https://openssl.localhost.pomium.io (opens new window) in your browser, the connection will fail. Back in the terminal, the OpenSSL server should output errors containing:
SSL routines:tls_process_client_certificate:peer did not return a certificate
This confirms that the upstream service is now requiring a trusted client certificate to allow connections.
Update the Pomerium route block to provide the client certificate:
- from: https://openssl.pomerium.localhost.io to: https://localhost:44330 tls_server_name: openssl.localhost tls_client_cert_file: /firstname.lastname@example.org tls_client_key_file: /email@example.com policy: - allow: or: - email: is: firstname.lastname@example.org
Now when you visit https://openssl.localhost.pomium.io (opens new window) you should see additional output under "Client certificate", confirming that the upstream service has read and validated Pomerium's client certificate. Your upstream service is now using mTLS for mutual authentication!
Obviously, the OpenSSL server is a trivial upstream service, and is purpose-built to work with encryption and not do much else. Practical mutual authenticate can be a much more complex setup depending on the service, and may require a sidecar or service mesh. This exercise merely serves to demonstrate how easy it is to configure Pomerium to provide client certificates to the upstream service.