Use PKI with external policy services
Note
The Certificate Issuance External Policy Service requires a Vault Enterprise license.
Challenge
When internal teams submit PKI certificate signing requests (CSRs), operators can be limited by Vault semantics when validating and controlling the issued certificate from PKI secrets engines.
Solution
Vault Enterprise version 1.15.0 introduces a new Certificate Issuance External Policy Service (CIEPS) feature for the PKI secrets engine. CIEPS enables PKI operators to have fine grained control over issued certificates using custom policies from a service defined and built by your organization that operates outside of Vault.
If a CSR does not conform to the rules defined in your policy service, the request is rejected.
Operators can also specify any subject attributes or extensions for issued certificates in the policy service.
Personas
The hands-on lab scenario described in this tutorial involves one persona:
alice
a security operator with privileged permissions to administer Vault secrets engines.
Prerequisites
To perform the tasks described in the tutorial hands-on lab, you need the following.
Vault Enterprise version 1.15.0 or greater binary installed and in your system PATH with valid license. Learn how to request a trial license.
Familiarity with TLS concepts like certificate fields and extended key usage.
Experience with creating and managing certificate authorities with the Vault PKI secrets engine. If you are unfamiliar with these concepts, complete the Build your own certificate authority (CA), and then return to this tutorial.
A text editor that you prefer to work with for creating a script.
Git installed and in your system PATH.
Go installed and in your system PATH for building the example policy service.
- Familiarity with building Go programs is helpful but not required.
A development environment that includes
make
and openssl for building the example policy service binary.curl installed and in your system PATH for API steps.
jq installed and in your system PATH for API output examples.
Versions used for this tutorial
This tutorial was last tested 23 Sep 2023 on macOS using the following software versions.
$ sw_vers --productVersion13.5.2
go version go version go1.21.1 darwin/amd64
$ make --version | head -n 1GNU Make 3.81
$ openssl version OpenSSL 3.1.2 1 Aug 2023 (Library: OpenSSL 3.1.2 1 Aug 2023)
$ vault versionVault v1.15.0-rc1+ent (d7e7b24dbf5f06fe8b3ba29cfcf4d8f7d961ed45), built 2023-09-11T17:44:31Z
$ curl --version | head -n 1 | awk '{print $2}'8.1.2
$ jq --version jq-1.7
Scenario introduction
As the Alice persona, you'll set up a lab environment consisting of an example external policy service and a Vault server.
You'll then configure a PKI infrastructure in Vault consisting of a root CA and intermediate CA. You'll then configure the intermediate CA to use the example external policy service.
After establishing the infrastructure, you can request a certificate and key through the external policy service and then examine the certificate to learn about the extension added by the policy service.
Lab setup
Your goal for this section is to deploy and prepare your external policy service and Vault environments for the steps which follow. This involves cloning the example service repository, building the example service from source, and starting the service.
You can a run Vault dev mode server from a terminal session for the purposes of this hands-on lab.
Create hands-on lab home
You can create a temporary directory to hold all the content needed for this hands-on lab and then assign its path to an environment variable for later reference.
Open a terminal, and create the directory
/tmp/learn-vault-pgp
.$ mkdir /tmp/learn-vault-pki
Export the hands-on directory path as the value to the
HC_LEARN_LAB
environment variable.$ export HC_LEARN_LAB=/tmp/learn-vault-pki
External policy service
The CIEPS protocol is a REST-based, optionally mTLS protected webhook. You can learn more about the protocol along with its request and response formats for interacting with Vault in the Certificate Issuance External Policy (CIEPS) documentation.
Development of an external policy service is beyond the scope of this tutorial, but you'll have access to source code for a public example that your engineering team can review as a starting point.
The example code shows how to add an extension to a certificate and how to return warnings to the end user. You can extend this example with validation logic as necessary to meet your use case requirements.
Build and run example policy service
Your goal for this section is to build and run an example external policy service in a terminal session.
The example external service is open source, written in Go, and its source code is publicly available from the vault-pki-cieps-example repository.
When you use the example policy service, the code from the Evaluate()
function in internal/business/policy.go
performs evaluation on the request.
In the example version, it also adds a custom extension to certificates so that you know it was processed with the service:
extMsg := "CIEPS Demo Server Certificate"
You should spot the string CIEPS Demo Server Certificate as an extension when decoding the certificate later.
Note
The example policy server processes the certificate request, but does not actually validate the request and instead returns a warning that no validation occurred. You must implement your own validation logic in your policy server code.
Clone the repository.
$ git clone https://github.com/hashicorp/vault-pki-cieps-example \ $HC_LEARN_LAB/vault-pki-cieps-example
Change into the vault-pki-cieps-example project directory.
$ cd $HC_LEARN_LAB/vault-pki-cieps-example
Build the example external service binary.
$ make build
Example output:
go build github.com/hashicorp/vault-pki-cieps-example/cli/cieps-servergo: downloading github.com/hashicorp/vault/sdk v0.10.0
The
Makefile
controls building of the binary and downloads the Vault client SDK for Go if necessary.The binary named
cieps-server
, is built and located in the present working directory.Run
cieps-server
with the--help
flag to learn about the configuration.$ ./cieps-server --help
Example output:
Usage of ./cieps-server:-listen string TCP listen address in host:port format (default "0.0.0.0:443")-server-cert string Path to the server certificate file (default "server.crt")-server-key string Path to the server key file corresponding to the given certificate file (default "server.key")
The service requires TLS certificate and key material to run; build the TLS certificate and key.
$ make certs
Example output:
openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt -sha256 -days 3650 -nodes -subj "/CN=localhost" -addext "subjectAltName = DNS:localhost".+...+......+.+........+++++++++++++++++++++++++++++++++++++++*.............+...+...+.....+...+.+.........+++++++++++++++++++++++++++++++++++++++*....+..+...+....+..+...+............+.+..+.+.....+.......+..+..................+.........+.............+...............+...+...+..+...+.........+.......+......+........+.+...........+.........+....+.........+........+....+.......................+..........++++++.........+...+..+......+.+.....+.+...............+.....+....+......+...+.....+....+++++++++++++++++++++++++++++++++++++++*......+++++++++++++++++++++++++++++++++++++++*.....+..................+.........................+..+...+.+...+.....+......+....+...+........++++++-----
The service is now ready for use.
Start the example policy service, instructing it to listen on all interfaces on TCP port 8443, and use the certificate and key files created from the last step. Redirect the standard output and standard error to the file
policy-server.log
, and background the process.$ ./cieps-server \ -listen=localhost:8443 \ -server-cert=./server.crt \ -server-key=./server.key > policy-server.log 2>&1 &
The example policy server is ready for use.
Vault dev mode server
Run a Vault dev mode server as a background process from your terminal session, and prepare it for use.
Change back into the hands-on lab home directory.
$ cd $HC_LEARN_LAB
Export an environment variable with a valid Vault Enterprise license.
$ export VAULT_LICENSE='02MC0FFEEBK5...'
Update example value
Use your actual Vault Enterprise license string in place of the example value shown.
Open a terminal and start a Vault dev server using the Vault Enterprise binary with
root
as the initial root token value.$ vault server -dev -dev-root-token-id root > "$PWD"/vault-server.log 2>&1 &
The Vault dev server defaults to running at
127.0.0.1:8200
. The server logs to the filevault-server.log
in the present working directory, and gets automatically initialized and unsealed.Dev mode is not for production
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and all contents are lost when the Vault server process is stopped.
Export the environment variable for the
vault
CLI to address the Vault server.$ export VAULT_ADDR='http://127.0.0.1:8200'
Check Vault status.
$ vault status
Example output:
Key Value--- -----Seal Type shamirInitialized trueSealed falseTotal Shares 1Threshold 1Version 1.15.0+entBuild Date 2023-09-11T17:44:31ZStorage Type inmemCluster Name vault-cluster-040c159eCluster ID 78d63616-f8ac-ef95-bcea-05387a693771HA Enabled false
The Vault server is ready for you to proceed with the hands-on lab.
Create a certificate authority
Your goal for this section is to enable PKI infrastructure consisting of 2 PKI secrets engine instances representing a root CA (pki) and intermediate CA (pki_int) for the example.com
domain.
This configuration is based on steps just like those in the Build Your Own Certificate Authority (CA) tutorial. You are encouraged to complete the hands-on lab in that tutorial if you are unfamiliar with the PKI secrets engine.
To save time and typing, you can use this example shell script to enable and configure the secrets engines for the hands-on lab.
Use a text editor to write and save this content to the file
/tmp/learn-vault-pki/enable_engines.sh
.enable_engines.sh
#!/bin/bash set -euxo pipefail vault secrets enable pki && \vault secrets tune -max-lease-ttl=87600h pki && \vault write -field=certificate pki/root/generate/internal \ common_name="example.com" \ issuer_name="root-2023" \ ttl=87600h > root_2023_ca.crt && \vault write pki/roles/2023-servers allow_any_name=true && \vault write pki/config/urls \ issuing_certificates="$VAULT_ADDR/v1/pki/ca" \ crl_distribution_points="$VAULT_ADDR/v1/pki/crl" && \vault secrets enable -path=pki_int pki && \vault secrets tune -max-lease-ttl=43800h pki_int && \vault pki issue \ --issuer_name=example-dot-com-intermediate \ /pki/issuer/root-2023 \ /pki_int/ \ common_name="example.com Intermediate Authority" \ o="example" \ ou="education" \ key_type="rsa" \ key_bits="4096" \ max_depth_len=1 \ permitted_dns_domains="test.example.com" \ ttl="43800h"vault write pki_int/roles/example-dot-com \ issuer_ref="$(vault read -field=default pki_int/config/issuers)" \ allowed_domains="example.com" \ allow_subdomains=true \ max_ttl="12000h"
Note
The values used here are just for the hands-on lab. You should change the values of
common_name
andissuer_name
to match your own values when using these commands outside of the context of the hands-on lab.Make the script executable.
$ chmod +x $HC_LEARN_LAB/pki/enable_engines.sh
Execute the script to enable and configure the secrets engines.
$ ./enable_engines.sh
Abbreviated expected output:
+ vault secrets enable pkiSuccess! Enabled the pki secrets engine at: pki/+ vault secrets tune -max-lease-ttl=87600h pkiSuccess! Tuned the secrets engine at: pki/ + vault write -field=certificate pki/root/generate/internal common_name=example.com issuer_name=root-2023 ttl=87600h+ vault write pki/roles/2023-servers allow_any_name=true...snip...street_address []ttl 0suse_csr_common_name trueuse_csr_sans trueuse_pss false
List the enabled PKI secrets engines.
$ vault secrets list
Example expected output:
Path Type Accessor Description---- ---- -------- ----------- cubbyhole/ cubbyhole cubbyhole_5030c17b per-token private secret storageidentity/ identity identity_21924a85 identity storepki/ pki pki_fd06124b n/apki_int/ pki pki_05cfa4c0 n/asecret/ kv kv_83eeb931 key/value secret storage sys/ system system_c0e34cd1 system endpoints used for control, policy and debugging
You have enabled the PKI secrets engines with a basic configuration.
Enable and configure CIEPS
Now that you've created your CAs in Vault, you need to enable and configure CIEPS on the intermediate CA so that you can generate a certificate and key with the external policy service.
Configure CIEPS using the Root 2023 CA and the PEM bundle created earlier.
$ vault write /pki_int/config/external-policy \ enabled=true \ external_service_url='https://localhost:8443/evaluate' \ trusted_leaf_certificate_bundle=@vault-pki-cieps-example/server.crt
Example output:
Key Value--- -----enabled trueentity_jmespath n/aexternal_service_last_updated 2023-09-25T15:16:23-04:00external_service_url https://localhost:8443/evaluateexternal_service_validated falsegroup_jmespath n/alast_successful_request n/atimeout 15000000000trusted_ca n/atrusted_leaf_certificate_bundle -----BEGIN CERTIFICATE-----MIIDHzCCAgegAwIBAgIUBueTj/kjYKpD+CNZl7RBQ/pNo28wDQYJKoZIhvcNAQELBQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIzMDkyNTE5MTEyMFoXDTMzMDkyMjE5MTEyMFowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvnc3P/Qvek2BaUbAs6rWqFboFYy8T+arRKmZSBnqyLt2z/xK4AsvFPVqMLt/VLKPpEGBIE78h1sxXuTCVNilG42I++xm5hD1V1M2D67nL3/oMgHRStjSn5c9dMYRcqrLu6+i+jhr53aomaw79EtImlPJdDQmO9rGvvP0287/CjU85OwKpzeUlvuJt9BsxwadWlzkrkSgwnHn+SqsQotWy2tY36H47almIzNDcQUh8ZVQuBmK6wzOazUHhQzh0zbR9UYztMw4Jsx/UgPeWlIIfy9o0V+rj4zAhwiEKEqOj4b4P0K3/2S5F/Oa7g0RmYZcRX6NPfNc9YEiVH6oLyEVkwIDAQABo2kwZzAdBgNVHQ4EFgQUjB4p4AZ9DVoanVbRsN1OrVLtkB4wHwYDVR0jBBgwFoAUjB4p4AZ9DVoanVbRsN1OrVLtkB4wDwYDVR0TAQH/BAUwAwEB/zAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZIhvcNAQELBQADggEBAFapM4qWqhVpd3wC84Yd12nJfKoBpTham8m6AXKI49x6oK6c6PwoPxqG7WyaN6GAXjLfZOShlSq5FTl8daqLxoHsh7Ym1uzCkqO+/1PpGP+AcJUc+wJTw6uOML6Ck9kcePqZ8ufYVwNn3ZfLkwEjgEFRnRUNVKJGt4maYDJMD+lWKkfLx9UpkGiWtKrQ1xb1in6qPRGzW2yra0ZC91bd9NiecGkAo8PVCWxdNZcZOEnSGqESdiUkWlVuO/b1xDLnAWkMbDePHavJx6GqnxX5s5em2CVGAA3+2aO4l62S20G1VkelJA8KhgK3Z2pOzyeCXJDvrgGIpoHGuAoxs7I3C0I=-----END CERTIFICATE-----vault_client_cert_bundle_no_keys n/a
Generate certificate and key with external policy
Your goal in this section is to use the Vault CLI or API to request a certificate and key from Vault that uses the example external policy service.
Tip
It is suggested to limit access to the path-overridden issue API endpoint (on
/pki/issuer/:issuer_ref/external-policy/issue/:policy
) and let the CIEPS
engine override the issuer as necessary.
Generate a certificate and key with the external policy service.
$ vault write \ /pki_int/issuer/example-dot-com-intermediate/external-policy/issue \ key_type=rsa \ key_bits=2048
Example abbreviated output:
WARNING! The following warnings were returned from Vault: * result from demo server; no validation occurredKey Value--- -----ca_chain [-----BEGIN CERTIFICATE-----MIIE3jCCA8agAwIBAgIUE4wpL+2L0VxF5lKxX6SR1V5xBzUwDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMjMwOTI1MTkxNTA2WhcNMjgw...snip...9po1Ulp0ctcjIL78+IeERNzz/wEFWkylTIzCBnOubkKqhg2I/FXYmviTt8TyvRLolz1QaWMcRc2nDCQy7NzriRbtFaywP8l5bPsisU6BvMVpQnLebj9Weg==-----END RSA PRIVATE KEY-----private_key_type rsaserial_number 61:8e:17:6f:15:fb:e6:1f:3b:1f:d1:40:f1:cb:66:96:d4:7c:96:60
Vault returns the certificate and key as expected. This behavior is the same for a client whether using CIEPS or not. The difference in this case though, is that the external policy service handled this request and emitted a warning message.
WARNING! The following warnings were returned from Vault: * result from demo server; no validation occurred
The service also added a custom extension to the certificate which Vault cannot natively add. You can parse the certificate with a tool to learn about this extension.
Capture a certificate and export its value in the CIEPS_CERTIFICATE
environment variable.
$ export CIEPS_CERTIFICATE="$(vault write -field=certificate \ /pki_int/issuer/example-dot-com-intermediate/external-policy/issue \ key_type=rsa \ key_bits=2048)"
Examine a certificate
You can use a tool to examine the certificate and learn about the fields included. You can use a CLI tool like openssl
or another application to examine the certificate.
Here, you'll use a web based tool to visualize the certificate content.
From your terminal session, copy the contents of the certificate file to your system clipboard.
$ echo $CIEPS_CERTIFICATE | pbcopy
Open your browser and navigate to http://lapo.it/asn1js/.
Paste the certificate content from your clipboard into the text area.
Click decode.
You can view all the certificate fields, including the custom extension and its message CIEPS Demo Server Certificate.
Next steps
You've built and run an example CIEPS server, deployed a Vault dev mode server and used the CIEPS server with the Vault PKI secrets engine. You also learned about the example CIEPS server code, user warnings, and how to a custom extension is added to a certificate by the CIEPS server.
The Vault PKI secrets engine is a popular solution, and there is much for you to learn about it. Some tutorials you can explore next to expand your PKI knowledge include Enable ACME with PKI secrets engine, PKI Unified CRL and OCSP with cross cluster revocation, and PKI secrets engine with managed keys.
Cleanup
Unset the environment variables.
$ unset VAULT_ADDR HC_LEARN_LAB
Stop the external policy service by executing the following command in your terminal session.
$ pgrep -f cieps-server | xargs kill
Stop the Vault dev server by executing the following command in your terminal session.
$ pgrep -f vault | xargs kill
Change into your home directory.
$ cd
Remove the hands-on lab home directory,
/tmp/learn-vault-pki
.$ rm -rf "$HC_LEARN_LAB"