Manage secrets across namespaces
Enterprise Feature
Vault namespaces feature is a part of Vault Enterprise.
This tutorial demonstrates the capability introduced in Vault Enterprise 1.13.0. If you are not familiar with Vault Namespaces, complete the Secure Multi-Tenancy with Namespaces tutorial first. In addition, the Vault Namespace and Mount Structuring Guide provides some guidance on designing the namespace hierarchy.
Challenge
The purpose of Vault namespaces is to create an isolated Vault environment within a cluster so that each organization, team, or application can manage secrets independently. Therefore, Vault clients must authenticate into a specific target namespace where the secrets live.
In some use cases, this imposes a burden on the Vault clients especially if the client must read secrets from dozens of independent namespaces (namespaces with no hierarchical relationship). This operational overhead becomes more prominent when you are using Vault Agent to automate secrets management.
Tip
When namespaces have hierarchical relationship, you can write namespace-aware policies to allow access to secrets in the child namespaces.
Solution
Vault Enterprise 1.13.0 introduced the group_policy_application_mode
flag
which enables secrets sharing across multiple independent namespaces. With this
change, a single instance of the Vault Agent can fetch secrets across multiple
namespaces. (See Cross namespace secret sharing
for an Agent injector example.)
Tutorial example use case
This tutorial demonstrates the secrets management across namespaces
funcationality using the userpass
auth method, however, all auth methods are
supported. Refer to the configure cross namespace access
documentation
for more information.
Prerequisites
- Familiarity with Vault namespaces or completed the Secure Multi-Tenancy with Namespaces tutorial
- Vault Enterprise binary v1.13.0 or later
Lab setup
Open a terminal and set your Vault Enterprise license string as the value of the
VAULT_LICENSE
environment variable.$ export VAULT_LICENSE=02MV4UU43BK5....
Start a Vault dev server with
root
as the root token.$ vault server -dev -dev-root-token-id root
The Vault dev server defaults to running at
127.0.0.1:8200
. The server is also initialized and unsealed.Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the
vault
CLI to address the Vault server.$ export VAULT_ADDR=http://127.0.0.1:8200
Export an environment variable for the
vault
CLI to authenticate with the Vault server.$ export VAULT_TOKEN=root
The Vault server is ready.
Scenario setup
To demonstrate the secrets sharing across namespaces, you need the following:
- New namespaces:
us-west-org
andus-east-org
- In the
us-west-org
namespace:- Enabled userpass auth method and created a "tam-user" with
customer-info-read-only
policy attached. The password is "my-long-password" - Enabled KV v2 secrets engine at
kv-customer-info
with some test data - Create a
customer-info-read-only
policy allowing to read secrets atkv-customer-info/data/*
- Created an identity entity, "TAM" and added "tam-user" as a member (an entity alias)
- Enabled userpass auth method and created a "tam-user" with
- In the
us-east-org
namespace:- Enabled KV v2 secrets engine at
kv-marketing
with some test data - Created a marketing-read-only policy allowing read permission on the
kv-marketing/data/campaign
path - Create an entity group named "campaign-admin" with the marketing-read-only policy attached
- Enabled KV v2 secrets engine at
Tip
If you are not familiar with entities and groups, read the Identity: Entities and Groups tutorial.
Use your preferred text editor, paste in the following to create a script to setup the Vault elements needed for demonstration.
lab-setup.sh
# Create new namespaces - they are peervault namespace create us-west-orgvault namespace create us-east-org#--------------------------# us-west-org namespace#--------------------------VAULT_NAMESPACE=us-west-org vault secrets enable -path="kv-customer-info" kv-v2VAULT_NAMESPACE=us-west-org vault kv put kv-customer-info/customer-001 name="Example LLC" contact_email="admin@example.com"# Create a policy to allow read access to kv-marketingVAULT_NAMESPACE=us-west-org vault policy write customer-info-read-only -<<EOFpath "kv-customer-info/data/*" { capabilities = ["read"]}EOFVAULT_NAMESPACE=us-west-org vault auth enable userpassVAULT_NAMESPACE=us-west-org vault write auth/userpass/users/tam-user password="my-long-password" policies=customer-info-read-only# Create an entityVAULT_NAMESPACE=us-west-org vault auth list -format=json | jq -r '.["userpass/"].accessor' > accessor.txtVAULT_NAMESPACE=us-west-org vault write -format=json identity/entity name="TAM" | jq -r ".data.id" > entity_id.txtVAULT_NAMESPACE=us-west-org vault write identity/entity-alias name="tam-user" canonical_id=$(cat entity_id.txt) mount_accessor=$(cat accessor.txt)#--------------------------# us-east-org namespace#--------------------------VAULT_NAMESPACE=us-east-org vault secrets enable -path="kv-marketing" kv-v2VAULT_NAMESPACE=us-east-org vault kv put kv-marketing/campaign start_date="March 1, 2023" end_date="March 31, 2023" prise="Certification voucher" quantity="100"# Create a policy to allow read access to kv-marketingVAULT_NAMESPACE=us-east-org vault policy write marketing-read-only -<<EOFpath "kv-marketing/data/campaign" { capabilities = ["read"]}EOF# Create a groupVAULT_NAMESPACE=us-east-org vault write -format=json identity/group name="campaign-admin" policies="marketing-read-only" member_entity_ids=$(cat entity_id.txt)
Change the file permission to make it executable.
$ chmod +x lab-setup.sh
Run the script.
$ ./lab-setup.sh
Enable secrets sharing
Vault 1.13.0 introduced a new API endpoint,
sys/config/group-policy-application
which sets the
group_policy_application_mode
parameter to allow Vault clients to manage
secrets across multiple namespaces.
Look up the current setting.
$ vault read sys/config/group-policy-applicationKey Value--- -----group_policy_application_mode within_namespace_hierarchy
The default setting is
within_namespace_hierarchy
.Set the parameter value to
any
.$ vault write sys/config/group-policy-application \ group_policy_application_mode="any"
Output:
Success! Data written to: sys/config/group-policy-application
Read it back to verify.
$ vault read sys/config/group-policy-applicationKey Value--- -----group_policy_application_mode any
Verification: Read secrets
Log in as "tam-user" and store the returned token in token.txt. Remember that auth methods are tied to the namespace they are configured. The "tam-user" is a user created in the userpass auth method in the us-west-org namespace.
$ VAULT_NAMESPACE=us-west-org vault login -field=token -method=userpass \ username=tam-user password="my-long-password" > token.txt
Tip
You can ignore the returned warning about the VAULT_TOKEN environment variable to take precedence.Check to see if you can read the
kv-marketing/data/campaign
in the us-east-org namespace using the generated token.$ VAULT_NAMESPACE=us-east-org VAULT_TOKEN=$(cat token.txt) \ vault kv get kv-marketing/campaign
Output:
======= Secret Path =======kv-marketing/data/campaign======= Metadata =======Key Value--- -----created_time 2023-02-17T04:46:03.22039Zcustom_metadata <nil>deletion_time n/adestroyed falseversion 1======= Data =======Key Value--- -----end_date March 31, 2023prise Certification voucherquantity 100start_date March 1, 2023
You can read the secrets in the
us-east-org
namespace.(Optional) The "tam-user" token has the
customer-info-read-only
policy attached. Therefore, you can read the secrets atkv-customer-info/
in the us-west-org namespace.$ VAULT_NAMESPACE=us-west-org VAULT_TOKEN=$(cat token.txt) \ vault kv get kv-customer-info/customer-001
Example output:
=========== Secret Path ===========kv-customer-info/data/customer-001======= Metadata =======Key Value--- -----created_time 2023-02-17T17:47:06.147303Zcustom_metadata <nil>deletion_time n/adestroyed falseversion 1======== Data ========Key Value--- -----contact_email admin@example.comname Example LLC
Revert back the parameter setting
Set the
group_policy_application_mode
parameter value back towithin_namespace_hierarchy
.$ vault write sys/config/group-policy-application \ group_policy_application_mode="within_namespace_hierarchy"
Try to read the secrets again.
$ VAULT_NAMESPACE=us-east-org VAULT_TOKEN=$(cat token.txt) \ vault kv get kv-marketing/campaign
Output:
Error making API request.Namespace: us-east-org/URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/kv-marketing/campaignCode: 403. Errors:* preflight capability check returned 403, please ensure client's policies grant access to path "kv-marketing/campaign/"
When the
group_policy_application_mode
parameter is set towithin_namespace_hierarchy
, you cannot access the secrets in the namespaces that has no hierarchical relationship.
Clean up
You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
$ pgrep -f vault | xargs kill
Unset the
VAULT_TOKEN
andVAULT_ADDR
environment variables.$ unset VAULT_TOKEN VAULT_ADDR
Remove the entity / token / accessor files.
$ rm accessor.txt entity_id.txt token.txt
Summary
This tutorial demonstrated the new API endpoint,
sys/config/group-policy-application
and its group_policy_application_mode
parameter introduced in Vault 1.13.0 to allow Vault clients to manage secrets
across multiple independent namespaces.
Although secrets engines, auth methods, policies, and tokens are tied to each namespaces, the entity group can pull entities and groups from other namespaces. If you are not familiar with entities and groups, read the Identity: Entities and Groups tutorial.