Managing Secrets in Kubernetes with Vault by HashiCorp

If you’re serious about security in Kubernetes, you need a secret management tool that provides a single source of secrets, credentials, attaching security policies, etc. In other words, you need Hashicorp Vault.

This post outlines a process to use vault within Kubernetes to make the secret management more secure and robust with CoreOS vault operator.

Prerequisites:

• A Kubernetes cluster

Tools Used:

• CoreOS vault operator for setting up a vault cluster backed by etcd.

Deploying CoreOS vault operator:

• Clone the repository https://github.com/coreos/vault-operator
• Follow the README.mdinstructions for setting up rbac, etcd operator followed by a vault operator and a vault cluster.

Note: If you don’t want to use the coreos vault operator and simply want to create a vault cluster backed by the consul, you only need two commands:

$ helm install --name consul stable/consul

$ helm install incubator/vault --set vault.dev=false --set vault.config.storage.consul.address="consul:8500",vault.config.storage.consul.path="vault"

Getting Started with Vault:

Prerequisites:

• Vault Commands (CLI) installed
• A vault cluster

Initialize a vault cluster:

Before doing any operations on the vault cluster, the vault cluster needs to be initialized. For doing that:

• Configure port-forwarding between the local machine and the sealed vault node:

$ kubectl -n default get vault vault-cluster-coreos -o jsonpath=’{.status.vaultStatus.sealed[0]}’ | xargs -0 -I {} kubectl -n default port-forward {} 8200

• Open another terminal and set the below environment variables:

$ export VAULT_ADDR=’https://localhost:8200'
$ export VAULT_SKIP_VERIFY=”true”

• Verify if the vault server is accessible via CLI:

$ vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            n/a
HA Enabled         true

• Now, initialize the vault cluster by running:

$ vault operator init -key-shares=1 -key-threshold=1
Unseal Key 1: SQYCzJGn16fTe92h6QaZqCaWcmrUwSeZITCRsSZ1Bo4=
Initial Root Token: a1ce0fe8–56c0–7a23–6756-f41a7d15eeb4
Vault initialized with 1 key shares and a key threshold of 1. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 1 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 1 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See “vault operator rekey” for more information.

Keep a note of Unseal key and Root token values as this will be required in the next step.

• Try writing a secret by running:

$ vault write secret/foo value=bar
Error writing data to secret/foo: Error making API request.
URL: PUT https://localhost:8200/v1/secret/foo
Code: 403. Errors:
* permission denied

Looks like we missed something. Well, yes, we did. We first need to unseal the vault cluster (with the unseal key which was printed while initializing the vault cluster) and then authenticate to the vault cluster using a root token. For doing that, run:

$ vault operator unseal
Unseal Key (will be hidden):
Key             Value
---             -----
Seal Type       shamir
Initialized     false
Sealed          false
Total Shares    1
Threshold       1
Version         0.9.1
Cluster Name    vault-cluster-6a21908f
Cluster ID      713de97e-d905-495a-7138-f53f71d08d26
HA Enabled      true
HA Cluster      https://vault-cluster-coreos.default.svc:8201
HA Mode         active
$ vault login <your-generated-root-token>
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run “vault login”
again. Future Vault requests will automatically use this token.
Key                  Value
---                  -----
token                a1ce0fe8-56c0-7a23-6756-f41a7d15eeb4
token_accessor       1cae5428-0446-373b-2a0e-257f9fdc8b7d
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Alternatively, you can also set VAULT_TOKEN environment variable.

• Now, try writing/reading the secret again:

$ vault write secret/foo value=bar
Success! Data written to: secret/foo
$ vault read secret/foo
Key                 Value
---                 -----
refresh_interval    768h
value               bar

This SHOULD work.

Using Vault’s Kubernetes Auth Backend:

So far, we’ve been successful in authenticating with vault, creating/reading secrets. This guide is focused on using vault’s Kubernetes auth backend for authenticating with Kubernetes service accounts and storing secrets.

Prerequisites:

• A running Kubernetes cluster
• A running vault cluster created in the previous guide

Kubernetes auth backend setup:

• Create a service account vault-coreos-test by running:

$ kubectl -n default create serviceaccount vault-coreos-test

• Create a clusterrolebinding for the vault-coreos-test serviceaccount by running:

$ kubectl -n default create -f example/k8s_auth/vault-tokenreview-binding.yaml

Before running this, make sure that you have a correct service account name in the file example/k8s_auth/vault-tokenreview-binding.yaml since the repository uses a service account named vault-tokenreview whereas here we are using vault-coreos-test.

• Get the JWT token, Kubernetes API URL and certificate authority cert from the serviceaccount by running:

# TR_ACCOUNT_TOKEN is the final JWT token
$ SECRET_NAME=$(kubectl -n default get serviceaccount vault-coreos-test -o jsonpath=’{.secrets[0].name}’)
$ TR_ACCOUNT_TOKEN=$(kubectl -n default get secret ${SECRET_NAME} -o jsonpath=’{.data.token}’ | base64 — decode)
$ kubectl cluster-info # Note the API URL
# Getting ca.crt for the service account
$ export VAULT_SA_NAME=$(kubectl get sa vault-coreos-test -o jsonpath=”{.secrets[*][‘name’]}”)
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath=”{.data[‘ca\.crt’]}” | base64 — decode; echo)

• Enable the Kubernetes auth backend in the vault and authenticate by running:

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
$ vault write auth/kubernetes/config kubernetes_host=”https://MASTER_IP:6443" kubernetes_ca_cert=”$SA_CA_CRT” token_reviewer_jwt=$TR_ACCOUNT_TOKEN
Success! Data written to: auth/kubernetes/config

• Next, we’ll configure role and policy. The Kubernetes backend authorizes an entity by granting it a role mapped to a serviceaccount. A role is configured with policies which control the entity’s access to paths and operations in Vault.
• Create a new policy demo-policy using an example policy file policy.hcl

$ vault write sys/policy/demo-policy policy=@example/k8s_auth/policy.hcl
Success! Data written to: sys/policy/demo-policy

If you take a closer look at what the policy contains:

$ cat example/k8s_auth/policy.hcl
path “secret/demo/*” {
 capabilities = [“create”, “read”, “update”, “delete”, “list”]
}

It simply grants the privileges to create secrets in secret/demo path.

• Next, create a role for binding the policy to a service account.

$ vault write auth/kubernetes/role/demo-role \
 bound_service_account_names=vault-coreos-test \
 bound_service_account_namespaces=default \
 policies=demo-policy \
 ttl=1h

• Now, authenticate with the role we just created. For that, get the service account token by running:

$ DEFAULT_ACCOUNT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath=”{.data[‘token’]}” | base64 — decode; echo )

Login to the vault with the token by running:

$ vault write auth/kubernetes/login role=demo-role jwt=${DEFAULT_ACCOUNT_TOKEN}
Key                                       Value
---                                       -----
token                                     e19b9562-fabc-462f-a240-08a41ef6f716
token_accessor                            872ab113-141c-9249-a9f1-1d727f84abe7
token_duration                            1h
token_renewable                           true
token_policies                            []
identity_policies                         []
policies                                  ["default" "demo-policy"]
token_meta_role                           demo-role
token_meta_service_account_name           vault-coreos-test
token_meta_service_account_namespace      default
token_meta_service_account_secret_name    vault-coreos-test-token-fztrq
token_meta_service_account_uid            860a020e-1f08-11e9-82b4-002590ad511d

Set the VAULT_TOKEN environment variable to the token value shown in the above output.

• Create a secret at secret/demo path:

$ vault write secret/demo/foo value=bar
Success! Data written to: secret/demo/foo

Next, try to create a secret at a different path:

$ vault write secret/foo value=bar
Error writing data to secret/foo: Error making API request.
URL: PUT https://localhost:8200/v1/secret/foo
Code: 403. Errors:
* permission denied

Summary:

So, far we’ve seen how to create roles, policies, attach them to a serviceaccount, authentication. However, it is common if you need:

• Automatic authentication
• Secure delivery/storage of tokens
• Lifecycle management of token (renewal and re-authentication)

This is where vault agent comes into the scene. For using vault agent for automatic authentication and using the secrets in deployments, see — https://learn.hashicorp.com/vault/identity-access-management/vault-agent-k8s

Questions?

Please feel free to join uns on Kubernauts’ Slack and ask any questions.