Ingesting Secrets with Vault and External Secrets Operator

In a previous lab, you deployed HashiCorp Vault. In this lab, you will deploy the External Secrets Operator to your Kubernetes cluster and configure it to pull secrets from HashiCorp Vault. It will then generate Kubernetes secrets with the captured information. Afterwards, you will deploy a simple application that consumes the secrets generated by the operator.

The External Secrets Operator is a Kubernetes operator that connects to a secret management system, pulls data from it, and injects the data into a Kubernetes Secrets object for use by an application

In this lab you will:

Enable a Key Value (kv-v2) Secrets Engine in Vault
Create an initial secret
Enable the Kubernetes Authentication Method
Deploy the External Secrets Operator with Helm and ArgoCD
Configure a SecretStore dedicated for your application
Create an ExternalSecret resource for your application
Deploy an application that uses the generated Secret

We will be interacting with Vault via the CLI tool. You can target your cluster by setting the following environment variable:

Note: You can see your Vault address via: kubectl get ingress --namespace vault.

export VAULT_ADDR=https://<YOUR_VAULT_ADDRESS>

Once set, grab your Vault Root token that was generated during the Vault lab and authenticate to Vault:

vault login <YOUR_ROOT_TOKEN>

Enable a kv-v2 Secrets Engine in Vault #

Enable a kv-v2 Secrets Engine at path vault-external-secrets. You will store the secrets for our application in that location.

vault secrets enable -path=vault-external-secrets kv-v2

You will see an output similar to below:

Success! Enabled the kv-v2 secrets engine at: vault-external-secrets/

Create a Secret #

Next, create a secret. You can create a secret with the following vault command:

vault kv put vault-external-secrets/app/secrets username='foo' password='bar' description='This is a secret stored in Vault'

The above command creates a secret at path: vault-external-secrets/app/secrets with the following key/value pairs:

username: foo
password: bar
description: This is a secret stored in Vault

Enable Kubernetes Authentication Method #

Next, enable the Kubernetes Authentication Method, create a role for the External Secrets Operator, and assign a policy to the operator. This will grant the operator the necessary permissions to read the previously created secret, enabling seamless authentication to Vault for fetching secrets.

Enable the Kubernetes Authentication Method:

vault auth enable kubernetes

Configure the Authentication Method to point to our Kubernetes cluster:

vault write auth/kubernetes/config \
kubernetes_host="https://kubernetes.default.svc.cluster.local:443"

Create a policy that will allow our application to only read a secret the specified path:

vault policy write external-secrets-policy - <<EOF
path "vault-external-secrets/data/app/secrets" {
  capabilities = ["read"]
}
EOF

Create a role called app that will map the Kubernetes Service Account (called my-eso-sa) to this role in Vault:

vault write auth/kubernetes/role/app \
bound_service_account_names=my-eso-sa \
bound_service_account_namespaces=vault-external-secrets \
policies=external-secrets-policy \
ttl=4h

Deploy the External Secrets Operator with Helm and ArgoCD #

You can now deploy the External Secrets Operator through ArgoCD.

Create a dedicated directory for this lab and switch into it:

cd ~

mkdir vault-external-secrets && cd vault-external-secrets

Copy the Application manifest and apply it to your cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.8.3
    chart: external-secrets
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Open the ArgoCD UI and wait for the application to report back Healthy.

You can also validate that the application is deployed via the following argo command:

argoPass=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

argocd login argocd.<YOUR_STUDENT_ID>.<RANDOM_ID>.workshops.acceleratorlabs.ca --username admin --password $argoPass

argocd app list

You will see an output similar to below:

NAME          CLUSTER                         NAMESPACE  PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                 PATH  TARGET
...
argocd/external-secrets  https://kubernetes.default.svc  external-secrets      default  Synced  Healthy  Auto-Prune  <none>      https://charts.external-secrets.io         0.8.3

Configure a SecretStore for your application #

In this section, you will deploy a SecretStore, a resource that describes a secure external location for storing secrets.

First, create a namespace called vault-external-secrets with the label lab=vault-external-secrets.

Next, you will create a Service Account that will be responsible for authenticating to Vault on behalf of ESO. Copy the manifest below and apply it to your cluster:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-eso-sa
  namespace: vault-external-secrets

Create the SecretStore manifest with the following contents and it apply it to your cluster:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: vault-external-secrets
spec:
  provider:
    vault:
      server: "http://vault.vault.svc.cluster.local:8200"
      path: "vault-external-secrets"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app"
          serviceAccountRef:
            name: "my-eso-sa"

Apply the manifest and validate that the SecretStore is ready. You should see an output similar to the following:

NAME                         AGE   STATUS   CAPABILITIES   READY
random-facts-vault-backend   20s   Valid    ReadWrite      True

Create an ExternalSecret resource for your application #

You can now instruct External Secrets Operator to pull a secret and to create a Kubernetes Secret with the data.

Create the following ExternalSecret manifest with the following contents and it apply it to your cluster:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secret
  namespace: vault-external-secrets
spec:
  refreshInterval: "15s" # How often ESO should check Vault to see if the secret has changed
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: generated-app-secret # This is the name of the generated k8s secret
  data:
  - secretKey: username
    remoteRef:
      key: app/secrets
      property: username
  - secretKey: password
    remoteRef:
      key: app/secrets
      property: password
  - secretKey: description
    remoteRef:
      key: app/secrets
      property: description

Apply the manifest and validate that the SecretStore is ready. You should see an output similar to the following:

NAME                       STORE                        REFRESH INTERVAL   STATUS         READY
random-facts-app-secrets   random-facts-vault-backend   15s                SecretSynced   True

Next, validate that ESO created a Kubernetes secret with the following command:

kubectl get secret --namespace vault-external-secrets

You should see an output similar to the following:

NAME                                TYPE     DATA   AGE
random-facts-generated-app-secret   Opaque   3      23s

You will notice the data count is 3 which matches how many secrets we stored in Vault.

Deploy an Application that Consumes this Secret #

Finally, deploy a simple NGINX application that prints the secret onto a webpage.

Create a Pod with the following contents and it apply it to your cluster:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-app
  namespace: vault-external-secrets
  labels:
    lab: vault-external-secrets
spec:
  initContainers:
  - name: create-html
    image: busybox:1.28
    command:
    - 'sh'
    - '-c'
    - 'echo "<html><head><title>Demo app</title></head><body>$DESCRIPTION<br><b>Username:</b> $USERNAME<br><b>Password:</b> $PASSWORD</body></html>" > /usr/share/nginx/html/index.html'
    volumeMounts:
    - name: html
      mountPath: "/usr/share/nginx/html"
    env:
      - name: DESCRIPTION
        valueFrom:
          secretKeyRef:
            name: generated-app-secret
            key: description
      - name: USERNAME
        valueFrom:
          secretKeyRef:
            name: generated-app-secret
            key: username
      - name: PASSWORD
        valueFrom:
          secretKeyRef:
            name: generated-app-secret
            key: password
  containers:
  - name: nginx
    image: nginx:1.24.0-bullseye
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 80
      name: http
    volumeMounts:
    - name: html
      mountPath: "/usr/share/nginx/html"
  volumes:
  - name: html
    emptyDir:
      sizeLimit: 10Mi

Validate that the Pod is running and then create a Service with the following manifest:

apiVersion: v1
kind: Service
metadata:
  name: nginx-app-service
  namespace: vault-external-secrets
  labels:
    lab: vault-external-secrets
spec:
  selector:
    lab: vault-external-secrets
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 80
  type: ClusterIP

Create an Ingress object with the following manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-app-ingress
  namespace: vault-external-secrets
spec:
  ingressClassName: nginx
  rules:
  - host: vault-app.<YOUR_STUDENT_ID>.<RANDOM_ID>.workshops.acceleratorlabs.ca
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-app-service
            port:
              number: 8080

You should now be able to see your application at: http://vault-app.<YOUR_STUDENT_ID>.<RANDOM_ID>.workshops.acceleratorlabs.ca and see your secrets being displayed.