External Secrets Operator

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

Why you need this?

The goal of External Secrets Operator is to synchronize secrets from external APIs into Kubernetes. ESO is a collection of custom API resources – ExternalSecret, SecretStore and ClusterSecretStore that provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you.

Suppose you have some secrets in GCP Secrets Manager, you need to have the same secrets accessible in your Kubernetes cluster. you also have to change the secret in the cluster when the GCP secret gets Updated. Here we can use the External Secrets Operator (ESO). You can  integrate external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault , We are going to talk about the GCP Integration.

GCP Integration

Your Google Kubernetes Engine (GKE) applications can consume GCP services like Secrets Manager without using static, long-lived authentication tokens. This is our recommended approach of handling credentials in GCP. ESO offers two options for integrating with GKE workload identity: pod-based workload identity and using service accounts directly. 

We are going to use service account directly.

1. Create a Service account with roles/secretmanager.secretAccessor role
2. Create a Private Key for your service account and save to local as Json format.

we will assume that you installed ESO using helm and that you named the chart installation external-secrets and the namespace where it lives es like:

$ Kubectl create namespace es
$ helm repo add external-secrets https://charts.external-secrets.io
$ helm install external-secrets external-secrets/external-secrets --namespace es

GCP Service Account authentication

You can use GCP Service Account to authenticate with GCP. These are static, long-lived credentials. A GCP Service Account is a JSON file that needs to be stored in a Kind=Secret. ESO will use that Secret to authenticate with GCP. See here how you manage GCP Service Accounts.

Create gcpsm-secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: gcpsm-secret
  labels:
    type: gcpsm
type: Opaque
stringData:
  secret-access-credentials: |-
    {
      "type": "service_account",
      "project_id": "external-secrets-operator",
      "private_key_id": "",
      "private_key": "-----BEGIN PRIVATE KEY-----\nA key\n-----END PRIVATE KEY-----\n",
      "client_email": "test-service-account@external-secrets-operator.iam.gserviceaccount.com",
      "client_id": "client ID",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test-service-account%40external-secrets-operator.iam.gserviceaccount.com"
    }

replace the secret-access-credentials with the key that we downloaded from GCP earlier. Then create the secret :

$ kubectl apply -f gcpsm-secret.yml -n es

Create a Secret in GCP secrets manager

create a secret in gcpsm named as dev-secret-test. Please add some test data as secrets. We will be pulling this secret from gcp to Kubernates cluster.

Create secret store

Be sure the gcpsm provider is listed in the Kind=SecretStore

SecretStore.yml

apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: example
spec:
  provider:
      gcpsm:                                  # gcpsm provider
        auth:
          secretRef:
            secretAccessKeySecretRef:
              name: gcpsm-secret              # secret name containing SA key
              key: secret-access-credentials  # key name containing SA key
        projectID: myproject                  # name of Google Cloud project

NOTE: In case of a ClusterSecretStore, Be sure to provide namespace for SecretAccessKeyRef with the namespace of the secret that we just created.

Apply the secrets store :

$ kubectl apply -f SecretStore.yml -n es

Creating external secret

To create a kubernetes secret from the GCP Secret Manager secret a Kind=ExternalSecret is needed.

ExternalSecret.yml

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h           # rate SecretManager pulls GCPSM
  secretStoreRef:
    kind: SecretStore
    name: example               # name of the SecretStore (or kind specified)
  target:
    name: secret-to-be-created  # name of the k8s Secret to be created
    creationPolicy: Owner
  data:
  - secretKey: dev-secret-test  # name of the GCPSM secret key
    remoteRef:
      key: dev-secret-test

Apply the external secret:

$ kubectl apply -f ExternalSecret.yml -n es

The operator will fetch the GCP Secret Manager secret and inject it as a Kind=Secret

kubectl get secret secret-to-be-created -n <namespace> | -o jsonpath='{.data.dev-secret-test}' | base64 -d

Now