Kubernetes Secrets Replication with ESO

Replicating secrets across namespaces is a common challenge in Kubernetes environments, particularly when multiple applications require shared access to sensitive data like database credentials or API keys. While there are tools like Kyverno that can handle this, they often fall short in terms of synchronization and integration with external secret stores.

In this guide, we’ll demonstrate how to use the External Secrets Operator (ESO) to achieve seamless replication of secrets across namespaces. By leveraging ClusterSecretStore and ClusterExternalSecret, you can centralize secret management, maintain synchronization with external sources, and avoid manual intervention.

Why Use ESO for Replication in Kubernetes?

When dealing with secrets in Kubernetes, replication across namespaces can be tricky:

• Manual Processes: Tools like Kyverno require the source secret to exist before replication begins.
• No Real-Time Sync: Changes to the source secret aren’t automatically reflected in the replicated copies.
• Complexity in Scaling: Managing replication policies for multiple namespaces can quickly become unmanageable.

External Secrets Operator solves these issues by directly syncing secrets from external providers and ensuring they are always up-to-date in target namespaces. It simplifies secrets replication while adhering to Kubernetes best practices.

Step-by-Step Guide

1. Prepare Your Environment

To get started, ensure the following:

• Kubernetes cluster with External Secrets Operator installed.
• Access to an external secrets provider (e.g., AWS Secrets Manager, GCP Secret Manager, etc.).

For this demonstration, we’ll replicate a secret from a source namespace across all other namespaces in the cluster.

2. Create a Source Namespace and Secret

Begin by creating a namespace (source-ns) and a source secret (source-secret):

apiVersion: v1
kind: Namespace
metadata:
  name: source-ns
---
apiVersion: v1
kind: Secret
metadata:
  name: source-secret
  namespace: source-ns
data:
  foo: YmFy  ## bar

3. Configure Service Account and Permissions

The External Secrets Operator needs permissions to read the source secret. Create a service account and the necessary RBAC roles:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-store
  namespace: source-ns
--- 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-source-secret
  namespace: source-ns
rules:
- apiGroups: [""]
  resourceNames:
  - source-secret
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - authorization.k8s.io
  resources:
  - selfsubjectrulesreviews
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-source-secret
  namespace: source-ns
subjects:
- kind: ServiceAccount
  name: k8s-store
  namespace: source-ns
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: read-source-secret
  apiGroup: rbac.authorization.k8s.io

4. Create a ClusterSecretStore

The ClusterSecretStore connects External Secrets Operator to the source namespace:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: k8s-store
spec:
  provider:
    kubernetes:
      # with this, the store is able to pull only from `default` namespace
      remoteNamespace: source-ns
      server:
        url: "https://kubernetes.default" # Change accordingly to your setup
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
          namespace: source-ns
      auth:
        serviceAccount:
          name: "k8s-store"
          namespace: source-ns

5. Create a ClusterExternalSecret

The ClusterExternalSecret specifies the secret to replicate and its target namespaces:

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: "copy-secret"
spec:
  # The name to be used on the ExternalSecrets
  externalSecretName: "copy-secret-es"
  namespaceSelectors:
  - matchLabels: {} #deploy to every namespace
  refreshTime: "1h" #Cluster External Secret refresh Time
  externalSecretSpec:
    secretStoreRef:
      name: k8s-store
      kind: ClusterSecretStore
    refreshInterval: "10m" #External Secret Refresh Time
    target:
      name: copy-secret
      creationPolicy: 'Owner'
    dataFrom:
      - extract: 
          key: source-secret

6. Apply the Configurations and Verify

Apply all the manifests and check the status of the ExternalSecrets:

kubectl apply -f <your-manifest-files>
kubectl get es -A

default                 copy-secret-es   k8s-store   10m0s              SecretSynced   True
source-ns            copy-secret-es   k8s-store   10m0s              SecretSynced   True

And with it, every new namespace creation automatically triggers a new copy of that secret, effectively replicating the sensitive information across namespaces.

7. Automatic Secrets Replication for New Namespaces

With this setup, any new namespace created in the cluster will automatically receive the replicated secret, ensuring consistent access across your environment.

Conclusion

Managing secrets replication across namespaces in Kubernetes doesn’t have to be a challenge. By leveraging the External Secrets Operator, you can automate this process, ensure synchronization with external secret providers, and maintain a scalable and secure workflow. This approach not only simplifies secrets management but also ensures you’re following Kubernetes best practices for handling sensitive data. For more advanced use cases or enterprise-level features, explore our solutions at www.externalsecrets.com.