AWS Parameter Store is a feature part of Systems Manager that allows users to manage application parameters and configuration within AWS. While AWS Parameter Store is convenient for managing application parameters, it poses challenges when used within Kubernetes clusters, it does still have an inherent problem when using it within a Kubernetes Cluster. It is also one of the biggest Secret Store bases for external-secrets based on the open source documentation website.
A common way that people configure their external-secrets setup is by having a collection of Parameters available as a Kubernetes Secret, leading to manifests like this one below:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: example
spec:
secretStoreRef:
kind: SecretStore
name: parameter-store
target:
name: secret-name-in-k8s
data:
- secretKey: parameter1
remoteRef:
key: path/to/parameter1
- secretKey: parameter2
remoteRef:
key: path/to/parameter2
And while this works 👆 it does cause a problem when it comes to updating the parameter.
Here, the underlying problem is the fact that, as opposed to e.g. database credentials which are expected to change only on a rotation period, parameters are also part of the application configuration, and may change on a different lifecycle just because that’s what developers need to do.
So, commonly, we configure a faster refreshInterval to allow this information to be propagated instantly:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: extract-data
spec:
refreshInterval: 1m # Refresh as soon as possible
secretStoreRef:
kind: SecretStore
name: parameter-store
target:
name: secret-name-in-k8s
data:
- secretKey: parameter1
remoteRef:
key: path/to/parameter1
- secretKey: parameter2
remoteRef:
key: path/to/parameter2
The problem here lies on a limitation of ParameterStore - it only allows for up to 40 requests per second on GetParameter calls (https://docs.aws.amazon.com/general/latest/gr/ssm.html#limits_ssm).
In this example with one parameter, things are going to be alright - but it is quite common that we need to expose hundreds of those within our Kubernetes infrastructure. Not only that, we frequently need a different set of parameters for different applications, meaning it is quite common to suffer from the rate limit of 40 transactions per second of the default Parameter Store.
Of course, you can enable High throughput (https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-throughput.html), bumping that limit to up to 10,000 transactions per second, however you need to pay for that throughput availability.
So, 3 clusters with 10 ExternalSecrets Manifests configured to get 100 Parameters each refreshing at 1 minute (an average of 50 transactions per second - still way lower than the max throughput amount of 10,000 transactions per second).
This means it will cost you $650 per month on API calls alone! Even if roughly only one or two parameters really changed, you need to pay the bill for all the parameters that composes the Secret.
However, it is possible to not need to upgrade ParameterStore throughput at all!
Implementing a different OSS Configuration
Even though it is not obvious, the manifest shared above has a similar version using a different approach, dataFrom.find:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: extract-data
spec:
refreshInterval: 1m # Refresh as soon as possible
secretStoreRef:
kind: SecretStore
name: parameter-store
target:
name: secret-name-in-k8s
dataFrom:
find:
name:
path: /path/to
regexp: .*
Will fetch all parameters with the cost of 1 call to AWS Parameter Store. This is because implementation-wise, dataFrom.find implements GetParametersByPath, which means one single call can get many parameters. This can already make your costs go down as it means the same setup (3 Clusters, 10 ExternalSecrets, 100 Parameters) will now cost effectively zero as we safely back on track to the standard limits: going from 3400 requests per second to roughly 0.5 requests per second.
While this approach is good enough for most organizations consuming a lot of parameters, it does have the downside of only supporting a maximum of 100 requests per second total, even in high throughput. Plus, it does not scale if your cluster has more ExternalSecrets objects, as they will now count towards this individual limit. This means a setup with 3 clusters, 1000 ExternalSecrets each fetching 1 parameter - even though it is the same amount of information as the previous example, will get you rate limited anyways.
One Step Further with Just-In-Time Rotation
While we can already reduce our costs and stay in quota for most of the cases, we still need to wait for the refreshInterval to get the new information as soon as it is added. What is even worse, if our system is making 400 calls per day to ParameterStore, only a fraction of it is really important as they are the ones that will fetch the changes.
With External Secrets Inc. Just-In-Time rotation, you can leverage AWS Event Bridge and a SQS Topic to automatically trigger ExternalSecrets rotation for you:
With this setup, any changes caused to any watched parameter will automatically propagate to all ExternalSecrets objects that refer to such parameters. This means instantaneous updates across namespaces, clusters, basically everywhere where such parameters have changed!.
Sign up now to start saving costs and achieving real-time updates with External Secrets Inc.! https://app.externalsecrets.com/signup
