Post Syndicated from James Beswick original https://aws.amazon.com/blogs/compute/using-the-aws-parameter-and-secrets-lambda-extension-to-cache-parameters-and-secrets/
This post is written by Pal Patel, Solutions Architect, and Saud ul Khalid, Sr. Cloud Support Engineer.
Serverless applications often rely on AWS Systems Manager Parameter Store or AWS Secrets Manager to store configuration data, encrypted passwords, or connection details for a database or API service.
Previously, you had to make runtime API calls to AWS Parameter Store or AWS Secrets Manager every time you wanted to retrieve a parameter or a secret inside the execution environment of an AWS Lambda function. This involved configuring and initializing the AWS SDK client and managing when to store values in memory to optimize the function duration, and avoid unnecessary latency and cost.
The new AWS Parameters and Secrets Lambda extension provides a managed parameters and secrets cache for Lambda functions. The extension is distributed as a Lambda layer that provides an in-memory cache for parameters and secrets. It allows functions to persist values through the Lambda execution lifecycle, and provides a configurable time-to-live (TTL) setting.
When you request a parameter or secret in your Lambda function code, the extension retrieves the data from the local in-memory cache, if it is available. If the data is not in the cache or it is stale, the extension fetches the requested parameter or secret from the respective service. This helps to reduce external API calls, which can improve application performance and reduce cost. This blog post shows how to use the extension.
Overview
The following diagram provides a high-level view of the components involved.
The extension can be added to new or existing Lambda. It works by exposing a local HTTP endpoint to the Lambda environment, which provides the in-memory cache for parameters and secrets. When retrieving a parameter or secret, the extension first queries the cache for a relevant entry. If an entry exists, the query checks how much time has elapsed since the entry was first put into the cache, and returns the entry if the elapsed time is less than the configured cache TTL. If the entry is stale, it is invalidated, and fresh data is retrieved from either Parameter Store or Secrets Manager.
The extension uses the same Lambda IAM execution role permissions to access Parameter Store and Secrets Manager, so you must ensure that the IAM policy is configured with the appropriate access. Permissions may also be required for AWS Key Management Service (AWS KMS) if you are using this service. You can find an example policy in the example’s AWS SAM template.
Example walkthrough
Consider a basic serverless application with a Lambda function connecting to an Amazon Relational Database Service (Amazon RDS) database. The application loads a configuration stored in Parameter Store and connects to the database. The database connection string (including user name and password) is stored in Secrets Manager.
This example walkthrough is composed of:
- A Lambda function.
- An Amazon Virtual Private Cloud (VPC).
- Multi-AZ Amazon RDS Instance running MySQL.
- AWS Secrets Manager database secret that holds database connection.
- AWS Systems Manager Parameter Store parameter that holds the application configuration.
- An AWS Identity and Access Management (IAM) role that the Lambda function uses.
Lambda function
This Python code shows how to retrieve the secrets and parameters using the extension.
import pymysql
import urllib3
import os
import json
### Load in Lambda environment variables
port = os.environ['PARAMETERS_SECRETS_EXTENSION_HTTP_PORT']
aws_session_token = os.environ['AWS_SESSION_TOKEN']
env = os.environ['ENV']
app_config_path = os.environ['APP_CONFIG_PATH']
creds_path = os.environ['CREDS_PATH']
full_config_path = '/' + env + '/' + app_config_path
### Define function to retrieve values from extension local HTTP server cachce
def retrieve_extension_value(url):
http = urllib3.PoolManager()
url = ('http://localhost:' + port + url)
headers = { "X-Aws-Parameters-Secrets-Token": os.environ.get('AWS_SESSION_TOKEN') }
response = http.request("GET", url, headers=headers)
response = json.loads(response.data)
return response
def lambda_handler(event, context):
### Load Parameter Store values from extension
print("Loading AWS Systems Manager Parameter Store values from " + full_config_path)
parameter_url = ('/systemsmanager/parameters/get/?name=' + full_config_path)
config_values = retrieve_extension_value(parameter_url)['Parameter']['Value']
print("Found config values: " + json.dumps(config_values))
### Load Secrets Manager values from extension
print("Loading AWS Secrets Manager values from " + creds_path)
secrets_url = ('/secretsmanager/get?secretId=' + creds_path)
secret_string = json.loads(retrieve_extension_value(secrets_url)['SecretString'])
#print("Found secret values: " + json.dumps(secret_string))
rds_host = secret_string['host']
rds_db_name = secret_string['dbname']
rds_username = secret_string['username']
rds_password = secret_string['password']
### Connect to RDS MySQL database
try:
conn = pymysql.connect(host=rds_host, user=rds_username, passwd=rds_password, db=rds_db_name, connect_timeout=5)
except:
raise Exception("An error occurred when connecting to the database!")
return "DemoApp sucessfully loaded config " + config_values + " and connected to RDS database " + rds_db_name + "!"
In the global scope the environment variable PARAMETERS_SECRETS_EXTENSION_HTTP_PORT is retrieved, which defines the port the extension HTTP server is running on. This defaults to 2773.
The retrieve_extension_value function calls the extension’s local HTTP server, passing in the X-Aws-Parameters-Secrets-Token as a header. This is a required header that uses the AWS_SESSION_TOKEN value, which is present in the Lambda execution environment by default.
The Lambda handler code uses the extension cache on every Lambda invoke to obtain configuration data from Parameter Store and secret data from Secrets Manager. This data is used to make a connection to the RDS MySQL database.
Prerequisites
- Git installed
- AWS SAM CLI version 1.58.0 or greater.
Deploying the resources
- Clone the repository and navigate to the solution directory:
git clone https://github.com/aws-samples/parameters-secrets-lambda-extension- sample.git - Build and deploy the application using following command:
sam build sam deploy --guided
This template takes the following parameters:
- pVpcCIDR — IP range (CIDR notation) for the VPC. The default is 172.31.0.0/16.
- pPublicSubnetCIDR — IP range (CIDR notation) for the public subnet. The default is 172.31.3.0/24.
- pPrivateSubnetACIDR — IP range (CIDR notation) for the private subnet A. The default is 172.31.2.0/24.
- pPrivateSubnetBCIDR — IP range (CIDR notation) for the private subnet B, which defaults to 172.31.1.0/24
- pDatabaseName — Database name for DEV environment, defaults to devDB
- pDatabaseUsername — Database user name for DEV environment, defaults to myadmin
- pDBEngineVersion — The version number of the SQL database engine to use (the default is 5.7).
Adding the Parameter Store and Secrets Manager Lambda extension
To add the extension:
- Navigate to the Lambda console, and open the Lambda function you created.
- In the Function Overview pane. select Layers, and then select Add a layer.
- In the Choose a layer pane, keep the default selection of AWS layers and in the dropdown choose AWS Parameters and Secrets Lambda Extension
- Select the latest version available and choose Add.
The extension supports several configurable options that can be set up as Lambda environment variables.
This example explicitly sets an extension port and TTL value:
Testing the example application
To test:
- Navigate to the function created in the Lambda console and select the Test tab.
- Give the test event a name, keep the default values and then choose Create.
- Choose Test. The function runs successfully:
To evaluate the performance benefits of the Lambda extension cache, three tests were run using the open source tool Artillery to load test the Lambda function. This can use the Lambda URL to invoke the function. The Artillery configuration snippet shows the duration and requests per second for the test:
config:
target: "https://lambda.us-east-1.amazonaws.com"
phases:
-
duration: 60
arrivalRate: 10
rampTo: 40
scenarios:
-
flow:
-
post:
url: "https://abcdefghijjklmnopqrst.lambda-url.us-east-1.on.aws/"
- Test 1: The extension cache is disabled by setting the TTL environment variable to 0. This results in 1650 GetParameter API calls to Parameter Store over 60 seconds.
- Test 2: The extension cache is enabled with a TTL of 1 second. This results in 106 GetParameter API calls over 60 seconds.
- Test 3: The extension is enabled with a TTL value of 300 seconds. This results in only 18 GetParameter API calls over 60 seconds.
In test 3, the TTL value is longer than the test duration. The 18 GetParameter calls correspond to the number of Lambda execution environments created by Lambda to run requests in parallel. Each execution environment has its own in-memory cache and so each one needs to make the GetParameter API call.
In this test, using the extension has reduced API calls by ~98%. Reduced API calls results in reduced function execution time, and therefore reduced cost.
Cleanup
After you test this example, delete the resources created by the template, using following commands from the same project directory to avoid continuing charges to your account.
sam delete
Conclusion
Caching data retrieved from external services is an effective way to improve the performance of your Lambda function and reduce costs. Implementing a caching layer has been made simpler with this AWS-managed Lambda extension.
For more information on the Parameter Store, Secrets Manager, and Lambda extensions, refer to:
- Using Parameter Store parameters in AWS Lambda functions
- Use AWS Secrets Manager secrets in AWS Lambda functions
- Introducing AWS Lambda Extensions
- Caching data and configuration settings with AWS Lambda extensions
For more serverless learning resources, visit Serverless Land.
















You can also automate the preceding COPY commands using tasks, which can be scheduled to run at a set frequency for automatic copy of CDC data from Snowflake to Amazon S3.
Now the tasks will run every 5 minutes and look for new data in the stream tables to offload to Amazon S3.As soon as data is migrated from Snowflake to Amazon S3, Redshift Auto Loader automatically infers the schema and instantly creates corresponding tables in Amazon Redshift. Then, by default, it starts loading data from Amazon S3 to Amazon Redshift every 5 minutes. You can also 
















































































DoYeun Kim is the Head of Data Engineering at SOCAR. He is a passionate software engineering professional with 19+ years experience. He leads a team of 10+ engineers who are responsible for the data platform, data warehouse and MLOps engineering, as well as building in-house data products.
SangSu Park is a Lead Data Architect in SOCAR’s cloud DB team. His passion is to keep learning, embrace challenges, and strive for mutual growth through communication. He loves to travel in search of new cities and places.
YoungMin Park is a Lead Architect in SOCAR’s cloud infrastructure team. His philosophy in life is-whatever it may be-to challenge, fail, learn, and share such experiences to build a better tomorrow for the world. He enjoys building expertise in various fields and basketball.
Younggu Yun is a Senior Data Lab Architect at AWS. He works with customers around the APAC region to help them achieve business goals and solve technical problems by providing prescriptive architectural guidance, sharing best practices, and building innovative solutions together. In his free time, his son and he are obsessed with Lego blocks to build creative models.
Vicky Falconer leads the AWS Data Lab program across APAC, offering accelerated joint engineering engagements between teams of customer builders and AWS technical resources to create tangible deliverables that accelerate data analytics modernization and machine learning initiatives.





