I was chatting about AWS Lambda layers with a colleague and referred to an older blog post titled How to Create AWS Lambda Layers for Python. I realize that I was doing things the hard way and that a few tweaks were in order.
AWS Lambda layers are constructs that provide libraries and modules to code being executed by Lambda during runtime. For Python, this means modules stored in
site-packages. Layers are handy for deduplicating effort and code across a common set of Lambda functions, which would alternatively have dependencies bundled directly into the source code. For a few of my projects, many Lambda functions depend on identical or nearly identical module sets. Thus, a layer provides a single point of dependency control across a multitude of functions.
The Easy Way
venv module is packaged directly with modern version of Python 3. I use a virtual environment with every project. Once I reach a state of development for which publishing a Lambda function makes sense, I wipe the virtual environment. This avoids any chance of contaminating my requirements list from previous experiments during development.
From there, I create a new virtual environment and activate it.
py -m venv .venv .\.venv\Scripts\activate
Make sure to update
py -m pip install --upgrade pip
pip list to validate a clean working environment is presented back.
pip list Package Version ---------- ------- pip 21.2.4 setuptools 49.2.1
Install your known dependencies and run the code to validate a successful execution.
pip to the source code directory to couple them tightly. I do not like putting requirements in the repo’s root directly because Explicit is better than Implicit. Note that for Lambda deployments, you can skip installing
boto3 as this module is always available to executed code. You will need it locally, however. 🙂
pip freeze > src\requirements.txt
I use this little script to create the zip archive. It copies over the site-package modules to the root of the virtual environment, renames the folder
python to comply with AWS Lambda layer requirements, generates a zip from the results, and then purges the
Copy-Item -Path .\.venv\Lib\site-packages\ .\.venv -Recurse Rename-Item -Path .\.venv\site-packages python Compress-Archive -Path .\.venv\python\ -DestinationPath .\.venv\python_layer.zip Remove-Item -Path .\.venv\python\ -Recurse -Force
From there, switch into the
.venv directory and publish the layer or update an already published layer.
aws lambda publish-layer-version ` --layer-name python-common-app-modules ` --description "Common dependencies for Python application modules" ` --compatible-runtimes python3.6 python3.7 python3.8 python3.9 ` --zip-file fileb://python_layer.zip
arn return value. It is the unique identifier for the layer.
Update the Lambda function(s).
aws lambda update-function-configuration ` --function-name wahl-awesome-app ` --layers arn:aws:lambda:us-west-2:0000000000000:layer:python-common-app-modules:1
Test your application. If it breaks due to a missing dependency, go back through the loop starting with installing known dependencies. Sometimes I find myself missing something.