I was chatting about AWS Lambda layers with a colleague and referred to an older blog post titled How to Create AWS Lambda Layers for Python. I realize that I was doing things the hard way and that a few tweaks were in order.
AWS Lambda layers are constructs that provide libraries and modules to code being executed by Lambda during runtime. For Python, this means modules stored in site-packages. Layers are handy for deduplicating effort and code across a common set of Lambda functions, which would alternatively have dependencies bundled directly into the source code. For a few of my projects, many Lambda functions depend on identical or nearly identical module sets. Thus, a layer provides a single point of dependency control across a multitude of functions.
The Easy Way
The venv module is packaged directly with modern version of Python 3. I use a virtual environment with every project. Once I reach a state of development for which publishing a Lambda function makes sense, I wipe the virtual environment. This avoids any chance of contaminating my requirements list from previous experiments during development.
From there, I create a new virtual environment and activate it.
py -m venv .venv
.\.venv\Scripts\activate Make sure to update pip.
py -m pip install --upgrade pipCheck the pip list to validate a clean working environment is presented back.
pip list
Package Version
---------- -------
pip 21.2.4
setuptools 49.2.1Install your known dependencies and run the code to validate a successful execution.
Freeze pip to the source code directory to couple them tightly. I do not like putting requirements in the repo’s root directly because Explicit is better than Implicit. Note that for Lambda deployments, you can skip installing boto3 as this module is always available to executed code. You will need it locally, however. 🙂
pip freeze > src\requirements.txtI use this little script to create the zip archive. It copies over the site-package modules to the root of the virtual environment, renames the folder python to comply with AWS Lambda layer requirements, generates a zip from the results, and then purges the python folder.
Copy-Item -Path .\.venv\Lib\site-packages\ .\.venv -Recurse
Rename-Item -Path .\.venv\site-packages python
Compress-Archive -Path .\.venv\python\ -DestinationPath .\.venv\python_layer.zip
Remove-Item -Path .\.venv\python\ -Recurse -ForceFrom there, switch into the .venv directory and publish the layer or update an already published layer.
aws lambda publish-layer-version `
--layer-name python-common-app-modules `
--description "Common dependencies for Python application modules" `
--compatible-runtimes python3.6 python3.7 python3.8 python3.9 `
--zip-file fileb://python_layer.zipSave the arn return value. It is the unique identifier for the layer.
Update the Lambda function(s).
aws lambda update-function-configuration `
--function-name wahl-awesome-app `
--layers arn:aws:lambda:us-west-2:0000000000000:layer:python-common-app-modules:1Test your application. If it breaks due to a missing dependency, go back through the loop starting with installing known dependencies. Sometimes I find myself missing something.
Enjoy!
