How to use pandas in AWS Lambda

gcptutorials.com AWS

pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. pandas library is by default not available in AWS Lambda Python environments. If you try to import pandas in aws lambda function, you will get below error.

    
  import pandas
  
  def lambda_handler(event, context):
  
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }
  
  # Output 
  
  Response
  {
  "errorMessage": "Unable to import module 'lambda_function': No module named 'pandas'",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "a9dfd983-8cd6-4fe2-85c3-5e107ac230a4",
  "stackTrace": []
  }

For using pandas library in Lambda function a Lambda Layer needs to attached to the Lambda function. This tutorials lists the required steps for creating and attaching Lambda Layer for pandas module.

Note: Step 1 to Step 6 needs to performed on EC2 instance which uses the same Amazon Linux version as AWS Lambda to have proper dependencies.The steps for this tutorial as are performed with Python 3.9.7, to follow the steps make sure you are using Python 3.9.7

Step 1: Create Python Virtual Environment

  
  
  python3.9 -m venv test_venv

Step 2: Activate Virtual Environment

  
  source test_venv/bin/activate

Step 3: Check Python Version

  
  
  python --version

Step 4: Create directory with name python

  
  
  mkdir python

Step 5: Install `pandas` library in python directory created in Step 4

  
  
  pip install pandas -t python

Step 6: Zip python directory

  
  
  zip -r pandas.zip python

Step 7: Login to AWS account and Navigate to AWS Lambda Service.

Step 8: In AWS Lambda select Layers from Additional resources.

Step 9: Click on create layer, enter the required information.

Name: pandas_layer
Description: Lambda layer for pandas module
Select Upload a .zip file, click on upload and choose pandas.zip created in Step 6
Compatible architectures - optional: x86_64
Compatible runtimes - Choose run time as per the python version from output of Step 3

Step 10: Click on Create

Step 11: Navigate to AWS Lambda function and select Functions

Step 12: Click on Create function

Step 13: Select Author from scratch

Step 14: Enter Below details in Basic information

Function name: test_lambda_function
Runtime: choose run time as per the python version from output of Step 3
Architecture: x86_64

Step 15: Click on create function

Step 16: In the Function overview pane click on Layers or Scroll down to select Layers section

Step 17: Click on Add a layer

Step 18: Select Custom layers , choose layer created in Step 9, select version 1 and click on Add.

Step 19: Write below code in lambda function and click on Deploy

  
  
  import logging
  import pandas as pd
  
  logger = logging.getLogger()
  logger.setLevel(logging.INFO)
  
  
  def lambda_handler(event, context):
    logger.info(pd.__version__)