How to read files from S3 using Python AWS Lambda

This post explains how to read a file from S3 bucket using Python AWS Lambda function. We will use boto3 apis to read files from S3 bucket.

In this tutorial you will learn how to

1. Read a file from S3 using Python Lambda Function.
2. List and read all files from a specific S3 prefix using Python Lambda Function.

See Also

• Download object from S3 bucket using CLI
• Using request module in AWS Lambda
• Steps to configure AWS CLI
• Python extract text from pdf files

Create Lambda Function

1. Login to AWS account and Navigate to AWS Lambda Service.

   

2. Navigate to AWS Lambda function and select Functions

   

3. Click on Create function

4. Select Author from scratch

5. Enter Below details in Basic information

   • Function name: test_lambda_function
   • Runtime: choose run time as per the python version from output of Step 3
   • Architecture: x86_64
   • Select appropriate role that is having proper S3 bucket permission from Change default execution role

6. Click on create function

Read a file from S3 using Lambda function

• Import boto3 and create S3 client

         
  import boto3
  s3_client = boto3.client("s3")
         

• Define bucket name

         
  S3_BUCKET_NAME = 'BUCKET_NAME'
         

• Define lambda handler

  Write below code in Lambda function and replace the OBJECT_KEY.

         
  def lambda_handler(event, context):
    object_key = "OBJECT_KEY"  # replace object key
    file_content = s3_client.get_object(
        Bucket=S3_BUCKET, Key=object_key)["Body"].read()
    print(file_content)
         

• Complete code for reading a S3 file with AWS Lambda Python

   
  import boto3
  
  s3_client = boto3.client("s3")
  S3_BUCKET = 'BUCKET_NAME'
  
  def lambda_handler(event, context):
    object_key = "OBJECT_KEY"  # replace object key
    file_content = s3_client.get_object(
        Bucket=S3_BUCKET, Key=object_key)["Body"].read()
    print(file_content)
        
         

List and read all files from a specific S3 prefix

• Define bucket name and prefix.

     
  import json
  import boto3
  
  s3_client = boto3.client("s3")
  S3_BUCKET = 'BUCKET_NAME'
  S3_PREFIX = 'BUCKET_PREFIX'
       

• Write below code in Lambda handler to list and read all the files from a S3 prefix. Replace BUCKET_NAME and BUCKET_PREFIX.

   
    
  def lambda_handler(event, context):
    response = s3_client.list_objects_v2(
        Bucket=S3_BUCKET, Prefix=S3_PREFIX, StartAfter=S3_PREFIX,)
    s3_files = response["Contents"]
    for s3_file in s3_files:
        file_content = json.loads(s3_client.get_object(
            Bucket=S3_BUCKET, Key=s3_file["Key"])["Body"].read())
        print(file_content)
       

• Complete code snippet to list and read all files.

   
import json
import boto3

s3_client = boto3.client("s3")
S3_BUCKET = 'BUCKET_NAME'
S3_PREFIX = 'BUCKET_PREFIX'


def lambda_handler(event, context):
  response = s3_client.list_objects_v2(
      Bucket=S3_BUCKET, Prefix=S3_PREFIX, StartAfter=S3_PREFIX,)
  s3_files = response["Contents"]
  for s3_file in s3_files:
      file_content = json.loads(s3_client.get_object(
          Bucket=S3_BUCKET, Key=s3_file["Key"])["Body"].read())
      print(file_content)