logo

How to read files from S3 using Python AWS Lambda

gcptutorials.com AWS

This post explains how to read a file from S3 bucket using Python AWS Lambda function. We will use boto3 apis to read files from S3 bucket.

In this tutorial you will learn how to
  1. Read a file from S3 using Python Lambda Function.
  2. List and read all files from a specific S3 prefix using Python Lambda Function.

Create Lambda Function

  1. Login to AWS account and Navigate to AWS Lambda Service.

  2. Navigate to AWS Lambda function and select Functions

  3. Click on Create function
  4. Select Author from scratch
  5. Enter Below details in Basic information
    • Function name: test_lambda_function
    • Runtime: choose run time as per the python version from output of Step 3
    • Architecture: x86_64
    • Select appropriate role that is having proper S3 bucket permission from Change default execution role

  6. Click on create function

Read a file from S3 using Lambda function

  • Import boto3 and create S3 client
           
import boto3
s3_client = boto3.client("s3")
  • Define bucket name
           
S3_BUCKET_NAME = 'BUCKET_NAME'
  • Define lambda handler

    Write below code in Lambda function and replace the OBJECT_KEY.

           
def lambda_handler(event, context):
    object_key = "OBJECT_KEY"  # replace object key
    file_content = s3_client.get_object(
        Bucket=S3_BUCKET, Key=object_key)["Body"].read()
    print(file_content)
  • Complete code for reading a S3 file with AWS Lambda Python
     
import boto3

s3_client = boto3.client("s3")
S3_BUCKET = 'BUCKET_NAME'
  
def lambda_handler(event, context):
    object_key = "OBJECT_KEY"  # replace object key
    file_content = s3_client.get_object(
        Bucket=S3_BUCKET, Key=object_key)["Body"].read()
    print(file_content)

List and read all files from a specific S3 prefix

  • Define bucket name and prefix.
       
import json
import boto3

s3_client = boto3.client("s3")
S3_BUCKET = 'BUCKET_NAME'
S3_PREFIX = 'BUCKET_PREFIX'

  • Write below code in Lambda handler to list and read all the files from a S3 prefix. Replace BUCKET_NAME and BUCKET_PREFIX.

     
    
def lambda_handler(event, context):
    response = s3_client.list_objects_v2(
        Bucket=S3_BUCKET, Prefix=S3_PREFIX, StartAfter=S3_PREFIX,)
    s3_files = response["Contents"]
    for s3_file in s3_files:
        file_content = json.loads(s3_client.get_object(
            Bucket=S3_BUCKET, Key=s3_file["Key"])["Body"].read())
        print(file_content)

  • Complete code snippet to list and read all files.
    •    
import json
import boto3

s3_client = boto3.client("s3")
S3_BUCKET = 'BUCKET_NAME'
S3_PREFIX = 'BUCKET_PREFIX'
  
  
def lambda_handler(event, context):
    response = s3_client.list_objects_v2(
        Bucket=S3_BUCKET, Prefix=S3_PREFIX, StartAfter=S3_PREFIX,)
    s3_files = response["Contents"]
    for s3_file in s3_files:
        file_content = json.loads(s3_client.get_object(
            Bucket=S3_BUCKET, Key=s3_file["Key"])["Body"].read())
        print(file_content)

Category: AWS