Reading multiple files from S3 bucket and processing them using Lambda trigger - amazon-s3

Question

I am reading multiple files in S3, processing them and then making tables in AWS RDS with these processed dataframes. I am doing all this on my Mac OS using PyCharm.
I want to read these csv files from the S3 bucket and run this same python script to process these files in AWS and not on my local system. I want to use lambda to trigger this script and it should run only when all the needed files are uploaded in the bucket.
How would the code vary in AWS Lambda ?
My present code is as below -
import boto3
import pandas as pd
import numpy as np
import sys
client = boto3.client('s3')
resource = boto3.resource('s3')
my_bucket = resource.Bucket('test-s3')
#CREATE ALL THE NEEDED OBJECTS
obj1 = client.get_object(Bucket='test-s3', Key='file1.csv')
obj2 = client.get_object(Bucket='test-s3', Key='file2.csv')
obj3 = client.get_object(Bucket='test-s3', Key='file3.csv')
obj4 = client.get_object(Bucket='test-s3', Key='file4.csv')
obj5 = client.get_object(Bucket='test-s3', Key='file5.csv')
obj6 = client.get_object(Bucket='test-s3', Key='file6.csv')
obj7 = client.get_object(Bucket='test-s3', Key='file7.csv')
obj8 = client.get_object(Bucket='test-s3', Key='file8.csv')
obj9 = client.get_object(Bucket='test-s3', Key='file9.csv')
obj10 = client.get_object(Bucket='test-s3', Key='file10.csv')
obj11 = client.get_object(Bucket='test-s3', Key='file11.csv')
obj12 = client.get_object(Bucket='test-s3', Key='file12.csv')
obj13 = client.get_object(Bucket='test-s3', Key='file13.csv')
obj14 = client.get_object(Bucket='test-s3', Key='file14.csv')
obj15 = client.get_object(Bucket='test-s3', Key='file15.csv')
#CREATE ALL THE DATAFRAMES FROM RESPECTIVE OBJECTS
df_file1 = pd.read_csv(obj1['Body'], encoding='utf-8', sep = ',')
df_file2 = pd.read_csv(obj2['Body'], encoding='utf-8', sep = ',')
df_file3 = pd.read_csv(obj3['Body'], encoding='utf-8', sep = ',')
df_file4 = pd.read_csv(obj4['Body'], encoding='utf-8', sep = ',')
df_file5 = pd.read_csv(obj5['Body'], encoding='utf-8', sep = ',')
df_file6 = pd.read_csv(obj6['Body'], encoding='utf-8', sep = ',')
df_file7 = pd.read_csv(obj7['Body'], encoding='utf-8', sep = ',')
df_file8 = pd.read_csv(obj8['Body'], encoding='utf-8', sep = ',')
df_file9 = pd.read_csv(obj9['Body'], encoding='utf-8', sep = ',')
df_file10 = pd.read_csv(obj10['Body'], encoding='utf-8', sep = ',')
df_file11 = pd.read_csv(obj11['Body'], encoding='utf-8', sep = ',')
df_file12 = pd.read_csv(obj12['Body'], encoding='utf-8', sep = ',')
df_file13 = pd.read_csv(obj13['Body'], encoding='utf-8', sep = ',')
df_file14 = pd.read_csv(obj14['Body'], encoding='utf-8', sep = ',')
df_file15 = pd.read_csv(obj15['Body'], encoding='utf-8', sep = ',')
#+++++++++++ make a function to process the data frames ++++++++++++
def function(df_file1, df_file2):
*** some logic ***
return df_final
## MAKE THE TABLES IN RDS
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://USERNAME:PASSWORD#***.eu-central-1.rds.amazonaws.com:5432/DBNAME')
df_final.to_sql('table name', engine, schema='data')
I am a noob at AWS Lambda. How do I run this script on Lambda?
After taking Ninad's suggestion I edited the script. It's as below-
import boto3
import pandas as pd
import numpy as np
import sys
client = boto3.client('s3')
resource = boto3.resource('s3')
my_bucket = resource.Bucket('test-s3')
def function(df_file1, df_file2):
*** some logic ***
return df_final
def lambda_handler(event, context):
obj1 = client.get_object(Bucket='test-s3', Key='file1.csv')
obj2 = client.get_object(Bucket='test-s3', Key='file2.csv')
obj3 = client.get_object(Bucket='test-s3', Key='file3.csv')
df_file1 = pd.read_csv(obj1['Body'], encoding='utf-8', sep=',')
df_file2 = pd.read_csv(obj2['Body'], encoding='utf-8', sep=',')
df_file3 = pd.read_csv(obj3['Body'], encoding='utf-8', sep=',')
df_final = function(df_file1, df_file2)
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://USERNAME:PASSWORD#***.eu-central-1.rds.amazonaws.com:5432/DBNAME')
df_final.to_sql('table name', engine, schema='data')
I made a virtual environment in my local system and installed all the packages - pandas, SQLAlchemy etc. I zipped this package and the script and uploaded it to Lambda. Now I am getting this error -
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'pandas'
I have followed aws package deploy link to package all necessary stuff. Why Do I still get an error then ?

Answer 1

Use the console to create a lambda. Select the correct python version you want and also ensure you have alloted enough memory and put the timeout time to 15 minutes (maximum). When creating the lambda it will also let you attach a role to it. Create a role and attach a policy to the role that lets you access the the s3 bucket where your CSVs will be.
Next step is to create a layer for your lambda which will have all the dependencies that you need for your script to run. Lambda by default has boto3 package installed but you will need to install pandas (with all its dependencies), sqlalchemy and psycopg2. You can find a simple tutorial on how to do this here
Now that you have created a layer, attach that layer to your lambda.
We can finally move on to your script. Since you need to read all csv files on your s3 path, you will have to change your script to read the csv files dynamically. Currently you have hardcoded the names of the csv files. You can change your script to first get all the keys in your bucket using something like:
response = client.list_objects_v2(
Bucket=my_bucket
)['Contents']
This will give you a list of your keys. Filter them if you need.
Next you can create multiple dataframes by looping through the response like this:
d = {}
for idx, obj in enumerate(response):
d['df_'+idx] = pd.read_csv(client.get_object(Bucket='test-s3', Key=obj['Key'])['Body'], encoding='utf-8', sep = ',')
This will create a dictionary d with all your dataframes. Please try this code out locally first to correct any mistakes.
Now copy your final code and paste it in the lambda editor above the def lambda handler(): Call your function from within the lambda handler function.