What are the appraoches to process large S3 file from Lambda? - amazon-s3

Question

In project, there is a need to process a file having a record of 50K. But, as per current implementation by AWS, the response payload size returned by a Lambda function cannot exceed 6 mb.
Can anyone help me to know what are the approaches/implementation I can carried here?
Lambda code:
#Override
public String handleRequest(S3Event event, Context context) {
String bucket = null;
String key = null;
String jsonStringFromCSV = null;
csvToJsonParser = new FileToJsonParserServiceImpl();
context.getLogger().log("Received event: " + event);
try {
if (null != event.getRecords().get(0) && event.getRecords().size() > 0) {
bucket = event.getRecords().get(0).getS3().getBucket().getName();
key = event.getRecords().get(0).getS3().getObject().getKey();
S3Object response = s3.getObject(new GetObjectRequest(bucket, key));
String contentType = response.getObjectMetadata().getContentType();
context.getLogger().log("CONTENT TYPE: " + contentType);
InputStream objectData = response.getObjectContent();
BufferedReader bufferReader = new BufferedReader(new InputStreamReader(objectData, "UTF-8"));
jsonStringFromCSV = csvToJsonParser.getMemberObjectListAsJson(bufferReader);
}
return jsonStringFromCSV;
} catch (Exception e) {
e.printStackTrace();
context.getLogger().log(String.format("Error getting object %s from bucket %s. Make sure they exist and"
+ " your bucket is in the same region as this function.", key, bucket));
throw new RuntimeException(e);
}
}

Answer 1

The response size is not a problem here since you're already using S3, after all, your function is going to executed asynchronously and no other system will be able to grab it's response directly (you can, however, send its response to SNS or SQS and later process it, for example).
What you can do to achieve what you want, is to have your Lambda be triggered by an S3 Event. You would then process this file (it cannot exceed 15 mins, though, so if your file is really huge, then Lambda may not be the best tool here, but 50K records - considering the records are only a couple of (k)bytes long - should be fine) and output it somewhere else (be it DynamoDB, S3, RDS, SNS, SQS, etc).
Lambda's payload size limit is definitely not a constraint for you here.
EDIT: the OP wants to store the String generated out of the CSV file somewhere, so here's one way to achieve it:
You could store that big JSON string in a file in S3 without having to invoke another Lambda to do the job. Long story short, your function would process the CSV file, generate a string out of it and then save it to an S3 file. If you need to store it in a database, you could use DynamoDB, but since DDB has limits on the objects size, you could simply create an attribute in your table called processedFilePath which points to your S3 object.
If you want to decouple things a little bit, your source Lambda could store a file in a S3, called {some_identifier}_processed.txt and then put a message in SQS with this key (or use another S3 event, but let's use SQS to see different approaches). You could then wire up another Lambda to process this SQS message and store it in DynamoDB. Your object in DynamoDB would look like:
{
"id": "2511d862-53c1-48e2-beb4-ecc6c9deaac1",
"bucket": "my-processed-files-bucket",
"processedFileKey": "{some_identifier}_processed.txt"
}
This way you can circumvent payload sizes both on Lambda and DynamoDB and simply load the file based on processedFileKey when necessary