AWS SES S3 process inbound email - amazon-s3

Question

I'm working on a publish by email system based on AWS SES. For all incoming emails I've set routing to save messages in an S3 bucket so I can asynchronously process them. The problem I have is that the messages are saved in the S3 bucket in a raw format: headers, email body, etc + the encrypted attachment (a huge string) - all in a single file.
Is there a way to break the email message apart form the attachment and save both in separate files at AWS SES level? I'm trying to get the data in the format I need straight from AWS and avoid adding another processing step to the process.
If AWS SES doesn't provide such a feature, what would be the proper way to process these messages to obtain the result described above?

Answer 1

It doesn't look possible to have SES automatically split up the email for you. As per the documentation here:
Amazon SES provides you the raw, unmodified email, which is typically
in Multipurpose Internet Mail Extensions (MIME) format.
I would use S3 or SNS to trigger a Lambda function whenever SES puts a new email file to S3. The Lambda function could split the file however you wish, then write those new files to another S3 bucket.

Answer 2

For anyone coming back later on to this question, this is the link to the JSON structure that you get when you invoke a Lambda function from SES.
http://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-notifications-examples.html
It took some searching to arrive at that page ;-)
From the link, a Lambda notification would look like this,
{
"notificationType": "Received",
"receipt": {
"timestamp": "2015-09-11T20:32:33.936Z",
"processingTimeMillis": 406,
"recipients": [
"recipient#example.com"
],
"spamVerdict": {
"status": "PASS"
},
"virusVerdict": {
"status": "PASS"
},
"spfVerdict": {
"status": "PASS"
},
"dkimVerdict": {
"status": "PASS"
},
"action": {
"type": "S3",
"topicArn": "arn:aws:sns:us-east-1:012345678912:example-topic",
"bucketName": "my-S3-bucket",
"objectKey": "\email"
}
},
"mail": {
"timestamp": "2015-09-11T20:32:33.936Z",
"source": "0000014fbe1c09cf-7cb9f704-7531-4e53-89a1-5fa9744f5eb6-000000#amazonses.com",
"messageId": "d6iitobk75ur44p8kdnnp7g2n800",
"destination": [
"recipient#example.com"
],
"headersTruncated": false,
"headers": [
{
"name": "Return-Path",
"value": "<0000014fbe1c09cf-7cb9f704-7531-4e53-89a1-5fa9744f5eb6-000000#amazonses.com>"
},
{
"name": "Received",
"value": "from a9-183.smtp-out.amazonses.com (a9-183.smtp-out.amazonses.com [54.240.9.183]) by inbound-smtp.us-east-1.amazonaws.com with SMTP id d6iitobk75ur44p8kdnnp7g2n800 for recipient#example.com; Fri, 11 Sep 2015 20:32:33 +0000 (UTC)"
},
{
"name": "DKIM-Signature",
"value": "v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1442003552; h=From:To:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Date:Message-ID:Feedback-ID; bh=DWr3IOmYWoXCA9ARqGC/UaODfghffiwFNRIb2Mckyt4=; b=p4ukUDSFqhqiub+zPR0DW1kp7oJZakrzupr6LBe6sUuvqpBkig56UzUwc29rFbJF hlX3Ov7DeYVNoN38stqwsF8ivcajXpQsXRC1cW9z8x875J041rClAjV7EGbLmudVpPX 4hHst1XPyX5wmgdHIhmUuh8oZKpVqGi6bHGzzf7g="
},
{
"name": "From",
"value": "sender#example.com"
},
{
"name": "To",
"value": "recipient#example.com"
},
{
"name": "Subject",
"value": "Example subject"
},
{
"name": "MIME-Version",
"value": "1.0"
},
{
"name": "Content-Type",
"value": "text/plain; charset=UTF-8"
},
{
"name": "Content-Transfer-Encoding",
"value": "7bit"
},
{
"name": "Date",
"value": "Fri, 11 Sep 2015 20:32:32 +0000"
},
{
"name": "Message-ID",
"value": "<61967230-7A45-4A9D-BEC9-87CBCF2211C9#example.com>"
},
{
"name": "X-SES-Outgoing",
"value": "2015.09.11-54.240.9.183"
},
{
"name": "Feedback-ID",
"value": "1.us-east-1.Krv2FKpFdWV+KUYw3Qd6wcpPJ4Sv/pOPpEPSHn2u2o4=:AmazonSES"
}
],
"commonHeaders": {
"returnPath": "0000014fbe1c09cf-7cb9f704-7531-4e53-89a1-5fa9744f5eb6-000000#amazonses.com",
"from": [
"sender#example.com"
],
"date": "Fri, 11 Sep 2015 20:32:32 +0000",
"to": [
"recipient#example.com"
],
"messageId": "<61967230-7A45-4A9D-BEC9-87CBCF2211C9#example.com>",
"subject": "Example subject"
}
}
}

Answer 3

Regarding the question on how to write a Lambda. Here is a portion of our Lambda. The main thing to take out of it is the parseEvent function. and data.event.Records[0] which will give you details
exports.handler = function(event, context, callback) {
var AWS = require('aws-sdk');
// Validate characteristics of a SES event record.
if (!event ||
!event.hasOwnProperty('Records') ||
event.Records.length !== 1 ||
event.Records[0].hasOwnProperty('eventSource') ||
event.Records[0].eventSource !== 'aws:ses' ||
event.Records[0].eventVersion !== '1.0') {
callback(null, {'disposition':'STOP_RULE_SET'});
}
email = data.event.Records[0].ses.mail;
subjectLine = event.Records[0].ses.mail.commonHeaders.subject;
}
The key is the event.Record[0].ses.mail. Unfortunately, I can't find the structure of it via a Google search, I am sure I had seen it before.