Python Meta Client Copy 403 Head Object Forbidden - amazon-s3

Question

import boto3
import botocore
from botocore.config import Config
from datetime import timedelta, datetime, date
import json
def get_creds(role):
session = botocore.session.get_session()
aws_access_id = session.get_credentials().access_key
aws_secret_id = session.get_credentials().secret_key
aws_token = session.get_credentials().token
return aws_access_id, aws_secret_id, aws_token
def create_connection(item, type):
role = get_creds('arn:aws:iam::123456789:role/LambdaRole')
my_config = Config(
region_name='us-east-1',
signature_version='s3v4'
)
if type == 'client' :
c = boto3.client(item,
config = my_config,
aws_access_key_id=role[0],
aws_secret_access_key=role[1],
aws_session_token=role[2],
)
else :
c = boto3.resource(item,
config = my_config,
aws_access_key_id=role[0],
aws_secret_access_key=role[1],
aws_session_token=role[2],
)
return c
def lambda_handler(event, context):
src_bucket = 'source_bucket'
dest_bucket = 'destination_bucket'
copy_to_prefix = dest_bucket + "/" + date.today().strftime("%Y/%m/%d") + '/'
s3 = create_connection('s3', 'client')
results = s3.list_objects_v2(Bucket = src_bucket)
keys = []
next_token = ''
while next_token is not None:
if next_token == '':
results = s3.list_objects_v2(Bucket = src_bucket, Prefix = 'FBI/')
elif next_token != '':
results = s3.list_objects_v2(Bucket = src_bucket, Prefix = 'FBI/', ContinuationToken = next_token)
next_token = results.get('NextContinuationToken')
contents = results.get('Contents')
for i in contents:
k = i.get('Key')
keys.append(k)
s3_resource = create_connection('s3', 'resource')
for k in keys:
copy_source = {
'Bucket': '{}'.format(src_bucket),
'Key': '{}'.format(k)
}
extra_args = { 'ACL' : 'bucket-owner-full-control' }
s3_resource.meta.client.copy(copy_source, dest_bucket , copy_to_prefix + '{}'.format(k.split("/",1)[1]), extra_args)
This script is resulting in "errorMessage": "An error occurred (403) when calling the HeadObject operation: Forbidden"
The script does list the files, I can print(keys) and see the list. It also is able to create the destination; I deleted the "sub-folder" and ran the script and it creates the entire structure.
It seems the problem is the actual get from the other account.
Bucket Policy on source account:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::source_bucket",
"arn:aws:s3:::source_bucket/FBI/*"
]
}
]
}
Inline IAM Policy for the role on destination account:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::source_bucket",
"arn:aws:s3:::source_bucket/FBI/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::destination_bucket",
"arn:aws:s3:::destination_bucket/*"
]
}
]
}
I also added a bucket policy on the destination bucket matching the IAM policy, just to be sure.
Any ideas why I am seeing the error.
EDIT:
There was one point in time where the Account A cloudwatch showed "arn:aws:sts::803456671434:assumed-role/<role_name>/s3_session"
However, even adding that to the bucket policy has not changed the outcome.

Answer 1

As it turns out none of my code or permissions setup are the issue. I was unaware that the files I am attempting to copy were actually loaded to the account I am trying to copy from by another account.
Here is the crux of what happens:
Account A (the originator account that I did not know about) -> Account B (the account I am trying to copy from), but does not give the ACL for bucket-owner-full-control. Due to the lack of ACL updates Account C (my account) does not have permissions to the file, but it does have access to the bucket because Account B is the originating account for the bucket.
Possible solutions:
When Account A uploads the object to a bucket present in Account B, the object owner can provide permissions to both Account A and Account C by specifying canonical ID's via '—grant' option. In this case, the object owner (Account A) is directly providing access to Account C by mentioning in the ACL. Example `aws s3 cp /localpath/ s3://AccountAbucket name/ --grants-full id=AccountAcanonicalID_< Account B > id=AccountCcanonicalID_Account C
Account A uploading the objects normally to the bucket present in Account B by providing access to the bucket owner. Now, Account B can have an event notification triggering a Lambda function to overwrite the objects uploaded by Account A so that ownership of the objects is changed to Account B. In this case, the bucket and object owner would be Account B, then account B could set up bucket policy allowing Account C to access
There is possibly another solution that involves IAM role assumptions and setting up of chaining of roles. Essentially IAM role in Account A Allows an IAM Role in Account B to assume it and then Account C assumes the Account B role. That seems like a potentially cumbersome setup and the chaining could be complicated.