How to add metadata while using boto3 create_presigned_post? - amazon-s3

Want to add custom metadata to a file that I upload using create_presigned_post from boto3. I am running the following code but am getting 403 response. The code below is borrowed from here. Am I doing something wrong?
def create_presigned_post(bucket_name, object_name,
fields=None, conditions=None, expiration=3600):
# Generate a presigned S3 POST URL
s3_client = boto3.client('s3')
try:
response = s3_client.generate_presigned_post(bucket_name,
object_name,
Fields=fields,
Conditions=conditions,
ExpiresIn=expiration)
except ClientError as e:
print(e)
return None
# The response contains the presigned URL and required fields
return response
# Generate a presigned S3 POST URL
object_name = 'test-file.txt'
response = create_presigned_post('temp', object_name, fields={'x-amz-meta-test_key': 'test_val'})
# Demonstrate how another Python program can use the presigned URL to upload a file
with open('test-file.txt', 'rb') as f:
files = {'file': (object_name, f)}
http_response = requests.post(response['url'], data=response['fields'], files=files)
# If successful, returns HTTP status code 204
print(f'File upload HTTP status code: {http_response.status_code}')

As per document, fields dictionary will not be automatically added to the conditions list. You must specify a condition for the element as well.
response = create_presigned_post(bucket_name, object_name, fields={'x-amz-meta-test_key': 'test-val'}, conditions=[{'x-amz-meta-test_key': 'test-val'}])
It should work :)

Related

Is there a solution to uploading csv file to SQL

Anytime I tried uploading CSV file to Google Cloud Bigquery, I kept getting an error response. I tried Google drive to upload but it won't show the preview button on the table. I need help on how I can resolve this please.
You may want to try Loading CSV data from Cloud Storage. I used the following python code and I was able to load csv file to Bigquery successfully:
from google.cloud import bigquery
# Construct a BigQuery client object.
client = bigquery.Client()
# TODO(developer): Set table_id to the ID of the table to create.
table_id = "your-project.your_dataset.your_table_name"
job_config = bigquery.LoadJobConfig(
schema=[
bigquery.SchemaField("name", "STRING"),
bigquery.SchemaField("post_abbr", "STRING"),
],
skip_leading_rows=1,
# The source format defaults to CSV, so the line below is optional.
source_format=bigquery.SourceFormat.CSV,
)
uri = "gs://cloud-samples-data/bigquery/us-states/us-states.csv"
load_job = client.load_table_from_uri(
uri, table_id, job_config=job_config
) # Make an API request.
load_job.result() # Waits for the job to complete.
destination_table = client.get_table(table_id) # Make an API request.
print("Loaded {} rows.".format(destination_table.num_rows))

Read csv from s3 and upload to external api as multipart

I want to read the csv file from the s3 bucket using boto3 and upload it to external API using multipart/form-data request.
so far I am able to read the csv
response = s3.get_object(Bucket=bucket, Key=key)
body = response['Body']
Not sure on how to convert this body into multipart.
External api will be taking request in multipart/form-data.
Any Suggestions would be helpful.
Following method solved my issue.
body = response['Body'].read()
multipart_data = MultipartEncoder(
fields={
'file': (file_name, body, 'application/vnd.ms-excel'),
'field01': 'test'
}
)
.read() method will convert the file into binary string.

Boto3 generate presinged url does not work

Here is my code that I use to create a s3 client and generate a presigned url, which are some quite standard codes. They have been up running in the server for quite a while. I pulled the code out and ran it locally in a jupyter notebook
def get_s3_client():
return get_s3(create_session=False)
def get_s3(create_session=False):
session = boto3.session.Session() if create_session else boto3
S3_ENDPOINT = os.environ.get('AWS_S3_ENDPOINT')
if S3_ENDPOINT:
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_DEFAULT_REGION = os.environ["AWS_DEFAULT_REGION"]
s3 = session.client('s3',
endpoint_url=S3_ENDPOINT,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
region_name=AWS_DEFAULT_REGION)
else:
s3 = session.client('s3', region_name='us-east-2')
return s3
s3 = get_s3_client()
BUCKET=[my-bucket-name]
OBJECT_KEY=[my-object-name]
signed_url = s3.generate_presigned_url(
'get_object',
ExpiresIn=3600,
Params={
"Bucket": BUCKET,
"Key": OBJECT_KEY,
}
)
print(signed_url)
When I tried to download the file using the url in the browser, I got an error message and it says "The specified key does not exist." I noticed in the error message that my object key becomes "[my-bucket-name]/[my-object-name]" rather than just "[my-object-name]".
Then I used the same bucket/key combination to generate a presigned url using aws cli, which is working as expected. I found out that somehow the s3 client method (boto3) inserted [my-object-name] in front of [my-object-name] compared to the aws cli method. Here are the results
From s3.generate_presigned_url()
https://[my-bucket-name].s3.us-east-2.amazonaws.com/[my-bucket-name]/[my-object-name]?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAV17K253JHUDLKKHB%2F20210520%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210520T175014Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=5cdcc38e5933e92b5xed07b58e421e5418c16942cb9ac6ac6429ac65c9f87d64
From aws cli s3 presign
https://[my-bucket-name].s3.us-east-2.amazonaws.com/[my-object-name]?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYA7K15LJHUDAVKHB%2F20210520%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210520T155926Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=58208f91985bf3ce72ccf884ba804af30151d158d6ba410dd8fe9d2457369894
I've been working on this and searching for solutions for day and half and I couldn't find out what was wrong with my implementation. I guess it might be that I ignored some basic but important settings to create a s3 client using boto3 or something else. Thanks for the help!
Ok, myth is solved, I shouldn't provide the endpoint_url=S3_ENDPOINT param when I create the s3 client, boto3 will figure it out. After i removed it, everything works as expected.

"get_bucket_tagging" for s3 buckets give error, when there are no tags present

I am trying to get S3 bucket tags using "get_bucket_tagging".
Code:
response = client.get_bucket_tagging(Bucket='bucket_name')
print(response['TagSet'])
I am getting output till there are any tags present. But getting following error when there are 0 tags.
An error occurred (NoSuchTagSet) when calling the GetBucketTagging
operation: The TagSet does not exist
Is there any other method to check that?
From this document:
NoSuchTagSetError - There is no tag set associated with the bucket.
So when there is no tag set associated with the bucket, error/exception is expected. You need to handle this exception.
import boto3
client = boto3.client('s3')
try:
response = client.get_bucket_tagging(Bucket='bucket_name')
print(response['TagSet'])
except Exception, e:
# Handle exception
# Do something
print e

created s3 presigned url (put) with custom headers with boto3

My current code looks like this
s3 = boto3.client('s3')
presigned_url = s3.generate_presigned_url(
'put_object',
Params={'Bucket':bucket_name, 'Key':object_key},
ExpiresIn=3600,
HttpMethod='PUT' )
This is working, but I want to include custom headers like x-amz-meta-my-custom-meta-data. I'm pretty sure S3 supports this, so how can I do this with boto3?
Its not clear from the documentation.
Using Python 3.6
It is a NO and is still classified as a feature request as of Oct 2017.
https://github.com/boto/boto3/issues/1294
Hope it helps.
Send it as metadata
s3 = boto3.client('s3')
presigned_url = s3.generate_presigned_url(
'put_object',
Params={'Bucket':bucket_name, 'Key':object_key, "Metadata": {"mechaGodzilla": "anything is possible"}},
ExpiresIn=3600,
HttpMethod='PUT' )
In your request headers, you must include x-amz-meta-mechaGodzilla: "anything is possible"