I am using localstack version 0.12.19.4 on a Mac.
I have created an s3 bucket called mybucket
localstack start ------ s3 runs on port 4566
http://localhost:4566/health ---- everything is running
awslocal s3 mb s3://mybucket
awslocal s3api put-bucket-acl --bucket mybucket --acl public-read
I add some files to my s3 bucket and then I check with awslocal and aws
aws --endpoint-url=http://127.0.0.1:4566 s3 ls
awslocal s3 ls
shows my bucket existing.
Now from a docker image, when I try to access one of the files in mybucket s3 bucket, I get the following error:
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:4566/mybucket/dev/us/2020_08_11/eea9efc9-5970-426b-b867-9f57d6d9548f/850f35c8-0ada-44e4-96e1-e050e3040609"
when I check the contents of the s3 bucket, I do see the specific file existing.
one more fact when I retrieve docker port for localstack I see
4566/tcp -> 127.0.0.1:4566
4571/tcp -> 127.0.0.1:4571
Any ideas as to what I am doing wrong or missing?
Related
I had 10 subfolders named like 1-10 in my s3 bucket, but in my ec2 instance I had only 3 subfolders named as 1,2,3. So I want to sync (or cp) with s3 bucket only for to get these 3 folders data without having all 10 folders in my ec2. Is there any possibility to do something like this?
aws s3 sync s3://mybucket/ . --include "*"
But i'm getting all 10 folders in my ec2 which are present in s3 bucket.
You can try below command which will first exclude everything and then will only include those folders which you actually want :
aws s3 sync s3://mybucket/ . --exclude "*" --include "1" --include "2" --include "3"
It's been decreed that all our S3 buckets should have access logs and versioning enabled. Unfortunately I have a lot of S3 buckets. Is there an efficient way of doing this that doesn't involve setting the attributes on each one individually in the console?
You can also develop your own custom AWS Config rule to manage the compliance of AWS S3 Buckets. (versionning and logs enabled)
https://aws.amazon.com/config/
You can check a lot of examples here:
https://github.com/awslabs/aws-config-rules
You can adapt this one to your needs:
https://github.com/awslabs/aws-config-rules/blob/master/python/s3_bucket_default_encryption_enabled.py
For most of the tasks on AWS, the simplest way is using the AWS CLI, especially about the repetitive things.
You can use AWS CLI and simple bash script like this, by rtrouton:
#!/bin/bash
# This script is designed to check the object versioning status of all S3 buckets associated with an AWS account
# and enable object versioning on any S3 buckets where object versioning is not enabled.
# Get list of S3 buckets from Amazon Web Services
s3_bucket_list=$(aws s3api list-buckets --query 'Buckets[*].Name' | sed -e 's/[][]//g' -e 's/"//g' -e 's/,//g' -e '/^$/d' -e 's/^[ \t]*//;s/[ \t]*$//')
# Loop through the list of S3 buckets and check the individual bucket's object version status.
for bucket in $(echo "$s3_bucket_list")
do
version_status=$(aws s3api get-bucket-versioning --bucket "$bucket" | awk '/Status/ {print $2}' | sed 's/"//g')
if [[ "$version_status" = "Enabled" ]]; then
# If the object version status is Enabled, report that the S3 bucket has object versioning enabled.
echo "The $bucket S3 bucket has object versioning enabled."
elif [[ "$version_status" != "Enabled" ]]; then
# If the object version is a status other than Enabled, report that the S3 bucket does not have
# object versioning enabled, then enable object versioning
echo "The $bucket S3 bucket does not have object versioning enabled. Enabling object versioning on the $bucket S3 bucket."
aws s3api put-bucket-versioning --bucket "$bucket" --versioning-configuration Status=Enabled
fi
done
For more information you can check the following document on the AWS website:
https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-versioning.html
I want to list all of the files in an AWS S3 bucket that end in .css.
I saw this question: Filter S3 list-objects results to find a key matching a pattern and from reading it, I tried this:
aws s3api list-objects --bucket mybucket --query "Contents[?contains(Key, 'css')]"
That returned every file inside a /css folder as well as files with 'css' anywhere in the name. I want the equivalent of find "*.css". Is that possible?
Try using ends_with function.
aws s3api list-objects --bucket mybucket --query "Contents[?ends_with(Key, 'css')]"
I am trying to fetch a file from s3 using aws-cli
aws s3api get-object --bucket <bucket_name> --key /foo.com/bar/summary-report-yyyymmdd.csv.gz temp_file.csv.gz --profile <profile_name>
but I am getting the following error -
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
I've rechecked my configuration using
aws configure --profile <profile_name>
and everything seems to be correct there. I am using the same credentials to browse and fetch the file on S3 browser without any issue.
Documentation is of minimal use as I have very limited access to this bucket. I cannot verify the permissions or use
aws s3 --profile <profile_name> ls
AccessDenied can mean you dont have permission but its the error returned if the object does not exist (you can read here for the reason why to use this error)
You can make sure you have access to the bucket using the aws s3api list-objects command like
aws s3api list-objects --bucket <bucket_name> --query 'Contents[].{Key: Key, Size: Size}' --profile <profile_name>
Most probably in your case the issue is with the user of / in front of the key
aws s3api get-object --bucket <bucket_name> --key foo.com/bar/summary-report-yyyymmdd.csv.gz temp_file.csv.gz --profile <profile_name>
For me the issue was kms access. I found this helpful:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
Say I have a website that return me JSON data when I send a GET request using curl. I want to re-direct the output of curl to AWS S3. A new file should be created on S3 for it.
Currently I am able to redirect the output to store it locally.
curl -s -X GET 'http://website_that_returns_json.com' > folder_to_save/$(date +"%Y-%m-%d_%H-%M.json")
I have AWS CLI and s3cmd installed. How would I redirect the output of create to create a new file on AWS S3 ?
Assume :
AWS S3 access key and secret key are already set.
Location to store file : mybucket/$(date +"%Y-%m-%d_%H-%M.json"
The AWS Command-Line Interface (CLI) has the ability to stream data to/from Amazon S3:
The following cp command uploads a local file stream from standard input to a specified bucket and key:
aws s3 cp - s3://mybucket/stream.txt
So, you could use:
curl xxx | aws s3 cp - s3://mybucket/object.txt
However, it's probably safer to save the file locally and then copy it to Amazon S3.
In case you'd like to run the command on the remote, use aws ssm send-command.
Then to redirect the output of that command to S3, you can use --output-s3-bucket-name parameter.
Here is Bash script to run PowerShell script on the remote and upload it into S3 bucket:
instanceId="i-xyz"
bucketName="bucket_to_save"
bucketDir="folder_to_save"
command="Invoke-WebRequest -UseBasicParsing -Uri http://example.com).Content"
cmdId=$(aws ssm send-command --instance-ids "$instanceId" --document-name "AWS-RunPowerShellScript" --query "Command.CommandId" --output text --output-s3-bucket-name "$bucketName" --output-s3-key-prefix "$bucketDir" --parameters commands="'${command}'")
while [ "$(aws ssm list-command-invocations --command-id "$cmdId" --query "CommandInvocations[].Status" --output text)" == "InProgress" ]; do sleep 1; done
outputPath=$(aws ssm list-command-invocations --command-id "$cmdId" --details --query "CommandInvocations[].CommandPlugins[].OutputS3KeyPrefix" --output text)
echo "Command output uploaded at: s3://${bucketName}/${outputPath}"
aws s3 ls "s3://${bucketName}/${outputPath}"
To output the uploaded S3 files, run:
aws s3 ls s3://${bucketName}/${outputPath}/stderr.txt && aws s3 cp --quiet s3://${bucketName}/${outputPath}/stderr.txt /dev/stderr
aws s3 cp --quiet s3://${bucketName}/${outputPath}/stdout.txt /dev/stdout