It was far to difficult to figure this out. It wasn't obvious to me and lots of explanations left out key details. I will answer this with the solution. Sorry if it seems obvious to you, but given how many searches and experiments it took me to do this, I think it is quite worthwhile to show others how to do it.
UPDATE: According to a commenter, the extra parameters I show here may no longer be needed. I am not currently working with AWS, so I don't have a way to verify it. Anyway, I didn't change the rest of this post in case it is still needed in some case.
The trick is being explicit about both the source and destination regions. They might not always be required, but it doesn't hurt to always show them:
$ aws s3 cp s3://my-source-bucket-in-us-west-2/ \
s3://my-target-bucket-in-us-east-1/ \
--recursive --source-region us-west-2 --region us-east-1
Or on Windows
> aws s3 cp s3://my-source-bucket-in-us-west-2/ ^
s3://my-target-bucket-in-us-east-1/ ^
--recursive --source-region us-west-2 --region us-east-1
Related
Ideally, I'd like to be able to do this with aws cli, but I'm open to alternatives. Assuming I'm authenticated to a particular aws account, is there any way to look up basic information about a cluster, or all clusters in the account, without knowing what region the cluster is in? I'd love a way to get information about a cluster without already knowing meta information about it. I could write a script to cycle through all regions looking for for clusters, but I hope there's a better way.
Here is a bash for loop that should do the trick:
for region in `aws ec2 describe-regions --output text | cut -f4`
do
echo -e "\nListing Clusters in region: $region..."
aws eks list-clusters --region $region --output text --no-cli-pager
done
A handy command is eksctl get cluster --all-regions -o json.
I'm practicing deployment using aws s3 and cloudFront.
The issue I met was that account B approached S3, which made by account A, and uploaded the object using Github Actions.
I solved this issue with changing two things.
First, I added external account with Grantee in S3 ACL like picture.
Second, I added code in .yml file --acl bucket-owner-full-control like below for Github Actions.
...
- name: S3 Deploy
run: aws s3 sync ./dist s3://s3-bucket-name/ --acl bucket-owner-full-control
Both two options are set, the function is working.
But, I don't understand why this option --acl bucket-owner-full-control is needed.
I thought the first setting was enough for account B to access S3 owned by account A.
I found something that may be related to this topic. Is it related to this issue?
I want to check if some files are really in my s3 bucket. Using aws cli I can do it for one files with ls like
aws s3 ls s3://mybucket/file/path/file001.jpg
but I need to be able to do it for several files
aws s3 ls s3://mybucket/file/path/file001.jpg ls s3://mybucket/file/path/file002.jpg
Won't work
nor
aws s3 ls s3://mybucket/file/path/file001.jpg s3://mybucket/file/path/file005.jpg
Of course
aws s3 ls s3://mybucket/file/path/file001.jpg;
aws s3 ls s3://mybucket/file/path/file005.jpg
Works perfectly, but slowly. It takes about 1 sec to get one file, because it connects an close the connection each time.
I've hundreds of files to check on a regular basis, so I need a fast way to do it. Thanks
I'm not insisting on using ls, or passing a path, a "find" of the filenames would also do (but aws cli seems to lack find). Another tool (as long as it can be invoked with the command line), will be ok
I don’t want to get a list of all files or have a script looking at all files and then post process. I need a way to ask s3 give me fila a,r,z in one go.
I think s3api listobjects call should be the one but I fail at its syntax to ask several file names at once.
You can easily do that using python boto3 sdk for AWS
import boto3
s3 = boto3.resource('s3')
bucket=s3.Bucket('mausamrest');
for obj in bucket.objects.all():
print(obj.key)
where mausamrest is the bucket
I'm using s3 adapter to copy files from a snowball device to local machine.
Everything appears to be in order as I was able to run this command and see the bucket name:
aws s3 ls --endpoint http://snowballip:8080
But besides this, aws doesn't offer any examples for calling cp command. How do I provide the bucket name and the key with this --endpoint flag.
Further, when I ran this:
aws s3 ls --endpoint http://snowballip:8080/bucketname
It returned 'Bucket'... Not sure what that means because I expect to see the files.
I can confirm the following is correct for snowball and snowball edge, as #sqlbot says in the comment
aws s3 ls --endpoint http://snowballip:8080 s3://bucketname/[optionalprefix]
References:
http://docs.aws.amazon.com/cli/latest/reference/
http://docs.aws.amazon.com/snowball/latest/ug/using-adapter-cli.html
Just got one in the post
I've been using riofs for AWS S3 bucket to filesystem mounting on an EC2 instance.
Yes yes, I know... you shouldn't use fuse for S3 mounting as its a leaky abstraction.
With that said, I didn't see anyone else post any questions regarding this issue (specifically with riofs) so I'm posting this question.
Why is it happening and what can I do it fix it?
It looks like fuse doesn't have a stable version past v2.9.7 so that is what I'm using.
s3fs does have an open issue which I'm guessing has the same problem (https://github.com/s3fs-fuse/s3fs-fuse/issues/152).
I have tried adding -o multireq_max=5 as a riofs command-line option without success.