I have multiple files in S3 bucket like
file1.txt
file2.txt
file3.txt
another-file1.txt
another-file1.txt
another-file1.txt
now, I want to download first 3 files, name startwith "file*", How can i download from aws s3 in local git-bash console?
Simply you can download with below command :
aws s3 cp --recursive s3://bucket-name/ /local-destination-folder/ --exclude "*" --include "file*"
Related
How can I include the _ (underscore) on the include pattern?
I have a S3 bucket with files with the following format
20220630_084021_abc.json
20220630_084031_def.json
20220630_084051_ghi.json
20220630_084107_abc.json
20220630_084118_def.json
So, I would like to get all the files that start with 20220630_0840*.
So, I've tried to fetch them using multiple variations on the include pattern, so far I had used the followings:
aws s3 cp s3://BUCKET/ LocalFolder --include "20220630_0840*" --recursive
aws s3 cp s3://BUCKET/ LocalFolder --include "[20220630_0840]*" --recursive
aws s3 cp s3://BUCKET/ LocalFolder --include "20220630\_0840*" --recursive
None of them really work I'm getting all the files that have their name starting with 20220630
I want to upload a directory (A folder consist of other folders and .txt files) to a folder(partition) in a specific S3 bucket along with a given KMS-id via CLI. The following command which is to upload a jar file to an S3 bucket, was found.
The command I found for upload a jar:
aws s3 sync /?? s3://???-??-dev-us-east-2-813426848798/build/tmp/snapshot --sse aws:kms --sse-kms-key-id alias/nbs/dev/data --delete --region us-east-2 --exclude "*" --include "*.?????"
Suppose;
Location (Bucket Name with folder name) - "s3://abc-app-us-east-2-12345678/tmp"
KMS-id - https://us-east-2.console.aws.amazon.com/kms/home?region=us-east-2#/kms/keys/aa11-123aa-45/
Directory to be uploaded - myDirectory
And I want to know;
Whether the same command can be used to upload a directory with a
bunch of files and folders in it?
If so, how this command should be changed?
the cp command works this way:
aws s3 cp ./localFolder s3://awsexamplebucket/abc --recursive --sse aws:kms --sse-kms-key-id a1b2c3d4-e5f6-7890-g1h2-123456789abc
I haven't tried sync command with kms, but the way you use sync is,
aws s3 sync ./localFolder s3://awsexamplebucket/remotefolder
I want to download the contents of a s3 bucket (hosted on wasabi, claims to be fully s3 compatible) to my VPS, tar and gzip and gpg it and reupload this archive to another s3 bucket on wasabi!
My vps machine only has 30GB of storage, the whole buckets is about 1000GB in size so I need to download, archive, encrypt and reupload all of it on the fly without storing the data locally.
The secret seems to be in using the | pipe command. But I am stuck even in the beginning of download a bucket into an archive locally (I want to go step by step):
s3cmd sync s3://mybucket | tar cvz archive.tar.gz -
In my mind at the end I expect some code like this:
s3cmd sync s3://mybucket | tar cvz | gpg --passphrase secretpassword | s3cmd put s3://theotherbucket/archive.tar.gz.gpg
but its not working so far!
What am I missing?
The aws s3 sync command copies multiple files to the destination. It does not copy to stdout.
You could use aws s3 cp s3://mybucket - (including the dash at the end) to copy the contents of the file to stdout.
From cp — AWS CLI Command Reference:
The following cp command downloads an S3 object locally as a stream to standard output. Downloading as a stream is not currently compatible with the --recursive parameter:
aws s3 cp s3://mybucket/stream.txt -
This will only work for a single file.
You may try https://github.com/kahing/goofys. I guess, in your case it could be the following algo:
$ goofys source-s3-bucket-name /mnt/src
$ goofys destination-s3-bucket-name /mnt/dst
$ tar -cvzf /mnt/src | gpg -e -o /mnt/dst/archive.tgz.gpg
I would like to execute aws s3 sync . s3://<some bucket> and use the --exclude flag to exclude all files with an extension in the filename and change the content-type.
Tried this but does not work. Still finds files with extension.
/usr/bin/aws s3 sync /home/www s3://<bucket name> --dryrun --exclude "*.*" --include "*" --content-type text/html
You just need to use --exclude to exclude files with extensions
aws s3 sync --exclude "*.*" --content-type="text/html" . s3://hernan-test-bucket/
Example execution:
:~# ls
file1 file2 file3.txt file4.txt
:~# aws s3 sync --exclude "*.*" --content-type="text/html" . s3://bucket/
upload: ./file1 to s3://bucket/file1
upload: ./file2 to s3://bucket/file2
How to list the files in S3 using regex (in linux cli mode)? I have the files in s3 bucket like sales1.txt, sales2.txt etc. When I ran the below command nothing is displaying. Is there a command to list the all the files in S3 bucket with regex?
Command:
aws s3 ls s3://test/sales*txt
Expected output:
sales1.txt
sales2.txt
sales3.txt
Use the following command
aws s3 ls s3://test/ | grep '[sales].txt'
The accepted solution is too broad and matches too much. Try this:
aws s3 ls s3://test/ | grep sales.*\.txt
I have been trying to sort this, {aws s3 ls } command is not supporting any regex or pattern matching option. Wr have to use bash commands grep or awk.
aws s3 ls s3://bucket/path/ | grep sales|grep txt
aws s3 ls s3://bucket/path/ | grep sales..txt