Team,
I need to upload all files matching this pattern console.X.log to s3, where X=0...any
I tried below and getting error.
aws s3 cp /var/log/console.* s3://test/dom0/
Unknown options: /var/log/console.70.log,s3://0722-maglev-avdc-provisions/dom0/
AWS s3 cli doesn't support regex, but there is an exclude and include for s3.
So you should be able to use:
aws s3 cp /var/log/ s3://test/dom0/ --recursive --exclude "*" --include "console.*"
Note the order of the exclude and include, if you switch them around then nothing will be uploaded. You can include more patterns by adding more includes.
Currently, there is no support for the use of UNIX-style wildcards in a command's path arguments. However, most commands have --exclude "<value>" and --include "<value>" parameters that can achieve the desired result.
The following pattern symbols are supported.
*: Matches everything.
?: Matches any single character.
[sequence]: Matches any character in the sequence.
[!sequence]: Matches any character, not in the sequence.
Any number of these parameters can be passed to a command. You can do this by providing an --exclude or --include argument multiple times, e.g. --include "*.txt" --include "*.png".
When there are multiple filters, the rule is the filters that appear later in the command take precedence over filters that appear earlier in the command.
For example, if the filter parameters passed to the command were --exclude "*" --include "*.txt". All files will be excluded from the command except for files ending with .txt However if the order of the filter parameters was changed to --include "*.txt" --exclude "*". All files will be excluded from the command.
This is a simple example to upload your source code to S3 and exclude the git files: aws s3 cp /tmp/foo s3://bucket/ --recursive --exclude ".git/*"
Source
Related
How can I include the _ (underscore) on the include pattern?
I have a S3 bucket with files with the following format
20220630_084021_abc.json
20220630_084031_def.json
20220630_084051_ghi.json
20220630_084107_abc.json
20220630_084118_def.json
So, I would like to get all the files that start with 20220630_0840*.
So, I've tried to fetch them using multiple variations on the include pattern, so far I had used the followings:
aws s3 cp s3://BUCKET/ LocalFolder --include "20220630_0840*" --recursive
aws s3 cp s3://BUCKET/ LocalFolder --include "[20220630_0840]*" --recursive
aws s3 cp s3://BUCKET/ LocalFolder --include "20220630\_0840*" --recursive
None of them really work I'm getting all the files that have their name starting with 20220630
I am trying to copy pdf files that contain the keyword 'log' on their titles. I have the command below. What am I missing?
aws s3api copy-object --copy-source --key 'Log' --bucket
Finally managed to change things around and ended up with this that worked.
aws s3 cp s3://source-bucket/ s3://destination-bucket/ --recursive --exclude "*" --include "log"
I have an S3 bucket with a few thousand files where the file names always match the pattern {hostname}.{contenttype}.{yyyyMMddHH}.zip. I want to create a script that will run once a day to move these files into folders based on the year and month in the file name.
If I try the following aws-cli command
aws s3 mv s3://mybucket/*.202001* s3://mybucket/202001/
I get the following error:
fatal error: An error occurred (404) when calling the HeadObject operation: Key "*.202001*" does not exist
Is there an aws-cli command that I could run on a schedule to achieve this?
I think the way forward would be through the --filter parameter used in S3 CLI commands.
So, for your case,
aws s3 mv s3://mybucket/ s3://mybucket/202001/ --recursive --exclude "*" --include "*.202001*"
should probably do the trick.
For scheduling the CLI command to run daily, I think you can refer to On AWS, run an AWS CLI command daily
I'm copying between s3 buckets files from specific dates that are not sequence.
In the example I'm copying from the 23, so I'd like to copy 15th, 19, and 23rd.
aws s3 --region eu-central-1 --profile LOCALPROFILE cp
s3://SRC
s3://DEST --recursive
--exclude "*" --include "2016-01-23"
This source mentions using sequences http://docs.aws.amazon.com/cli/latest/reference/s3/
for include.
It appears that you are asking how to copy multiple files/paths in one command.
The AWS Command-Line Interface (CLI) allows multiple --include specifications, eg:
aws s3 cp s3://SRC s3://DEST --recursive --exclude "*" --include "2016-01-15/*" --include "2016-01-19/*" --include "2016-01-23/*"
The first --exclude says to exclude all files, then the subsequent --include parameters add paths to be included in the copy.
See: Use of Exclude and Include Filters in the documentation.
I want to backup all homes of my lxc containers with duplicity and i use (commandline simplified to the problem):
duplicity --include '/data/lxc/**/rootfs/home' --exclude '/data/lxc'
which does not match the homes, while
duplicity --include '/data/lxc/oneofthecontainers/rootfs/home' --exclude '/data/lxc'
works.
further testing shows, that
duplicity --include '/data/lxc/oneofthecontainers/rootfs/home/**' --exclude '/data/lxc'
does not work either. The manpage of duplicity tells me first match wins and * and ** are allowed as wildcards, where ** matches everything and * only one path component.
I still do not know, why this does not work, but i solved it with --include-globbing-filelist and a filelist with "+ " and "- " entries for includes/excludes and now it works.