I have S3 bucket in EC2 . I want to remove multiple files between s3 folders . however it showing deleted files but files are still there
command:
aws s3 rm s3://mybucket/path1/publish/test/dummyfile_*.dat
got below message
delete: s3://mybucket/path1/publish/test/dummyfile_*.dat,. But file is still present
can anyone please help
"Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in all Regions."
from https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#CoreConcepts
If you make a copy of a S3 object to an EC2 instance, you simply made a copy of it.
You can use aws s3 sync to synchronize S3 objects (files) between S3 and your EC2 instance, see https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
Related
It created an S3 bucket. If I delete it, it just creates a new one. How can I set it to not create a bucket or to stop write permissions from it?
You cannot prevent AWS Elastic Beanstalk from creating S3 Bucket as it stores your application and settings as a bundle in that bucket and executes deployments. That bucket is required till the time you run/deploy your application using AWS EB. Please be vary of deleting these buckets as this may cause your deployments/applications to crash. Although, you may remove older objects (which may not be in use).
Take a look at this link for a detailed information on how EB uses S3 buckets for deployments https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.S3.html
In my pipeline I am trying to sync my local folder (or should I say repository folder) to the s3 bucket. Now I can do the aws s3 sync . s3:// but this off course gives an error, since the bucket is not specified. But basically that is exactly what I want. Exactly how my folder-structure locally is; is how I want in S3.
so locally:
bucket1/file1.txt
bucket1/file2.txt
bucket1/subbucket1/file3.txt
needs to go exactly to root of my s3 account... how to fix this?
btw; the sync might be an overkill since I only want to copy (and overwrite!) to the s3 folders, coming from the root. Not (yet) interested in deleting etc.
what can I do..?
The AWS Command-Line Interface (CLI) aws s3 sync command requires a bucket name.
Therefore, you will either need to write a script that extracts the bucket name and inserts it into the aws s3 sync command, or you'll need to write your own program to use in place of the AWS CLI.
If you have a limited number of buckets and they don't change that often, you could just write a script that repeatedly calls the AWS CLI, such as:
aws s3 sync bucket1/ s3://bucket1/
aws s3 sync bucket2/ s3://bucket2/
etc.
if somebody comes to the same question:
for file in `find -type f`;
do
newFilename="${file#./}"
dirName=$ENVIRONMENT-$(dirname "$newFilename")
#get first part of dir (only root)
dirName="${dirName%%/*}"
echo bucket: $dirName
if aws s3api head-bucket --bucket "$dirName" 2>/dev/null; then
echo "bucket already exists"
else
if [[ $dirName == *"/"* ]]; then
echo $dirName
echo "This bucket is a subfolder and will not be created"
else
aws s3 mb s3://$dirName
fi
fi
aws s3 cp $newFilename s3://$ENVIRONMENT-$newFilename
done
the scripts retrieves all the files that it can find;
then it will check the root directory (relative to the current folder)
it will check it the directory exists as a bucket. If not; it will be created.
And then every file will be copied.
Since i do not know if a root-directory exists (as a bucket) we have to manually check it.
I couldn't use the sync because I might not have an existing bucket.
If you do know that your root directory as a bucket exists; then i would use the sync, one liner vs 10-liner :see_no_evil:.
anyway, that was it for me!
I'm trying to make a copy of s3 bucket on aws and it is really pain.
My reference s3 bucket is: s3://original
Duplicated ver. of this bucket: s3://original-copy
My goal is:
generate kubernetes.tf file with kops create cluster ... => DONE
kops is kind enough to create --state=s3://original => DONE
now I want to create a new s3 bucket with exactly same content as in s3://original just the name is different s3://original-copy => PROBLEM
Command
aws s3 cp s3://original s3://original-copy --recursive --acl bucket-owner-full-control
Even though bucket is duplicated it seems like there is some problem with s3 bucket permissions
Then I am adjusting values in terraform/data folder with a new reference to s3://original-copy as well as at s3://original-copy
s3://original-copy/cluster_name/config
s3://original-copy/cluster_name/cluster.spec
files.
But there is a problem with permissions all the time.
Error:
s3context.go:145] unable to get bucket location from region "us-east-1"; scanning all regions: AccessDenied: Access Denied
Idea
The main idea is that kops will generate
kubernetes.tf file and data folder with proper files (all within terraform folder) just once
--state=s3://original bucket just once
Once we have some example (patterns) of s3 and kuberetes.tf we would stop using kops.
Is there a way I can autosave autocad files or changes on the autocad files directly to S3 Bucket?, probably an API I can utilize for this workflow?
While I was not able to quickly find a plug in that does that for you, what you can do is one of the following:
Mount S3 bucket as a drive. You can read more at CloudBerry Drive - Mount S3 bucket as Windows drive
This might create some performance issues with AutoCad.
Sync saved files to S3
You can set a script to run every n minutes that automatically syncs your files to S3 using aws s3 sync. You can read more about AWS S3 Sync here. Your command might look something like
aws s3 sync /path/to/cad/files s3://bucket-with-cad/project/test
I search a way to replicate between S3 buckets across regions.
The purpose is that if a file accidentally deleted because a bug in my application, I would be able to restore it from the other bucket.
There is any way to do it without upload the file twice (meaning, not in the application layer)?
Set versioning on your S3 Bucket. After that it will keep all version files which you uploaded or updated in S3 Bucket. After that you can restore any version of file from version listing. See - Amazon S3 Object Lifecycle Management