I have a folder structure like
Test
Test2
Test2-1.jpg
Test2-2.png
Using cp command I am able to copy the local structure to the S3 bucket. But I have a server configured to access files like this in a bucket
Test2/Test2-1.jpg, because I have copied it using cp command from a local directory, I can't set the key to Test2/Test2-1.jpg.
Before I was copying each and every file manually through Boto API by setting Key manually. That worked but its very long process.
Is there any way I can achieve this using cp command?
EDIT:
The actual issue causing the problem was content-encoding gzip. I was passing this encoding for not gz file. Due to that, the file is not stored properly and accessible.
If you are in a directory with Test2-1.jpg in it you can copy it to yourbucket/Test2/Test2-1.jpg by running
aws s3 cp ./Test2-1.jpg s3://yourbucket/Test2/Test2-1.jpg
You can copy an entire directory by using the sync command
aws s3 sync . s3://yourbucket/Test2/
Related
I need to "move" all the content of a folder, including its subfolders to a bucket in Google Cloud Storage.
The closest way is to use gsutil -rsync, but it clones all the data without moving the files.
I need to move all the data and keep data only in GCP and not in local storage. My local storage is being used only as a pass-thought server (Cause I only have a few GB to store data on local storage)
How can I achieve this?
Is there any way with gsutil?
Thanks!
To move the data to a bucket and reclaim the space on your local disk, you need to use mv command for example:
gsutil mv -r mylocalfolder gs://mybucketname
mv command copies the files to a bucket and delete them after the upload.
Is there a way to make gsutil rsync remove synced files?
As far as I know, normally it is done by passing --remove-source-files, but it does not seem to be an option with gsutil rsync (documentation).
Context:
I have a script that produces a large amount of CSV files (100GB+) I want those files to be transferred to Cloud Storage (and once transferred to be removed from my HDD).
Ended up using gcsfuse.
Per documentation:
Local storage: Objects that are new or modified will be stored in
their entirety in a local temporary file until they are closed or
synced.
One work-around for small buckets is delete all bucket contents and re-sync periodically.
I am trying to capture all incoming traffic on a specific port using GoReplay and to upload it directly to S3 servers.
I am running a simple file server on port 8000 and a gor instance using the (simple) command
gor --input-raw :8000 --output-file s3://<MyBucket>/%Y_%m_%d_%H_%M_%S.log
I does create a temporal file at /tmp/ but other than that, id does not upload any thing to S3.
Additional information :
The OS is Ubuntu 14.04
AWS cli is installed.
The AWS credentials are deffined within the environent
It seems the information you are providing or scenario you explained is not complete however to upload a file from your EC2 machine to S3 is simple as written command below.
aws s3 cp yourSourceFile s3://your bucket
To see your file you can see your file by using below command
aws s3 ls s3://your bucket
However, s3 is object storage and you can't use it to upload those files which are continually editing or adding or updating.
How to download simple storage service(s3) bucket files directly on user's local machine?
you can check the aws s3 cli so to copy a file from s3.
The following cp command copies a single object to a specified file locally:
aws s3 cp s3://mybucket/test.txt test2.txt
Make sure to use quotes " in case you have spaces in your key
aws s3 cp "s3://mybucket/test with space.txt" "./test with space.txt"
I am trying to use sf3f (https://github.com/s3fs-fuse/s3fs-fuse to handle files being uploaded by a different process to an s3 bucket. I can see files listed when I ls in hat mounted directory and the files are showing correctly timestamped and with the correct size. But trying to open the files either in code or using vi or nano just shows garbage such as
^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#
If I curl the s3 link then I do see content so it is available and there do not seem to be permission issues.
I am able to create a new file in this directory and the content is saving fine - can view it in the S3 explorer.
Any thoughts? Is this an s3fs-fuse problem and I may fair better using an alternate such as https://github.com/kahing/goofys?
I have signed up for the AWS EFS preview but no idea what the wait time is for that.