I am trying to configure CORS in my IBM Cloud Object Storage bucket. I dont see any option to do that from bucket configuration in UI and I can see it can only be done through CLI. The command looks similar to how its done in AWS CLI as well. This is the command to configure CORS,
ibmcloud cos bucket-cors-put --bucket BUCKET_NAME [--cors-configuration STRUCTURE] [--region REGION] [--output FORMAT]
It is expecting cors configuration STRUCTURE in JSON format from a file and add it as --cors-configuration file://<filename.json>. I have created a configuration file as cors.json and saved it on my Desktop. But when I am providing path for that file and running the command, I am getting this error,
The value in flag '--cors-configuration' is invalid
I am providing file path like this - --cors-configuration file:///C:/Users/KirtiJha/Desktop/cors.json
I am new with Cloud CLI. Am I doing wrong here? Any help is much appreciated
You can configure CORS in the CLI or via API and SDKs. On the CLI, you can use the IBM Cloud COS plugin in the bucket-cors-put command as you mentioned.
The file URI seems valid to me. You could try to set it in quotes ("file:///..."). Also, try to copy the file into your current directory and then test with --cors-configuration file://cors.json.
When I use gsutil to connect to my bucket on Google Cloud Storage, I usually use the following command:
gcloud auth activate-service-account --key-file="pathKeyFile"
What should I do if two scripts that are running on the same machine at the same time need two different Service Accounts?
I would like to use a command such as:
gsutil ls mybucket --key-file="mykeyspath"
I say this because in the case my script is running and another script changes the Service Account which is actually active, my script would not have permission to access the bucket anymore.
You can do this with BOTO file. You can create one as explained in the documentation.
Then you can specify which file to use when you run your gsutil command (here an example in linux)
# if you have several GSUTIL command to run
export BOTO_CONFIG=/path/to/.botoMyBucket
gsutil ls myBucket
# For only one command, you can define an env var inline like this
BOTO_CONFIG=/path/to/.botoMyBucket2 gsutil ls myBucket2
I need to copy many (1000+) files to s3 from GCS to leverage an AWS lambda function. I have edited ~/.boto.cfg and commented out the 2 aws authentication parameters but a simple gsutil ls s3://mybucket fails from either an GCE or EC2 VM.
Error is The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256..
I use gsutil version: 4.28 and locations of GCS and S3 bucket are respectively US-CENTRAL1 and US East (Ohio) - in case this is relevant.
I am clueless as the AWS key is valid and I enabled http/https. Downloading from GCS and uploading to S3 using my laptop's Cyberduck is impracticable (>230Gb)
As per https://issuetracker.google.com/issues/62161892, gsutil v4.28 does support AWS v4 signatures by adding to ~/.boto a new [s3] section like
[s3]
# Note that we specify region as part of the host, as mentioned in the AWS docs:
# http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
host = s3.eu-east-2.amazonaws.com
use-sigv4 = True
The use of that section is inherited from boto3 but is currently not created by gsutil config so it needs to be added explicitly for the target endpoint.
For s3-to-GCS, I will consider the more server-less Storage Transfer Service API.
I had a similar problem. Here is what I ended up doing on a GCE machine:
Step 1: Using gsutil, I copied files from GCS to my GCE hard drive
Step 2: Using aws cli (aws s3 cp ...), I copied files from GCE hard drive to s3 bucket
The above methodology has worked reliably for me. I tried using gsutil rsync but it fail unexpectedly.
Hope this helps
I'm using s3 adapter to copy files from a snowball device to local machine.
Everything appears to be in order as I was able to run this command and see the bucket name:
aws s3 ls --endpoint http://snowballip:8080
But besides this, aws doesn't offer any examples for calling cp command. How do I provide the bucket name and the key with this --endpoint flag.
Further, when I ran this:
aws s3 ls --endpoint http://snowballip:8080/bucketname
It returned 'Bucket'... Not sure what that means because I expect to see the files.
I can confirm the following is correct for snowball and snowball edge, as #sqlbot says in the comment
aws s3 ls --endpoint http://snowballip:8080 s3://bucketname/[optionalprefix]
References:
http://docs.aws.amazon.com/cli/latest/reference/
http://docs.aws.amazon.com/snowball/latest/ug/using-adapter-cli.html
Just got one in the post
I can push some content to an S3 bucket with my credentials through S3cmd tool with s3cmd put contentfile S3://test_bucket/test_file
I am required to download the content from this bucket in other computers that don't have s3cmd installed on them, BUT they have wget installed.
when I try to download some content from my bucket with wget I get this:
https://s3.amazonaws.com/test_bucket/test_file
--2013-08-14 18:17:40-- `https`://s3.amazonaws.com/test_bucket/test_file
Resolving s3.amazonaws.com (s3.amazonaws.com)... [ip_here]
Connecting to s3.amazonaws.com (s3.amazonaws.com)|ip_here|:port... connected.
HTTP request sent, awaiting response... 403 Forbidden
`2013`-08-14 18:17:40 ERROR 403: Forbidden.
I have manually made this bucket public through the Amazon AWS web console.
How can I download content from an S3 bucket with wget into a local txt file?
You should be able to access it from a url created as follows:
http://{bucket-name}.s3.amazonaws.com/<path-to-file>
Now, say your s3 file path is:
s3://test-bucket/test-folder/test-file.txt
You should be able to wget this file with following url:
http://test-bucket.s3.amazonaws.com/test-folder/test-file.txt
Go to S3 console
Select your object
Click 'Object Actions'
Choose 'Download As'
Use your mouse right-click to 'Copy Link Address'
Then use the command:
wget --no-check-certificate --no-proxy 'http://your_bucket.s3.amazonaws.com/your-copied-link-address.jpg'
AWS cli has a 'presign' command that one can use to get a temporary public URL to a private s3 resource.
aws s3 presign s3://private_resource
You can then use wget to download the resource using the presigned URL.
Got it ... If you upload a file in an S3 bucket with S3CMD with the --acl public flag then one shall be able to download the file from S3 with wget easily ...
Conclusion: In order to download with wget, first of one needs to upload the content in S3 with s3cmd put --acl public --guess-mime-type <test_file> s3://test_bucket/test_file
alternatively you can try:
s3cmd setacl --acl-public --guess-mime-type s3://test_bucket/test_file
notice the setacl flag above. THAT WILL set the file in s3 accessible publicly
then you can execute the wget http://s3.amazonaws.com/test_bucket/test_file
I had the same situation for couple of times. It’s the fastest and the easiest way to download any file from AWS using CLI is next command:
aws s3 cp s3://bucket/dump.zip dump.zip
File downloaded way faster than via wget, at least if you are outside of US.
I had the same error and I solved it by adding a Security Groups Inbound rule:
HTTPS type at port 443 to my IP address ( as I'm the only one accessing it ) for the subnet my instance was in.
Hope it helps anyone who forgot to include this
Please make sure that the read permission has been given correctly.
If you do not want to enter any account/password, just by wget command without any password, make sure the permission is like the following setting shows.
By Amazon S3 -> Buckets -> Permisions - Edit
Check the Object for "Everyone (public access)" and save changes.permission setting like this - screenshot
or choose the objest and go to "Actions" -> "Make public", would do the same thing under permission settings.
incase you do not have access to install aws client on ur Linux machine try below method.
got to the bucket and click on download as button. copy the link generated.
execute command below
wget --no-check-certificate --no-proxy --user=username --ask-password -O "download url"
Thanks
you have made the bucket public, you need to also make the object public.
also, the wget command doesn't work with the S3:// address, you need to find the object's URL in AWS web console.
I know I'm too late to this post. But thought I'll add something no one mentioned here.
If you're creating a presigned s3 URL for wget, make sure you're running aws cli v2.
I ran into the same issue and realized s3 had this problem
Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4
This gets resolved once you presign on aws cli v2
The simplest way to do that is to disable Block all public firstly.
Hit your bucket name >> go to Permissions >> Block public access (bucket settings)
If it is on >> hit Edit >> Uncheck the box, then click on Save changes
Now hit the object name >> Object action >> Make public using ACL >> then confirm Make public
After that, copy the Object URL, and proceed to download
I hope it helps the future askers. Cheers
I had the same mistake
I did the following :
created IAM role > AWS Service type > AmazonS3FullAccess policy inside
applied this role to the EC2 instance
in the the Security Groups opened inbound HTTP and HTTPS to Anywhere-IPv4
made the S3 bucket public
profit! wget works! ✅