I'm trying to upload documents to my cloudsearch domain through AWS CLI using the following command:
aws cloudsearchdomain upload-documents --endpoint-url
http://doc-domainname-id.region.cloudsearch.amazonaws.com/2013-01-01/documents/batch
--content-type application/json --documents documents-batch.json
My access policies are open to everyone search and update but i'm still getting an exception when every time i try to upload a batch of documents:
An error occurred (CloudSearchException) when calling the
UploadDocuments operation: Request forbidden by administrative rules.
I've already uploaded files before using the same commands and everything was fine. Now i'm getting this issue.
Any help would be welcome. Thank you.
After making some changes for an aws hosted static website, I deleted an aws s3 bucket through the AWS console. However, the bucket is now orphaned. Although it is not listed in the AWS console, I can see still reach what is left of it through the CLI and through the URI.
When I try to recreate a www bucket with the same name, the AWS console returns the following error:
Bucket already exists
The bucket with issues has a www prefix, so now I have two different versions (www and non-www) of the same website.
The problem URI is:
www.michaelrieder.com and www.michaelrieder.com.s3-eu-west-1.amazonaws.com
I made many failed attempts to delete the bucket using the aws s3 CLI utility. I tried aws rb force, aws rm, and any other command I remotely thought might work.
I need to delete and recreate the bucket with exactly the same name so I can have www website redirection working correctly as aws enforces static website naming conventions strictly.
When I execute the aws s3 CLI command for example:
aws s3 rb s3://www.michaelrieder.com --force --debug
A typical CLI error message is:
An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied
It thought it might be a cache related issue and that the the bucket would flush itself after a period of time, but the issue has persisted for over 48 hours.
It seems to be a permissions issue, but I cannot find a way to change the phantom bucket’s permissions or any method of deleting the bucket or even it’s individual objects, since I do not have access to the bucket via the AWS console or the aws s3 CLI.
Appreciate any ideas. Please help.
Create folders and upload files to my S3 bucket stopped working.
The remote server returned an error: (403) Forbidden.
Everything seems to work previously as i did not change anything recently
After days of testing - i see that i am able to create folders in my bucket from localhost but same code doesnt work on the EC2 instance.
I must resolve the issue ASAP.
Thanks
diginotebooks
Does your EC2 instance have a role? If yes, what is this role? Is it possible that someone detached or modified a policy that was attached to it?
If your instance doesn't have a role, how do you upload files to S3? Using the AWS CLI tools? Same questions for the IAM profile used.
If you did not change anything - are you using the same IAM credentials from the server and localhost? May be related to this.
Just random thoughts...
I'm trying to use Cyberduck CLI for uploading/downloading files from Amazon S3 bucket.
But I'm unable to formulate the correct S3 url.
Below is what I've tried so far for listing the bucket contents.
C:\>duck --list s3://<bucketname>.s3-<region>.amazonaws.com/<key> --username <access_key> --password <secret_key>
But I'm getting the error:
Listing directory failed. Java.lang.NullPointerException.
Please contact your web hosting service provider for assistance.
Can you please advise if there is any issue with the s3 URL?
Cyberduck version - 4.8
The documentation mentions to
reference the target container (aka bucket) name in the URI like s3://bucketname/key.
This means the region specific hostname s3-<region>.amazonaws.com can be omitted.
I can push some content to an S3 bucket with my credentials through S3cmd tool with s3cmd put contentfile S3://test_bucket/test_file
I am required to download the content from this bucket in other computers that don't have s3cmd installed on them, BUT they have wget installed.
when I try to download some content from my bucket with wget I get this:
https://s3.amazonaws.com/test_bucket/test_file
--2013-08-14 18:17:40-- `https`://s3.amazonaws.com/test_bucket/test_file
Resolving s3.amazonaws.com (s3.amazonaws.com)... [ip_here]
Connecting to s3.amazonaws.com (s3.amazonaws.com)|ip_here|:port... connected.
HTTP request sent, awaiting response... 403 Forbidden
`2013`-08-14 18:17:40 ERROR 403: Forbidden.
I have manually made this bucket public through the Amazon AWS web console.
How can I download content from an S3 bucket with wget into a local txt file?
You should be able to access it from a url created as follows:
http://{bucket-name}.s3.amazonaws.com/<path-to-file>
Now, say your s3 file path is:
s3://test-bucket/test-folder/test-file.txt
You should be able to wget this file with following url:
http://test-bucket.s3.amazonaws.com/test-folder/test-file.txt
Go to S3 console
Select your object
Click 'Object Actions'
Choose 'Download As'
Use your mouse right-click to 'Copy Link Address'
Then use the command:
wget --no-check-certificate --no-proxy 'http://your_bucket.s3.amazonaws.com/your-copied-link-address.jpg'
AWS cli has a 'presign' command that one can use to get a temporary public URL to a private s3 resource.
aws s3 presign s3://private_resource
You can then use wget to download the resource using the presigned URL.
Got it ... If you upload a file in an S3 bucket with S3CMD with the --acl public flag then one shall be able to download the file from S3 with wget easily ...
Conclusion: In order to download with wget, first of one needs to upload the content in S3 with s3cmd put --acl public --guess-mime-type <test_file> s3://test_bucket/test_file
alternatively you can try:
s3cmd setacl --acl-public --guess-mime-type s3://test_bucket/test_file
notice the setacl flag above. THAT WILL set the file in s3 accessible publicly
then you can execute the wget http://s3.amazonaws.com/test_bucket/test_file
I had the same situation for couple of times. It’s the fastest and the easiest way to download any file from AWS using CLI is next command:
aws s3 cp s3://bucket/dump.zip dump.zip
File downloaded way faster than via wget, at least if you are outside of US.
I had the same error and I solved it by adding a Security Groups Inbound rule:
HTTPS type at port 443 to my IP address ( as I'm the only one accessing it ) for the subnet my instance was in.
Hope it helps anyone who forgot to include this
Please make sure that the read permission has been given correctly.
If you do not want to enter any account/password, just by wget command without any password, make sure the permission is like the following setting shows.
By Amazon S3 -> Buckets -> Permisions - Edit
Check the Object for "Everyone (public access)" and save changes.permission setting like this - screenshot
or choose the objest and go to "Actions" -> "Make public", would do the same thing under permission settings.
incase you do not have access to install aws client on ur Linux machine try below method.
got to the bucket and click on download as button. copy the link generated.
execute command below
wget --no-check-certificate --no-proxy --user=username --ask-password -O "download url"
Thanks
you have made the bucket public, you need to also make the object public.
also, the wget command doesn't work with the S3:// address, you need to find the object's URL in AWS web console.
I know I'm too late to this post. But thought I'll add something no one mentioned here.
If you're creating a presigned s3 URL for wget, make sure you're running aws cli v2.
I ran into the same issue and realized s3 had this problem
Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4
This gets resolved once you presign on aws cli v2
The simplest way to do that is to disable Block all public firstly.
Hit your bucket name >> go to Permissions >> Block public access (bucket settings)
If it is on >> hit Edit >> Uncheck the box, then click on Save changes
Now hit the object name >> Object action >> Make public using ACL >> then confirm Make public
After that, copy the Object URL, and proceed to download
I hope it helps the future askers. Cheers
I had the same mistake
I did the following :
created IAM role > AWS Service type > AmazonS3FullAccess policy inside
applied this role to the EC2 instance
in the the Security Groups opened inbound HTTP and HTTPS to Anywhere-IPv4
made the S3 bucket public
profit! wget works! ✅