Goal: To be able to generate a link that allows the recipient to upload data to a specific folder in Google Cloud Storage regardless of them having a Google account (or any, for that matter). The link should authenticate them and give them access to the folder itself. This is intended to work in a very similar fashion as Drop Box where you can request files from users, and they get a URL to upload files to specified folders.
I've been able to follow the instructions from the GCP page and created a signed URL by running a gsutil command:
gsutil signurl -m PUT -d 1h -c application/octet-stream private_key.json gs://my_bucket/my_folder/
My expectation is to be able to copy the generated URL, and access it from a browser, to be welcomed into the GCS folder. But I keep getting this error:
<Error>
<Code>MalformedSecurityHeader</Code>
<Message>Your request has a malformed header.</Message>
<ParameterName>content-type</ParameterName>
<Details>Header was included in signedheaders, but not in the request.</Details>
</Error>
My gut tells me I'm either trying to use the signed URL in a way it's not meant to be used (maybe it should be part of a code that 'calls' that URL and lets the user access it via a UI). And I'm not a programmer, but more of an IT admin trying to automate file sharing/receiving. Or I'm doing something wrong; and thus my question here.
Any input you can give me would be greatly appreciated! Thanks in advance!
Related
Our application data storage is backed by Google Cloud Storage (and S3 and Azure Blob Storage). We need to give access to this storage to random outside tools (upload from local disk using CLI tools, unload from analytical database like Redshift, Snowflake and others). The specific use case is that users need to upload multiple big files (you can think about it much like m3u8 playlists for streaming videos - it's m3u8 playlist and thousands of small video files). The tools and users MAY not be affiliated with Google in any way (may not have Google account). We also absolutely need to data transfer to be directly to the storage, outside of our servers.
In S3 we use federation tokens to give access to a part of the S3 bucket.
So model scenario on AWS S3:
customer requests some data upload via our API
we give customers S3 credentials, that are scoped to s3://customer/project/uploadId, allowing upload of new files
client uses any tool to upload the data
client uploads s3://customer/project/uploadId/file.manifest, s3://customer/project/uploadId/file.00001, s3://customer/project/uploadId/file.00002, ...
other data (be it other uploadId or project) in the bucket is safe because the given credentials are scoped
In ABS we use STS token for the same purpose.
GCS does not seem to have anything similar, except for Signed URLs. Signed URLs have a problem though that they refer to a single file. That would either require us to know in advance how many files will be uploaded (we don't know) or the client would need to request each file's signed URL separately (strain on our API and also it's slow).
ACL seemed to be a solution, but it's only tied to Google-related identities. And those can't be created on demand and fast. Service users are also and option, but their creation is slow and generally they are discouraged for this use case IIUC.
Is there a way to create a short lived credentials that are limited to a subset of the CGS bucket?
Ideal scenario would be that the service account we use in the app would be able to generate a short lived token that would only have access to a subset of the bucket. But nothing such seems to exist.
Unfortunately, no. For retrieving objects, signed URLs need to be for exact objects. You'd need to generate one per object.
Using the * wildcard will specify the subdirectory you are targeting and will identify all objects under it. For example, if you are trying to access objects in Folder1 in your bucket, you would use gs://Bucket/Folder1/* but the following command gsutil signurl -d 120s key.json gs://bucketname/folderName/** will create a SignedURL for each of the files inside your bucket but not a single URL for the entire folder/subdirectory
Reason : Since subdirectories are just an illusion of folders in a bucket and are actually object names that contain a ‘/’, every file in a subdirectory gets its own signed URL. There is no way to create a single signed URL for a specific subdirectory and allow its files to be temporarily available.
There is an ongoing feature request for this https://issuetracker.google.com/112042863. Please raise your concern here and look for further updates.
For now, one way to accomplish this would be to write a small App Engine app that they attempt to download from instead of directly from GCS which would check authentication according to whatever mechanism you're using and then, if they pass, generate a signed URL for that resource and redirect the user.
Reference : https://stackoverflow.com/a/40428142/15803365
We are building a system which will allow users to store documents (images or pdf) in s3. These files can later be accessed via their URL like (https://my-bucket.s3.amazonaws.com/person/2/provider/test1.png)). We have no problem uploading and deleting documents in s3 using proper IAM keys and AWS SDK. However, when we try to access documents using their URL, we get the following error:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>V0DTW6T3F6J3ZFDG</RequestId>
<HostId>ZOiybrTAfx8t+NZQW2cpS4nw8vNhmQaemFfinQSBP41K2mZhDItF29156LTUwZh+SqZacfssLIE=</HostId>
</Error>
I understand the reason for the error but don't know how to resolve it.
Basically, we are building a health related portal. Patients can upload their documents (health records). Later on we want them to be able to view their documents while they are logged in. The idea was that the documents could be displayed via their URL.
The only solution I can think of is to first download the document locally (whether that is on the browser's local filesystem or on the mobile device if accessed through a mobile app) and then displayed from there. That does not sound like an ideal way. The other alternative is to make the bucket completely public which is not acceptable because these are health care records.
Any help/advice will be greatly appreciated.
I've uploaded a file using the SDK to my private S3 bucket.
I can access this file through the S3 UI.
However, I cannot access this file through a direct link. It gives me some XML that includes "AccessDenied" as a code and message.
It seems reasonable that since I'm authenticated in the browser and am clicking on a direct link to the file from the same browser, that I should be allowed through. At the very least, I should be directed to a login page.
Does anyone have any experience with this?
So after working on this for a bit, I discovered the best thing is to simply publish the console URL to the file.
https://s3.console.aws.amazon.com/s3/object/{your bucket}/{your file path}?region={the region of your bucket}&tab=overview
Be mindful to specify the correct region. If you're forming this programmatically, then use Amazon.RegionEndpoint.SystemName.
If you're not logged in, it will ask you for your login!
No signed URL is necessary.
Thanks to everyone who contributed!
There are 2 places that you need to make sure are set correctly based on how you want to setup access to the bucket. It will either have public or private access.
The properties tab:
Here you can set what you will use the bucket for.
The Permissions tab -> Bucket Policy:
With this, you can then setup access. I was able to generate a policy with this site:
http://awspolicygen.s3.amazonaws.com/policygen.html
EDIT:
Mine is working with the settings I have shown. I recommend asking the AWS boards if to get to the bottom of it. You could also try this:
You can use the direct link if you are inside a VPC. You have to :
1- Create a VPC endpoint for Amazon S3.
2- Add a bucket policy that allows access from the VPC endpoint.
All steps are described in the following link :
https://aws.amazon.com/premiumsupport/knowledge-center/s3-private-connection-no-authentication/?nc1=h_ls
I'm about to create a blog and wanted to host the content on Dropbox (only Markdown files, everything else is on my server) because I want to be able to quickly upload and edit posts from all my devices without having to get some FTP app on them.
So far I managed to access files via https://dl.dropboxusercontent.com/u/********/<sub-folder>/<file name>.md and like pointed out here I can get the last-modified attribute via etags.
However, I'm stuck at getting a file list (or a folder's metadata in general). Is this possible without using OAuth?
No, this isn't possible without using the API. But why are you against using the API?
Feel free to use my tool https://dbxoauth2.site44.com/ to get an OAuth access token for your account. Then just do an HTTP GET to https://api.dropbox.com/1/metadata/auto/<path> with an Authorization header of Bearer <token>. Ditto for getting the actual file contents, just use /files/auto/<path> instead.
Oh, unless you're doing this client-side in JavaScript? In that case, this won't work since you can't give your access token out to the client (where it could be abused by anyone who visited your website).
make "Public" folder and go https://www.dropbox.com/enable_public_folder
back to your dropbox folder (web) right mouse at file in public folder and chose "Copy public link..."
Update: As of September 1, 2017 public links have been disabled for all users.
https://help.dropbox.com/files-folders/share/public-folder
I'm building a web application and am looking into using Amazon S3 to store user uploads.
My concern is, I dont want user A to see his download link for a document he uploaded is urltoMyS3/doc1234.pdf and try urltoMyS3/doc1235.pdf and get another users document.
The only way I can think of to do this, is to only allow the web application to connect to S3, then check if the user has access to a file on the web application, have the web app download the file, and then serve it to the client. The problem with this method is the application would have to download the file first and would inevitably slow the download process down for the user.
How is user files typically handled with Amazon S3? Or is it simply not typically used in a scenario where the files should not be public? Is there another service for something like this?
Thanks
You can implement Query String Authentication, which will solve your problem.
Query string authentication is useful for giving HTTP or browser
access to resources that would normally require authentication. The
signature in the query string secures the request. Query string
authentication requests require an expiration date. You can specify
any future expiration time in epoch or UNIX time (number of seconds
since January 1, 1970).
You can do this by generating the appropriate links, see the following
https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth
If time-bound authentication will not work for (as suggested in other answers). You could consider implementing something like s3fs to mount your S3 bucket as a drive on your web application server. In this manner you can simply make your authentication and then serve up the file directly to the user, without them having any idea that the file resides in S3. Similarly, you can simply write uploaded files directly to this s3fs mount.
S3fs, also allows you to configure a local cache of the S3 directory on your machine for faster access.
This works nicely in a cluster web server environment as well, as you can just have each server mount the s3fs drive and perform/read/writes on it independently.
A link with more info