Reverse proxy Google Cloud storage to authenticate requests - authentication

Question

I'd like to provide an API to some clients to upload files to Google Cloud Storage. I don't want the clients to know the implementation details (whether they are uploading the files to Google Cloud Storage). The idea is that the API would authenticate the requests that they send and proxy them to another authenticated request to the Google Cloud API.
I'm not a backend/systems developer so I don't know what's the best way to approach this. The two options that I considered were:
Implementing a reverse proxy. My main concern with this approach is that those endpoints would have a lot of traffic and the API reverse proxy wouldn't be able to handle it.
Using the signed URL authentication mechanism from Google Cloud Storage. Users would ask the API for a URL to upload the resources to and the API would send that URL back to the user (that includes the authentication details embedded).
Has anyone experience with this kind of setup? How would you recommend me to do it?

Answer 1

I would suggest using App Engine Standard to setup the transfers. Unfortunately, not all the languages supported have equivalent libraries.
In php, here's an example of setting up direct transfers from users to the cloud:
https://cloud.google.com/appengine/docs/standard/php/googlestorage/user_upload
This requires the php CloudStorageTools, and although CloudStorageTools exist for other languages it isn't clear to me that they support the same operation.
In particular it isn't apparent that this exists in Python, but fortunately you can use the Blobstore API to do basically the same thing: https://cloud.google.com/appengine/docs/standard/python/blobstore/
The advantage of this approach is that the transfer doesn't go through App Engine, which both saves money and eliminates problems with App Engine timing out.
Java and Go don't seem to have equivalent methods available, but I don't use those languages so maybe I'm just missing them. Of course, if the files you are dealing with aren't too big you can upload them to App Engine and have App Engine store them before you timeout.
If you are setting up direct transfers you can't completely hide the URL of the destination, so your users could probably tell you were using Google Storage. I don't know if that is a problem, but if so you could use a bucket CNAME redirect to hide that too.
https://cloud.google.com/storage/docs/request-endpoints
Hope this is helpful.

Answer 2

There are examples on how to upload files using Cloud Storage. This is the example for Java, but there are other languages as well (Python, Node, Go, Ruby, etc).
The app uses a simple form to let the user choose a local file:
<!-- Rest of the code -->
<form method="POST" action="/upload" enctype="multipart/form-data">
<input type="file" name="file"> <input type="submit">
</form>
Then the form send this to the /upload servlet:
#Override
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException, ServletException {
final Part filePart = req.getPart("file");
final String fileName = filePart.getSubmittedFileName();
// Modify access list to allow all users with link to read file
List<Acl> acls = new ArrayList<>();
acls.add(Acl.of(Acl.User.ofAllUsers(), Acl.Role.READER));
// the inputstream is closed by default, so we don't need to close it here
Blob blob = storage.create(BlobInfo.newBuilder(BUCKET_NAME, fileName).setAcl(acls).build(),
filePart.getInputStream());
// return the public download link
resp.getWriter().print(blob.getMediaLink());
}
All of this happens in the back-end hence the user is not aware of what service you're using to upload the file, which is what you want.
In this particular case, make sure to avoid the lines:
resp.getWriter().print(blob.getMediaLink()); as it writes the address of the upload service:
https://www.googleapis.com/download/storage/v1/b/my-bucket-name/o/file-name.zip?generation=1524137198335299&alt=media
And also in the front-end, a line that tells the user:
<p>Select a file to upload to your Google Cloud Storage bucket.</p>