I have some questions about uploading images to cloud storage, I wish someone could help me. I would like answers according to best practices:
Which is better, send the images to my API with all other form information and then send them to the cloud or upload them to the cloud directly via Frontend and separate from other form information?
Do I need to make a request for each image or is it better to upload all images directly?
Backend generates a signed url at which the client uploads the video directly - this requires solid authentication
Depends if the client has a good internet connection or not but batching is usually good
Related
I have images stored in Google Cloud Storage. Whenever my frontend (Swift) requests a certain image I would like to send the image as quickly and efficiently as possible from my backend.
Conveniently, Google Cloud Storage has direct image links for every image.
Is it most efficient to send a multipart/form-data the same way I send an image captured by a user in the front end to the backend? Or is it more efficient to send the URL of the image stored in the cloud where the frontend can proceed to download the image from that URL?
This can indeed be done through a signed URL which provides limited permission and time to make a request. With signed URLs authentication information is contained in their query string, allowing users without credentials to perform specific actions on a resource.
I would like to point out however that Signed URLs can only be used to access resources in Cloud Storage through XML API endpoints.
Since you are using Swift for your frontend I would also like to direct you to explore Google APIs for iOS, such as CocoaPods.
Okay, I have a working apps that use amazon s3 multipart, they use CreateMultipart, UploadPart and CompleteMultiPart.
Now we are migrating to google cloud storage and we have a problem with multipart. As far as I understood google doesn't support s3 multipart, got info from here Google Cloud Storage support of S3 multipart upload.
So I see that google has closest method Compose https://cloud.google.com/storage/docs/composite-objects, where I just upload different objects and then send request to combine them, or I can use uploadType=multipart https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload#resumable, but this seems to be completely different from s3 multipart. And there is resumable upload https://cloud.google.com/storage/docs/resumable-uploads, that seems to allow upload files in chunks, but without complete multipart.
What is the best option to use? Some services already use CreateMultiPart, UploadPart, CompletePart and I need to write "adapter" for this services in order to make them compatible with google cloud storage.
Update: below answer is no longer correct. GCS does support multipart uploads: https://cloud.google.com/storage/docs/xml-api/post-object-multipart
You are correct. Google Cloud Storage does not currently support multipart upload.
The main benefits to multipart upload are allowing multiple streams to upload in parallel from one or more machines and allowing a partial upload failure not to ruin the whole upload. The best way to get those same benefits with GCS would be to upload the parts as separate objects and then using Compose to combine them into a final object. Indeed, this is exactly what the gsutil command-line utility does when uploading in parallel.
Resumable uploads are a great tool if you want to upload a single object in a single stream, in order, and you want the ability to resume if the connection is lost.
"uploadtype=multipart" uploads are a bit different. They are a way to specify an object's complete metadata and also its data in a single upload operation, using an HTTP multipart request.
Let's say I have a use case where users can buy mp3 files inside an app. The objects are stored in GCP Cloud Storage . What is the best practice to deliver those objects only to the users that purchased the files?
After researching the topic I came up with three solutions:
Client calls a REST (e.g. one running inside App Engine) service. This service downloads the files from Cloud Storage and then sends them back to the client.
Instead of sending the files via the REST call, I could send the download URL (from Cloud Storage) to the client. This would be more cost efficient, however this sounds like a security concern to me as anyone who simply monitors his network could capture the URL.
Creating a (time-limited) signed url to allow the user the download
Obviously a permission check would have to happen first, e.g. a database that contains if user X purchased mp3 Y.
This problem could also be applied to Azure Blob Storage or AWS S3...
In your use case, you have a constant:
You need a backend to authenticate the user (for example Authentication performed with Cloud Identity Platform and hosted on App Engine or Cloud Run
You need to check the list of MP3 that it has bought (stored in Firestore for example)
And then, you need to allow him to download the file. On this last point I recommend you to generated a signedURL. Download URL exists only in Firebase area (maybe your project is a firebase projet?) but it's the same thing than signerURL. Finally I don't recommend you the #1 proposal. It will work, but in case of long download (because network is poor), the connexion will be interrupted after 60 seconds. And this will keep your AppEngine up for nothing (and you will pay for this...).
Let's say we have a chat app and we want users to be able to send images.
Would it be appropriate to convert the image to Base64 in order to send the image through the websocket? A few posts I was reading mentioned this wouldn't be a good use of websockets, so would it be better to have an endpoint for images, and tie that image to the message on upload completion?
Totally new to websockets, so I'm not really sure what to do here.
You can use XMLHttpRequest for file upload and then send the link along with the data through websocket.
Uploading file through websocket will require extra code at the server side to convert the received data to files.
Also check http://binaryjs.com/
BinaryJS is bidrectional realtime binary data with binary websockets
As of today, XMLHttpRequest is the standard most of the companies are using for the file upload, some of the reasons for choosing HTTP are support for caching, gzipping, authentication, firewall, browser compatibility and horizontal scaling.
I am asking this here because Soundcloud does not have support. I going to build a website that people can purchase audio files from using Soundcloud to download the files (and stream before buy). I want to be able to access the download file link in the Soundcloud API without the download link being enabled and showing on the Soundcloud UI. I can't seem to find this info in the Soundcloud API docs. I am going to have a Paypal redirect after the payment to the download link. I know this is a weird way of doing this but I have certain criteria I have to meet. I would host the audio files on my server but they are huge. Anyone have experience with this or can help?
im not sure its possible to do what you want. (very easily at least)
there would be no way for the purchaser to access the 'download' track on soundcloud directly unless downloads are specifically enabled for that track.
really the only way to not host the files and still be able to provide the download would be to use the api to download or proxy the track from soundcloud to your server, using your credentials (because you always have access to your own tracks, download or stream). mind you this would use 2x the bandwidth usage (the server getting the track from soundcloud, and the client downloading the track), and storage space would only be impacked on a temporary bases. but. this is a pretty hacky way and not really a good/proper solution.
you can:
-compress/re-encode the audio as to not use as much disk space
-pay for more storage space at your web host, its usually pretty cheap thse days.
So you want to charge on something free? Well, I think all the downloader out there are middleware where they stream the track from soundcloud and response to client as attachment upon request, one of many examples is http://wittysound.com. Cheapest way to get thing done is providing direct link to soundcloud server like what http://soundflush.com does