I completed the upload file part in Workdocs via rest api's.
But in that, how does Workdocs handles large file upload, saw the InitiateDocumentVersionUpload api which doesnot indicate the restrictions on file size if any.
Related
I have a versioning enabled bucket in GCS.
The application I'm working on uses the AWS S3 .NET SDK to connect to a S3-compatible object storage and has been working just fine for this use case.
Now we're also about to support GCS object storage. From my research and testing, GCS offers S3-compatibility through their XML-api. I've tested this and sure enough, GetObject/PutObject/multipart uploads/chunked downloads are all working fine with the code using the S3 library.
However, the compatibility seems to stop when I tried testing the versioning feature: our application makes heavy use of object storage versioned buckets, requesting non-current versions by their VersionID.
With the S3 library connecting to the S3 object storage, everything works fine: PutObject and multipart uploads (= CompleteMultipartUpload response) return the VersionID properly.
For GCS though, this does not return their version of a "VersionID" (= the object Generation).
Response looks like this:
I would have expected that GCS returns this Generation as the VersionID in these responses, since they are conceptually the same. But as you can see, VersionID is null (and the bucket is definitely versioning-enabled).
I would just write another implementation class that uses the GCS .NET SDK, but our application heavily relies on chunked uploading where we retrieve a chunk of data from an external source one by one (so we don't have the full Stream of data). This works well with S3's multipart upload (each chunk is uploaded in a separate UploadPart call), but GCS resumable upload expects a Stream that just has all the data right away. So it looks like we really need multipart upload functionality, that we can use through the S3 library with GCS's XML API. If anyone has suggestions on how to make this work with GCS whereby we can upload chunk per chunk in separate calls to construct an object like multipart upload, would also be greatly appreciated.
So my questions are: will receiving the VersionID after uploading just not work with the AWS S3 SDK to Google Cloud Storage or am I doing it wrong? Do I have to look elsewhere in the response for it? Configure some setting to get this properly returned?
User will upload the videos from browser which need to be stored on server and playback. First understanding from google suggest I need to go for HTTP live streaming(HLS) here.
But I am not sure how it works internally ?
There are three components in above workflow i.e. client/server/data store for saving and retrieving videos.
Save flow :
I believe I need to plugin the HLS client for sending the streaming data.
Does Client in itself divide the file into chunks while sending and maintain the chaining of these chunk where each chunks points to next one ? something like this as I believe server is dumb and will work in same fashion as http upload functionality and no other intelligence is required here ?
But not sure how HLS server side component works here i.e. Will it save as single file or single file is split in to multiple files and then saved on disk ?
I believe it store the file as single file like regular http upload file ?
Retrieval part
In normal regular http file download, client asks for file data and server sends the response back in chunks but all response chunks are sent back against the same request.
I believe in case of HLS , its pull based where client initiate the pull request for each stream request. In each chunk pull request client gets the file name of next chunk and send the request to serverthe relevant chunk from single file for each poll request etc ? So for server its kind of regular http file download request and all intelligence lies with client
Save flow: When you upload a video, it must be converted into HLS format. You can use FFMPEG to do that. You'll end up creating manifest files, and all the segments of the video.
Retrieval part:
The player will read the manifest file to know which segments to request. I've written a post on how HLS playback works with the manifest files: https://api.video/blog/video-trends/what-is-hls-video-streaming-and-how-does-it-work
I am developing a website in HTML, javascript & jQuery. I want to upload (multiple images) to amazon s3 server in an ajax request. There is no such SDK to integrate s3 in Javascript. A PHP SDK is available, but it is not useful to me. Can anybody provide solution to this in javascript?
You can read the article - How to Upload Scanned Images to Amazon S3 Using Dynamic Web TWAIN, which introduces how to use PHP and JavaScript to upload files to Amazon S3. Key steps include:
Specify the bucket which is the place or the folder name used for
storing data on Amazon S3
Specify the Access Key and Secret Key you
obtained from your Amazon S3 account
Create a policy that specifies
what you permit and what you don’t permit for the data uploaded from a
client web page
Encode and encrypt these policies and signatures to
keep them confidential, and store the encoded and encrypted values in
the hidden input elements.
What is the best practice to implement file uploads using Restler framework?
I like to have a API call that get the file save it in CDN and return back the CDN file URL to the caller. What is the best way to implement it?
File upload to CDN using our API
This requires two steps, first is to get the file on the API server.
add UploadFormat to the supported formats
Adjust the static properties of UploadFormat to suit your need
From your api method use $_FILES and move_uploaded_file to get the file to the desired folder. This step is common for any php upload process.
Now that you have the file on the server
Upload it to CDN. You can use any means provided my your CDN. It can be ftp or using some SDK to do the upload
Construct the CDN url and return it to the client
I'm thinking about whether to host uploaded media files (video and audio) on S3 instead of locally. I need to check user's permissions on each download.
So there would be an action like get_file, which first checks the user's permissions and then gets the file from S3 and sends it using send_file to the user.
def get_file
if #user.can_download(params[:file_id])
# first, download the file from S3 and then send it to the user using send_file
end
end
But in this case, the server (unnecessarily) downloads the file first from S3 and then sends it to the user. I thought the use case for S3 was to bypass the Rails/HTTP server stack for reduced load.
Am I thinking this wrong?
PS. I'm using CarrierWave for file uploads. Not sure if that's relevant.
Amazon S3 provides something called RESTful authenticated reads, which are basically timeoutable URLs to otherwise protected content.
CarrierWave provides support for this. Simply declare S3 access policy to authenticated read:
config.s3_access_policy = :authenticated_read
and then model.file.url will automatically generate the RESTful URL.
Typically you'd embed the S3 URL in your page, so that the client's browser fetches the file directly from Amazon. Note however that this exposes the raw unprotected URL. You could name the file with a long hash instead of something predictable, so it's at least not guessable -- but once that URL is exposed, it's essentially open to the Internet. So if you absolutely always need access control on the files, then you'll need to proxy it like you're currently doing. In that case, you may decide it's just better to store the file locally.