Good morning or afternoon or evening.
I want to know if I understand Chunked Upload well.
Development environment
browser : chrome
http request library : axios
Restful API : koa js
( Assume that the content-type uses multipart/form-data. )
Separate large files into smaller sizes.
Append a small size to formdata.
Send formdata to the server. ( request-post )
However, I don't have a large files, there are many files, so the capacity is large.
I'm going to briefly explain the logic that I developed with the understanding above. I'd appreciate it if you could check if this is correct.
A large number of files are divided into 500 MB and added to form data.
Each formdata is asynchronously sent to Restful API.
Use busboy to save files to a specific directory.
I've developed it, but I need help because of an error.
I'll write about the situation I came across.
ERR_CONNECTION_RESET
ERR_CONNECTION_ABORT
The above two errors appear in DevTools-Network-waterfall as shown in the image below.
DevTools-Network-waterfall
As far as I understand the chunked upload, there is no problem with logic, but I don't know why this kind of error occurs.
In addition, the following image shows the information displayed in the DevTools console.
DevTools console
I would appreciate it if you could let me know your questions and what you know.
Related
For an app (web + phone) there are two options:
Image binaries in database. Server replies to app HTTP request with images as base64
Images in storage service like Amazon S3 or Azure Blob Storage or a self-hosted one. Image links in database. Server handles app HTTP requests by sending back only the links to images. The app fetches the images from storage by their link
Which option above is the standard practice? Which one has less trouble down the road?
To some extent, the answer to this question is always opinion based, and partly depends on the specific use case.
I would think that the second approach is used more often. One reason is that normally, storage within a database is slightly more expensive than file storage in many cases. Also, what is the real use case? Assuming you use HTML pages that reference images via the img element or via CSS as background image, then the base64 return value would not be that useful, and OTOH the more complicated graphic at the bottom of your picture would get a bit more simple from the client view: The resolution of the link would be resolved by the server when generating the HTML and determine the src of the img, and then the browser would simply apply standard HTML logic and request the image data from the storage service via HTTP.
However, if you would want to optimize load times (and your images would be more or less unique per page so that browser caching of images across pages would not help much), then you could use data URLs embedded into the HTML, and then the first approach could potentially be useful. In this case, all the logic including the generation of the data URL within the HTML would be handled on the server, and the browser would have a single http request.
I recently started building a little bot in VB.NET and got pretty far with it. Though, there are quite some things I'm simply not getting far with.
One of these things is uploading an image.
I use the Telegram API as a DLL which is pretty nice so far.
Though, I struggle with uploading pictures.
The code itself is Telegram.bot.SendPhoto.sendFromFile_id(chat_id, <fileID goes here>).
it is pretty much self explanatory, though I have no idea how I can obtain a fileID. The API itself does not explain how to do it, and I'm not that good in reading other codes from other languages and translate them over into the language I'm using at this moment.
Is there any kind of solution for this I could try to use?
Telegram API Document description about FileID ( Source:Telegram Bot API ) :
There are three ways to send files (photos, stickers, audio, media, etc.):
If the file is already stored somewhere on the Telegram servers, you don't need to reupload it: each file object has a file_id field, simply pass this file_id as a parameter instead of uploading. There are no limits for files sent this way.
Provide Telegram with an HTTP URL for the file to be sent. Telegram will download and send the file. 5 MB max size for photos and 20 MB max for other types of content.
Post the file using multipart/form-data in the usual way that files are uploaded via the browser. 10 MB max size for photos, 50 MB for other files.
I have same problem and solve this way :
1.first upload image to cloud and get image url
2.set URL file_ID
C# Code example :
String ID = bot.chat_id;
bot.SendPhoto.sendFromFile_id(ID, "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png");
I'm writing a fairly involved application for working with Sony cameras.
I can list the contents of the camera and copy image files no problem at all, but I can't seem to figure out the size of the files before I start to download them.
I'm receiving the file list using the standard getContentList API, and finding the files using the originals array in the response. That response seems to have no file size information in it.
Is this possible? Knowing the file size before downloading is important for a good user experience, and all the other camera APIs support it.
I do get the size when I start to download in the HTTP Content-Length header, but performing HEAD requests to hundreds of URLs in a row seems very inefficient!
Unfortunately the API does not support getting the file size.
I want to upload large video file. for that i want to create firefox addon. Is this possible by create firefox addons to upload large files on my server.
or is there any other way to upload large files on server.
please suggest.
If you are POSTing the data to the server as application/x-www-form-urlencoded then you should base64 encode it using btoa() and include it as one of the POST parameters in the request body (i.e. the string passed to XMLHttpRequest.send()):
postbody = "body=" + btoa(fileContents);
xhr.send(postbody);
If you are just downloading the file and uploading it right away, you might as well keep it in memory since you're presumably going to load it into memory anyway in order to base64 encode the contents.
Well if you're reading the file into memory then you should need an nsIFile at all. You can just download it using XMLHttpRequest and use responseText, uploading it in the way I described in the answer. If you do have an nsIFile then yes, that snippet describes how to read from it.
I assume you are wanting to upload via HTTP.
If so, the upload limit is usually decided by the server-side software. This affects both the maximum size and the length of time you have to upload it.
Without a server capable of taking an upload in chunks and reassembling it, you are limited in ways you can't get around through software.
If you want to upload via FTP on the other hand, there are a lot of options... look at FireFTP.
I have made firefox addons for fileupload.
I integrate jquery file upload.
I create widget. In the widget I made panel. In panel I create separate web page for file uploading. And panel is calling that page.
For more information you can mail me at chetansinghal1988#gmail.com
I have created one sample application for demonstrating a working of HTTP live streaming.
What I have done is, I have one library that takes input as video file (avi, mpeg, mov, .ts) and generating segments (.ts) and playlist (.m3u8) files for the given video file. I am storing playlist (as string) in a linked list, as an when i am getting playlist data from the library.
I have written one basic web server which will server the user requested segment and playlist files. I am requesting playlist.m3u8 file from the iPhone safari browser and it is launching the QuickTime player where it is requesting the segment.ts files listed in the received playlist files. after playing every segments (listed in current playlist) it again requests for the playlist, where i am responding with the next playlist file which contains the next set of segment.ts files listed in it.
Is this what we call HTTP live streaming?
Is there anything else, other that this i need to do for implementing HTTP live streaming?
Thanks.
Not much more. If you are taking input streams of media, encoding them, encapsulating in a format suitable for delivery and preparing the encapsulated media for distribution by placing it in such a way that they can be requested from the HTTP server, you are done. The idea behind the live streaming is that it leverages existing Internet architecture that is already optimized for serving HTTP requests for reasonably sized resources.
HTTP streaming renders many existing CDN solutions obsolete with their custom streaming protocols, custom routing and custom content caching.
You can also use media stream validator command line application for mac os x for validating streams generated by the HTTP Web server.
More or less but there's also adaptive bit-rate streaming to take care of if you want your server to push files to iOS devices. Which means your scope expands from having a single "index.m3u8" file that tracks all the TS files to a master index that then tracks the index files for each bitrate you'd want to support in your application which then individually track the TS files encoded at the respective bit-rates.
It's a good amount of work, but mostly routine/repetitive once you've got the hang of the basics.
For more on streaming, your bible, from the iOS standpoint, should ALWAYS be TN2224. Adhering closely to the specs in the Technote, is your best chance of getting through the App Store approval process vis-a-vis streaming.
Some people don't bother (building a streaming app over the past couple of months and looked at the HTTP logs of a whole bunch of video apps that don't quite seem to stick by the rules) - sometimes Apple notices, sometimes they don't, and sometimes the player is just too big for Apple to interfere.
So it's not very different there from every other aspect of the functionality of your app that undergoes Apple's scrutiny. It's just that there are ways you can be sure you're on the right track.
And of course, from a purely technical standpoint, as #psp1 mentioned the mediastreamvalidator tool can help you figure out if your streams are - at their very core, even if not in terms of their overall abilities - compatible with what's expected of HLS implementations.
Note: You can either roll with your own encoding solution (with ffmpeg, the plus being you have more control, the minus being it takes time to configure and get working just RIGHT. Plus once you start talking even the least amount of scale, you run into a whole host of other problems. And once you're done with all the technical hard-work, you'd find that was easy. Now you'd have to actually figure out which license you need to get for having a fancy H.264 encoder with you and jump through all the legal/procedural hoops to get one).
The easier solution for a developer without a legal/accounting team that could fill a football field, IMO, it's easier to go third-party with sites like Encoding.com, Zencoder etc who provide their encoding services a-la-carte or with a monthly fee. The plus is that they've taken care of all the licensing BS and are just providing you a simple "pay to use" service, which could also be extremely useful when you're building a project for a client. The minus is that you're now DEPENDENT on Zencoder/Encoding, the flip-side of which you'd know when your encoding jobs fail for a whole day because their servers are down, or even otherwise, when the API doesn't quite act as you expect or has been documented!
But anyhow that's about all the factors you got to Grok before pushing a HLS server into production!