I'm new to the Dropbox API and I'm trying to figure out if I can manage large file (1-2 Gb) updates without having to copy over the whole file.
Something similar to the initial chunked upload. I'd like a way to say, here's a chunk of this file, starting at offset XXXX and here's the new content. And send only 100Kb or 1Mb, but not the whole file over!
I'm surprised nobody needs something like that, since once you upload a large file, which is pretty easy to do with the chunked upload, one would still have an issue if the file needed updating. Especially if the updates are small!
Anyway, all feedback is appreciated!
That's very much incremental sync. There's a free one doing this, by the name of OpenVCDiff. But when it comes to dropbox API, I don't know - I found your post when I was searching for dropbox here within SO.
The Dropbox API doesn't support any sort of incremental update.
Related
We are wanting to implement an offline mode for our react-native application. We will be working with quite large amount of data (aprox. 40-50mb). It is an array of aprox. 16000 objects.
As far as I know, there are two ways to save this data.
Using AsyncStorage - android has a limit of 6mb, but I've read somewhere, that it can be increased.
Using json file - Downloading that data as json file using react-native-background-downloader and then using react-native-fs to save it and load it if the user has no connection to internet.
Personally I think that the second option is better, even though it requires permission to file storage.
Am I missing any other factors to consider? Are there any other options for offline access?
In the end opted out for usage of the json file as there is limit on android. On load of the application I take these data and load them into variable in mobX store. Which functions same as any variable.
I was afraid that mobile phones will have problem sorting across the 16000 objects in array, but there have been no reports of this thing going wrong so far. (In production for 4-5 months right now)
So basically when you hit "enable offline mode" I ask for the file storage permission and download the file using react-native-fs.
Then on the next startup of the application I just read the data off the JSON file.
I'm trying to use GUN to create a File sharing platform. I read the tutorial and API but I couldn't find a general way to upload/download a file.
I hear that there is a limitation of 5Mb of localStorage in GUN, if I want to upload a large file, I have to slice it then storage it into GUN. But right now I can't find a way to storage file into GUN.
I read the question from Retric and I know how to store the image into GUN, but can I store the other type of Files such as .zip or .doc File? Is there a general API for file storage?
I wrote a quick little app in 35 lines of HTML to demonstrates file sharing for images, videos, sound, etc.
https://github.com/amark/gun/blob/master/examples/basic/upload.html
I've sent 20MB files thru it, tho yeah, I'm sure there is a better way of splitting it up into 2MB chunks - that is currently not automatic, you'd have to code it.
We'll have a feature in the future that will automatically split up video files. Do you want to help with this?
I think on the download side, all you have to do is make sure you have the whole file (stitch it back together if you do write a splitter upper), and add it to some <a href=" target. Actually, I'm not sure exactly how, but I know browsers support download file attributes for a few years now, where you can create a download link even of a in-memory file... but you'll have to search online for how. Then please write a tutorial and share it with the community!!
I would recommend using IPFS for file storage and GUN to store the links to those files. GUN isn't meant for file storage I believe, primarily user/graph data. Thus the 5 MB limitation.
I'm relatively new to Objective-C/Cocoa development. I'm currently working on a Mac application where i need to upload a file to a web server using HTTP PUT requests. I'd like to break up the file to several chunks and stream it to the server rather than reading the whole file into the memory and uploading it in one go.
I have come across several third party libraries (ie: ASIHTTPRequest, AFNetworking) which can support this functionality out of the box. However, i'd like to go ahead without using third parties for the time being due to several constraints of the project.
Any assistance in this regard is greatly appreciated. Thanks in advance :)
If you are just uploading a file, without a Multipart MIME wrapper, then I believe you can setup an input stream directly from NSMutableURLRequest. Getting an NSInputStream for a file on-disk is easy, using +[NSInputStream inputStreamWithFileAtPath:]. I've not done this exactly myself, but I think it will work.
If you do end up needing to do a Multipart MIME wrapper, then I'd recommend using a library. It is a total pain to get right, and has some quirks to deal with depending on what OS version you are running on.
So I am in the process of moving all the thumbnails of my major sites to S3 and now I am thinking about how I can consistently put all my CSS/JS/images that power the actual sites to it. It's easy enough to upload everything the first time but I am trying to think of a way to somehow automate the process everytime I push out to production.
Does anyone have any clever ways of doing this?
I used to use s3sync to compare and update the assets just before upload the site files using a bash file to iterate through my files
This works well but when the amount of likes to compare (lets say thousands) gets big this process start being really slow. If you have an small architecture (in term of assets) this would do the trick
to make this better I would recommend capistrano or some other assistant that helps you to deploy...this way you can run at all once..
upload assets
deploy your files
In the other hand you could take a look to cloudfront (amazon's CDN) and set it up using ORIGIN..this way you dont need to worry about upload the files to s3 since they will be automatically pulled on demand. The down side of this approach is the caching if you need to update a file and keep the same name (AKA expire the object)...you can do this in cloudfront but will need an script to do the task.
Depending in the traffic (and other factors, ofcourse) one or other path will fit the best.
I'm building an application that needs to download web content for offline viewing on an iPad. At present I'm loading some web content from the web for test purposes and displaying this with a UIWebView. Implementing that was simple enough. Now I need to make some modifications to support offline content. Eventually that offline content would be downloaded in user selectable bundles.
As I see it I have a number of options but I may have missed some:
Pack content in a ZIP (or other archive) file and unpack the content when it is downloaded to the iPad.
Put the content in a SQLite database. This seems to require some 3rd party libs like FMDB.
Use Core Data. From what I understand this supports a number of storage formats including SQLite.
Use the filesystem and download each required file individually. OK, not really a bundle but maybe this is the best option?
Considerations/Questions:
What are the storage limitations and performance limitations for each of these methods? And is there an overall storage limit per iPad app?
If I'm going to have the user navigate through the downloaded content, what option is easier to code up?
It would seem like spinning up a local web server would be one of the most efficient ways to handle the runtime aspects of displaying the content. Are there any open source examples of this which load from a bundle like options 1-3?
The other side of this is the content creation and it seems like zipping up the content (option 1) is the simplest from this angle. The other options would appear to require creation of tools to support the content creator.
If you have the control over the content, I'd recommend a mix of both the first and the third option. If the content is created by you (like levels, etc) then simply store it on the server, download a zip and store it locally. Use CoreData to store an Index about the things you've downloaded, like the path of the folder it's stored in and it's name/origin/etc, but not the raw data. Databases are not thought to hold massive amounts of raw content, rather to hold structured data. And even if they can -- I'd not do so.
For your considerations:
Disk space is the only limit I know on the iPad. However, databases tend to get slower if they grow too large. If you barely scan though the data, use the file system directly -- may prove faster and cheaper.
The index in CoreData could store all relevant data. You will have very easy and very quick access. Opening a content will load it from the file system, which is quick, cheap and doesn't strain the index.
Why would you do so? Redirect your WebView to a file:// URL will have the same effect, won't it?
Should be answered by now.
If you don't have control then use the same as above but download each file separately, as suggested in option four. after unzipping both cases are basically the same.
Please get back if you have questions.
You could create a xml file for each bundle, containing the path to each file in the bundle, place it in a folder common to each bundle. When downloading, download and parse the xml first and download each ressource one by one. This will spare you the overhead of zipping and unzipping the content. Create a folder for each bundle locally and recreate the folder structure of the bundle there. This way the content will work online and offline without changes.
With a little effort, you could even keep track of file versions by including version numbers in the xml file for each ressource, so if your content has been partially updated only the files with changed version numbers have to be downloaded again.