Could someone please explain the key differences between JSONStore and CloudantSync (CDTStore) (and as a bonus PouchDB) on the MobileFirst Platform?
I am having difficulty deciding which one would be better to use.
I see that the documentation here states that JSONStore is better if you need FIPS 140-2 compliance, need to synchronize, or are building a hybrid app. But, it looks like you can do all three of those with CDTStore as well. Also, I've seen PouchDB mentioned in some tutorials. I am trying to understand the key differences between what looks like different methods of doing the same thing. Any insight would be greatly appreciated.
The answer to this question is a bit murky as functionality does overlap a bunch, so sorry in advance. I work on the Cloudant team, so am probably unintentionally biased. I'll try to elaborate on the similarities and differences below (as they stand right now), which will hopefully allow you (and others) to see better which libraries aligns with your app's needs.
To set the stage, the similarities are broadly:
Both store JSON data and expose a NoSQL rather than relational interface.
Both use SQLite as an underlying durable datastore.
However, neither expose SQLite to the client -- it's an implementation detail. Sync in particular has a somewhat complicated data model in SQLite to support synchronisation metadata.
Both are designed to securely store data from remote services locally on a device from remote services.
Both offer querying JSON data on device. Both use SQLite's indexing engine under-the-hood.
Sync offers a more flexible and powerful system based on Cloudant Query. Sync offers full text search via SQLite's FTS engine.
Sync also allows creating and dropping indexes at any point, whereas I believe JSONStore requires defining indexes when creating a collection.
Sync
Both Cloudant Sync and JSONStore were designed around the idea of storing data from remote services locally on the device for offline use, or just to make manipulating the data quicker for users.
We originally designed Cloudant Sync (CDTDatastore on iOS, sync-android on Android) with the purpose of synchronising data with a remote Cloudant or CouchDB server. When used with Cloudant/CouchDB, Cloudant Sync supports automatic, incremental synchronisation as local and remote data changes. Cloudant Sync also has a powerful and flexible model, taken from CouchDB, for resolving changes concurrently made on different servers and devices asynchronously. When used together, the Cloudant Sync + Cloudant/CouchDB are a powerful combination, and uses a reliable and long-in-production sync protocol.
As you found, JSONStore, on the other hand, is more agnostic as to what its remote database is. This, however, comes at the cost of sync being a fairly manual process -- importantly, though, JSONStore's data model provides capabilities to significantly help with the process.
The key piece here is that JSONStore can tell you the documents that have changed locally, so you can incrementally upload only changed data. But unless your remote database supports a similar capability of streaming changed data to the client, you have to either send full data snapshots or include the necessary data in your data model to allow for only changed data to be sent (and implement the server-side piece to allow for that to be queried and calculated). JSONStore doesn't support conflict resolution out of the box.
We provide similar client-accessible hooks for working out locally changed data in Cloudant Sync -- these are used by our own sync engine -- but they are in a rawer and less documented form than JSONStore's, so we generally recommend sticking with the better documented JSONStore. If you want to try out Sync's hooks, the docs for the iOS version are here.
In summary, if you're working with Cloudant/CouchDB, I suggest Cloudant Sync. If you're working with another datastore, JSONStore may be easier to get started with.
Security
Both JSONStore and Cloudant Sync support encrypting data on the client. Both use SQLCipher for encrypting JSON data. Additionally, Cloudant Sync can store "attachments" -- small binary blobs associated with JSON data -- which are also encrypted. JSONStore cannot store binary data.
When used alongside MobileFirst's client-side SDKs, JSONStore can be used in a FIPS 140-2 solution on MF's supported platforms. MF ships a self-contained OpenSSL binary for this.
Cloudant Sync uses SQLCipher and CommonCrypto on iOS. SQLCipher can be bought commercially in a FIPS 140-2 version and used with Cloudant Sync. Certain versions of CommonCrypto are FIPS 140-2 validated (see Apple's docs for more). Cloudant Sync for iOS uses exclusively FIPS 140-2 mandated crypto suites where it chooses them for itself.
On Android, Cloudant Sync again uses SQLCipher, with the same FIPS 140-2 version available for developers to integrate. Android Sync uses javax.crypto to encrypt attachments, however, which is usually not provided in a FIPS 140-2 validated version by vendors. Again, the Sync code uses FIPS 140-2 mandated suites (at the time of writing!).
What this means is that the situation is more complicated for Cloudant Sync and we've not yet been able to fully confirm our status w.r.t. FIPS 140-2. However, the actual security of the on-device encryption used in JSONStore and Cloudant Sync is identical.
The summary here is basically that if you don't need FIPS 140-2, go for whichever library is more suitable for your needs or has a function you require. JSONStore is currently recommended for FIPS 140-2 requirements as it's been fully vetted.
Platform support & Hybrid
JSONStore supports several platforms. Cloudant Sync is exclusively iOS and Android.
JSONStore can be used in hybrid applications targeting its supported platforms. Cloudant Sync cannot, yet, without manual wrapping of the library.
JSONStore's actual implementations are native per platform, so again if you are using iOS or Android the choice is mostly a choice based on other factors.
PouchDB
PouchDB can, similarly to Cloudant Sync, sync with Cloudant and CouchDB. As a JS library, it can be easily used in hybrid applications. Unfortunately I'm not that familiar with it, so can't say much more than that with any authority :-/
A couple of notes:
CDTDatastore is used for native app development on iOS (it is an Objective-C library). There's also an Android version available.
JSONStore is available for iOS, Android, JavaScript, and Cordova environments.
PouchDB is a JavaScript library that can run in multiple environments: Node.js, web, hybrid mobile, desktop (e.g. Electron), and even on some IoT devices that can run JavaScript.
The functionality between the three is more or less the same. The biggest differences will be in the APIs, the ability to encrypt data, and in which environments the libraries will run.
I see CDTStore (iOS) and CloudantSync (Android) referring only to native implementation, not to Hybrid (Cordova), so that's one key differentiation that is valid.
Related
I just have a question about how to choose between using a noSQL database or sql database when creating a ReactJS or React Native app? For instance why would you choose to use Firestore over PostgreSQL or vis versa?
Specifically for me, if I were making a react native (mobile) game where players each have their own bag with an inventory where all of this data is stored in a database which method should I use, Firestore or PostgreSQL and why? Additionally, I would want other players to look at each other's bags and rate them.
I had the same question, and wanted a definitive answer (I don't provide that here 😅), especially since some people on hacker news were preferring using generic services which don't lock you into a Cloud provider.
Benefits of PostgresQL
More people are familiar with PostgresQL than Firestore. It is the standard. Data scientists, infrastructure and normal developers will know how it works. Firestore is a niche offering by Google, and only people in the Firebase ecosystem will know what that is. It may be designed to be simple, but don't underestimate it: a few years ago, apps leaked all their sensitive data or allowed anyone to empty out their entire database (That was firebase 😁). Also, people were inefficiently making queries in Firebase and costing their company thousands of dollars with no real benefit. If using Firestore, you need to make sure you deeply understand its paradigm: e.g. how the queries work, and also the security rules, and most people won't.
If one day a new serverless database product is created or open sourced, it might be hard to migrate from Firestore to it, than it would be to migrate from Postgres to the same. You're locked into the Firebase ecosystem.
You don't have to install the firebase SDK into your app, who knows what that library does without watching the network traffic...
Benefits of Firestore
You only pay for reads, writes, deletes and the actual data size stored. Postgres will need to be hosted somewhere, so that will cost you money immediately, even if someone doesn't use your app. Sure, AWS some serverless database offering called Aurora Serverless, but you pay per hour, so even if nobody uses your app, you're paying!
Pricing summary: If nobody uses your app in one month, Firestore is free. AWS Aurora is $60 for a basic machine db.t3.medium. Firestore also has a free tier.
In both cases, you can say you pay what you "use", but in Firestore usage is actual benefit to the user, where as usage in AWS would be the machines youre using.
Integrated with the rest of Firebase
Firebase makes great videos and integrates with other Google stuff well (Android, Flutter), for example, this fun one and this serious one.
You can compare SQL vs NoSQL somewhere else, but I think the differences (data duplication vs normalised data, performance) are not a big deal compared to the other issues here I mentioned (productivity and cost), especially when you're making a basic apps. More complex applications will want a thorough evaluation of different databases, not just differentiating between NoSQL/ SQL.
I am developing an app in Worklight 5.06. My app needs to store around 15,000 documents for the same collection into JSONStore. I found that when I try to retrieve all of them at one time, it took me around 2 minutes to complete this action. Is there any ways to boost the performance?
Please feel free to give any suggestions. Thanks!
I would highly suggest you do the following:
Upgrade to the latest version of Worklight, performance tweaks have been part of new releases.
Read the JSONStore Performance section in the documentation. I linked to the documentation from v6.2, but most, if not all, of the statements translate into previous Worklight versions with JSONStore support.
Read my blog post on Offline Patterns and re-think your offline experience strategy. It may not apply to every application, but when possible you should selectively allow users to pick what content they want available offline. I used two popular applications in the App Store as examples. Spotify allows users to pick certain songs for offline availability, instead of loading all songs for the offline experience. Google Maps allows users to pick certain map regions to be available while lacking network connectivity, instead of loading all map data for offline use.
Update Sept. 7 2014: Read the blog post on JSONStore performance here.
I am creating a cocoa base core data application. I would like to protect the sqlite database, prevent to read it out of the application. How?
You could use cipher algorithms to encrypt the database and decrypt when you use it in your app. CommonCrypto or SecurityTransform may be your choice. Take a look at the Cryptographic Services Guide Apple Dev-Docs.
The needed credentials could be stored securely in the OS X keychain.
So the user could per app start/login decrypt the database and on leave or something, encrypt it.
Another way could be to hardcode the credentials (maybe not a good idea, depends on the security standard you want to use by your app) and do the en-/decrypt on the fly per read/write into the database, so that the database itself is not encrypted but the records in it are. That could be more fault tolerant if your app crashes.
So there is no "right" way to do the task, it depends on what you want to archive and how secure the data has to be.
But what ever you do, don't save any credentials in the NSUserDefaults, that is absolutely insecure.
That would be like to have a secured chest and the key for it lays right on the chest.
For the iOS side there is is the Project iMas-encrypted-core-data on github. It might help you on Cocoa, too.
Aim of the project is to:
Provides a Core Data store that encrypts all data that is persisted. Besides the initial setup, the usage is exactly the same as Core Data and can be used in existing projects that use Core Data.
Under the hood they use SQLCipher and wrap the coreData methods into sql. So you get an encrypted storage but can use coreData syntax for accessing. No need to know about SQL syntax.
The project looks rather promising. It is definitely worth a look.
I wrote a Windows 8 / JavaScript application that uses indexeddb as its storage mechanism based on this guidance by MS (I have large amount of data).
http://msdn.microsoft.com/en-us/library/windows/apps/hh781225.aspx
I would like now to provide a way for users to synchronize across their Win 8 devices.
What options are available to synchronize the indexeddb data across devices?
IndexedDB is designed for client-side storage scenarios. The API is not designed to handle synchronization to a server-side database. You are going to have to write some code to do that. There are a few of ways that you can accomplish the task; from least fidelity least work to most fidelity most work:
Use the built in Windows 8 app data roaming capabilities. The
pros of this approach are that it is built in functionality. The
cons are that there are limits to how much data you can synchronize
across devices. See Roaming Your App Data for a further
discussion.
You can use a back-end service that already exists like Skydrive
or Azure Mobile Services. The pros here are that the
server-side work is easier and the APIs already exist. The con (I
guess) is that there may be less fidelity than the last option
(below).
You can build your own back end that handles the sync. The pros here
are that this option offers you the most fidelity and control of how
the system works. The con is pretty obvious -- this is the most
work.
Which option you choose depends on your particular needs.
I would like to create an app where the user can add and view data. Either adding at a desktop/tablet or phone and reading from either source. I would like the data store to be synced between any of the user's devices.
I'm starting to play with the Trial of Azure, and it looks promising. Probably a solid way to sync data through to cloud between users' devices. Other than syncing between a users devices, I have no need for cloud services currently.
I've seen some apps that do a 'Backup/Restore' model with the user's SkyDrive account. But this seems to be a manual process. I'd like to see it done seamlessly.
I've looked into Sync services, but that would be more like a hub/spoke solution. Again, I don't need a central database.
What are some options? At this point, I would be fine using just Windows 8 patterns/practices.
Because they are separate devices, you will need to have some service layer to do the store/forward for you. With that you have two basic choices, use the end user's own storage (aka SkyDrive) or use your own storage (aka Windows Azure).
SkyDrive is fully supported through the Live SDKs and provides a secure way to allow a user to share store their data, and synchronize it across multiple devices. You get security, and there is no cost for the server side storage on your part. The user owns their storage, not you. The limitation is that you may have issues sharing that same data across other devices or users where SkyDrive (or whatever file sync service you use) is not available.
With a service layer, like Azure, you have a lot more flexibility, but you also will be responsible for maintaining (and paying for) that server side storage / services. Have you looked at "Windows Azure Mobile Services". With your Azure account you get 10 free Azure Mobile Services. You will pay for the SQL data storage on the backend, and that cost will depend on the amount of data you store on the server side. You also need to make sure to architect your application in a way to protect an individual users' data, but it is actually pretty easy to do, well documented, and gives you a lot of options.
Lastly, you may consider what type of data you want to share. SkyDrive is great for "Files". Pics, Songs, Videos, etc. Windows Azure Mobile Services (WAMS) is great for "Data".
Neither model is right or wrong. It just depends on your goals.
Hope that helps you go through the thought process