I wrote a Windows 8 / JavaScript application that uses indexeddb as its storage mechanism based on this guidance by MS (I have large amount of data).
http://msdn.microsoft.com/en-us/library/windows/apps/hh781225.aspx
I would like now to provide a way for users to synchronize across their Win 8 devices.
What options are available to synchronize the indexeddb data across devices?
IndexedDB is designed for client-side storage scenarios. The API is not designed to handle synchronization to a server-side database. You are going to have to write some code to do that. There are a few of ways that you can accomplish the task; from least fidelity least work to most fidelity most work:
Use the built in Windows 8 app data roaming capabilities. The
pros of this approach are that it is built in functionality. The
cons are that there are limits to how much data you can synchronize
across devices. See Roaming Your App Data for a further
discussion.
You can use a back-end service that already exists like Skydrive
or Azure Mobile Services. The pros here are that the
server-side work is easier and the APIs already exist. The con (I
guess) is that there may be less fidelity than the last option
(below).
You can build your own back end that handles the sync. The pros here
are that this option offers you the most fidelity and control of how
the system works. The con is pretty obvious -- this is the most
work.
Which option you choose depends on your particular needs.
Related
My organization moves data for customers between systems, these integrations are in BizTalk and are done by file, sometimes to/from APIs. More and more customers are switching to APIs so we are facing more and more API to API integrations.
I'm mostly a backend developer but have been tasked with finding out how we can find a more generic pattern or system to make these integrations, we are talking close to a thousand of integrations.
But not thousands of different APIs, many customers use the same sort of systems.
What I want is a solution that:
Fetches data from the source api
Transforms the data to the format for the target api
Sends the data to the target api
Another requirement is that it should be possible to set a schedule when these jobs should run.
This is easily done in BizTalk but as mentioned there will be thousands of integrations and if we need to change something in one of the steps it will be a lot of work.
My vision is something that holds interfaces to all APIs that we communicate with and also contains the scheduled jobs we want to be run between them. Preferrably with logging/tracking.
There must be something out there that does this?
Suggestions?
NOTE: No cloud-based solutions since they are not allowed in our organization.
You can easily implement this using temporal.io open source project. You can code your integrations using a general-purpose programming language. Temporal ensures that the integration runs to completion in the presence of all sorts of intermittent failures. Scheduling is also supported out of the box.
Disclaimer: I'm a founder of the Temporal project.
I am building a service to handle a large number of devices, for a large number of users.
We have a complex schema of access roles assigned to each entity. Some data entries can be written to by certain users, while some users can only read from some entities (but can write to others).
This is a cloud service: there are more devices, and users than can be handled by a single server machine (we are using non relational Cloud databases for this).
I was wondering if there was an established cloud-scale user/role management backend system which I could integrate to enforce the access rules, instead of writing my own. This tech should preferably be cloud agnostic, so I would prefer not to use a SAAS solution, but deploy my own.
I am looking for a system which can scale to millions of users, and billions of data entities
I think authentication is not going to be a big issue, there are very robust cloud based solutions available for storing identities and authenticating millions of users. Authorization will be trickier, and will depend a lot on how granular you want it to be. You could look at Apigee for example as a very scalable proxy that might help you implement this. So getting to the point where you have a token that you can verify the users identity with and that might contain some scopes is not going to be hard imo. If that is enough for you then I would just look at Auth.0, Okta and the native IDM solution of whatever cloud platform you are using (Cognito, Cloud Identity etc.).
I think you will find that more features come with a very hefty pricetag. So Auth.0 is far superior compared to Cognito, but Cognito still has enough features for basic use cases and will end up costing a fraction of Auth.0 in large deployments. So everything comes with pros and cons. If you have very complex requirements such as a bunch of big legacy repositories that you need to integrate then products like Auth.0 rapidly start looking more attractive.
Personally I would look at Auth.0, Cognito and Apigee and my decision would depend massively on parameters that you haven't mentioned in your question. Obviously these are all SaaS solutions, which I think you should definitely be using anyways. I would not host this myself unless I had no other choice, and going that route will radically limit your choices and probably increase expenses. All the cool stuff is happening in the cloud.
I am trying to do something new, something I have never done before. I am looking for advice or point me into right direction how to choose technology. I am trying to build race simulation app that will have thousands of iot devices streaming data into central platform. While I understand that I can use some sort of IOT hub with cloud providers, but what technology do I choose for storing data?
Example is online indoor biking app. There are apps where you can connect your indoor bike online and have simulated race. For my project I am trying to build something similar. Do I use NO SQL db in this scenario? What technology will allow better scale of application like this since it could be millions of devices around the world in "simulated" race. I am not worried about front-end and things like that, but backend, IOT hub, storing data, presenting-real time?
At this point it is important to understand what kind of data your IoT devices will stream, and at what kind of a rate. It will have significant impact on your question.
That it is if it's just location information and some other small data sent lets say once a second, then if you're talking about tens of thousands of devices - this is not a big load of information, and any standard database, like MySQL will be able to deal with it. You will of course need a multi-threaded server(s) capable of handling many requests in parallel.
If your IoT devices will stream HD video, then you're looking at a completely different solution, with a much stronger server, capable of handling allot of streams in parallel, with significant bandwidth requirements from your hosting company, as well as storage space for all the videos. In this case you will store the streams as files (if you'll need them later on), and you won't need any special database either.
In any case, once you'll reach millions of users, you'll be able to scale most modern databases and servers, like MySQL replication capability. For example, take a look how Wikipedia is relying on MySQL: wikipedia - MySQL https://www.mysql.com/why-mysql/case-studies/mysql-cs-wikipedia.html
So I wouldn't be worried regarding the database on this stage, but make sure that the design of my system is in accordance to the the type of data and rate it is streamed.
Hope this gives you a pointer.
Could someone please explain the key differences between JSONStore and CloudantSync (CDTStore) (and as a bonus PouchDB) on the MobileFirst Platform?
I am having difficulty deciding which one would be better to use.
I see that the documentation here states that JSONStore is better if you need FIPS 140-2 compliance, need to synchronize, or are building a hybrid app. But, it looks like you can do all three of those with CDTStore as well. Also, I've seen PouchDB mentioned in some tutorials. I am trying to understand the key differences between what looks like different methods of doing the same thing. Any insight would be greatly appreciated.
The answer to this question is a bit murky as functionality does overlap a bunch, so sorry in advance. I work on the Cloudant team, so am probably unintentionally biased. I'll try to elaborate on the similarities and differences below (as they stand right now), which will hopefully allow you (and others) to see better which libraries aligns with your app's needs.
To set the stage, the similarities are broadly:
Both store JSON data and expose a NoSQL rather than relational interface.
Both use SQLite as an underlying durable datastore.
However, neither expose SQLite to the client -- it's an implementation detail. Sync in particular has a somewhat complicated data model in SQLite to support synchronisation metadata.
Both are designed to securely store data from remote services locally on a device from remote services.
Both offer querying JSON data on device. Both use SQLite's indexing engine under-the-hood.
Sync offers a more flexible and powerful system based on Cloudant Query. Sync offers full text search via SQLite's FTS engine.
Sync also allows creating and dropping indexes at any point, whereas I believe JSONStore requires defining indexes when creating a collection.
Sync
Both Cloudant Sync and JSONStore were designed around the idea of storing data from remote services locally on the device for offline use, or just to make manipulating the data quicker for users.
We originally designed Cloudant Sync (CDTDatastore on iOS, sync-android on Android) with the purpose of synchronising data with a remote Cloudant or CouchDB server. When used with Cloudant/CouchDB, Cloudant Sync supports automatic, incremental synchronisation as local and remote data changes. Cloudant Sync also has a powerful and flexible model, taken from CouchDB, for resolving changes concurrently made on different servers and devices asynchronously. When used together, the Cloudant Sync + Cloudant/CouchDB are a powerful combination, and uses a reliable and long-in-production sync protocol.
As you found, JSONStore, on the other hand, is more agnostic as to what its remote database is. This, however, comes at the cost of sync being a fairly manual process -- importantly, though, JSONStore's data model provides capabilities to significantly help with the process.
The key piece here is that JSONStore can tell you the documents that have changed locally, so you can incrementally upload only changed data. But unless your remote database supports a similar capability of streaming changed data to the client, you have to either send full data snapshots or include the necessary data in your data model to allow for only changed data to be sent (and implement the server-side piece to allow for that to be queried and calculated). JSONStore doesn't support conflict resolution out of the box.
We provide similar client-accessible hooks for working out locally changed data in Cloudant Sync -- these are used by our own sync engine -- but they are in a rawer and less documented form than JSONStore's, so we generally recommend sticking with the better documented JSONStore. If you want to try out Sync's hooks, the docs for the iOS version are here.
In summary, if you're working with Cloudant/CouchDB, I suggest Cloudant Sync. If you're working with another datastore, JSONStore may be easier to get started with.
Security
Both JSONStore and Cloudant Sync support encrypting data on the client. Both use SQLCipher for encrypting JSON data. Additionally, Cloudant Sync can store "attachments" -- small binary blobs associated with JSON data -- which are also encrypted. JSONStore cannot store binary data.
When used alongside MobileFirst's client-side SDKs, JSONStore can be used in a FIPS 140-2 solution on MF's supported platforms. MF ships a self-contained OpenSSL binary for this.
Cloudant Sync uses SQLCipher and CommonCrypto on iOS. SQLCipher can be bought commercially in a FIPS 140-2 version and used with Cloudant Sync. Certain versions of CommonCrypto are FIPS 140-2 validated (see Apple's docs for more). Cloudant Sync for iOS uses exclusively FIPS 140-2 mandated crypto suites where it chooses them for itself.
On Android, Cloudant Sync again uses SQLCipher, with the same FIPS 140-2 version available for developers to integrate. Android Sync uses javax.crypto to encrypt attachments, however, which is usually not provided in a FIPS 140-2 validated version by vendors. Again, the Sync code uses FIPS 140-2 mandated suites (at the time of writing!).
What this means is that the situation is more complicated for Cloudant Sync and we've not yet been able to fully confirm our status w.r.t. FIPS 140-2. However, the actual security of the on-device encryption used in JSONStore and Cloudant Sync is identical.
The summary here is basically that if you don't need FIPS 140-2, go for whichever library is more suitable for your needs or has a function you require. JSONStore is currently recommended for FIPS 140-2 requirements as it's been fully vetted.
Platform support & Hybrid
JSONStore supports several platforms. Cloudant Sync is exclusively iOS and Android.
JSONStore can be used in hybrid applications targeting its supported platforms. Cloudant Sync cannot, yet, without manual wrapping of the library.
JSONStore's actual implementations are native per platform, so again if you are using iOS or Android the choice is mostly a choice based on other factors.
PouchDB
PouchDB can, similarly to Cloudant Sync, sync with Cloudant and CouchDB. As a JS library, it can be easily used in hybrid applications. Unfortunately I'm not that familiar with it, so can't say much more than that with any authority :-/
A couple of notes:
CDTDatastore is used for native app development on iOS (it is an Objective-C library). There's also an Android version available.
JSONStore is available for iOS, Android, JavaScript, and Cordova environments.
PouchDB is a JavaScript library that can run in multiple environments: Node.js, web, hybrid mobile, desktop (e.g. Electron), and even on some IoT devices that can run JavaScript.
The functionality between the three is more or less the same. The biggest differences will be in the APIs, the ability to encrypt data, and in which environments the libraries will run.
I see CDTStore (iOS) and CloudantSync (Android) referring only to native implementation, not to Hybrid (Cordova), so that's one key differentiation that is valid.
I would like to create an app where the user can add and view data. Either adding at a desktop/tablet or phone and reading from either source. I would like the data store to be synced between any of the user's devices.
I'm starting to play with the Trial of Azure, and it looks promising. Probably a solid way to sync data through to cloud between users' devices. Other than syncing between a users devices, I have no need for cloud services currently.
I've seen some apps that do a 'Backup/Restore' model with the user's SkyDrive account. But this seems to be a manual process. I'd like to see it done seamlessly.
I've looked into Sync services, but that would be more like a hub/spoke solution. Again, I don't need a central database.
What are some options? At this point, I would be fine using just Windows 8 patterns/practices.
Because they are separate devices, you will need to have some service layer to do the store/forward for you. With that you have two basic choices, use the end user's own storage (aka SkyDrive) or use your own storage (aka Windows Azure).
SkyDrive is fully supported through the Live SDKs and provides a secure way to allow a user to share store their data, and synchronize it across multiple devices. You get security, and there is no cost for the server side storage on your part. The user owns their storage, not you. The limitation is that you may have issues sharing that same data across other devices or users where SkyDrive (or whatever file sync service you use) is not available.
With a service layer, like Azure, you have a lot more flexibility, but you also will be responsible for maintaining (and paying for) that server side storage / services. Have you looked at "Windows Azure Mobile Services". With your Azure account you get 10 free Azure Mobile Services. You will pay for the SQL data storage on the backend, and that cost will depend on the amount of data you store on the server side. You also need to make sure to architect your application in a way to protect an individual users' data, but it is actually pretty easy to do, well documented, and gives you a lot of options.
Lastly, you may consider what type of data you want to share. SkyDrive is great for "Files". Pics, Songs, Videos, etc. Windows Azure Mobile Services (WAMS) is great for "Data".
Neither model is right or wrong. It just depends on your goals.
Hope that helps you go through the thought process