I sync values from database to local collections as source values and perform operations on these docs subsequently I push updated docs back to the corresponding db table, at success of push I want to pull only those docs from updated database that are not present at local collection of my device.
I could have used sync time stamp but in some peculiar situation either some docs get duplicated or got missing at my device.
I must also bring to notice that the local database also gets augmented while I perform operations at the device.
JSON Store is what you need in this case. More details here -
JSONStore
Related
I have a mininet (v2.2.2) network with openvswitch (v2.5.2), controlled by OpenDaylight Carbon. My application is an OpenDaylight karaf feature.
The application creates a flow (for multicasts) to a group table (type=all) and adds/removes buckets as needed.
To add/remove buckets, I first check if there is an existing group table:
InstanceIdentifier<Group> groupIid = InstanceIdentifier.builder(Nodes.class)
.child(Node.class, new NodeKey(NodId))
.augmentation(FlowCapableNode.class)
.child(Group.class, grpKey)
.build();
ReadOnlyTransaction roTx = dataBroker.newReadOnlyTransaction();
Future<Optional<Group>> futOptGrp = rwTx.read(LogicalDatastoreType.OPERATIONAL, groupIid);
If it doesn't find the group table, it is created (SalGroupService.addGroup()). If it does find the group table, it is updated (SalGroupService.updateGroup()).
The problem is that it takes some time after the RPC call add/updateGroup() to see the changes in the data model. Waiting for the Future<RPCResult<?>> doesn't guarantee that the data model has the same state as the device.
So, how do I read the group table and bucket list from the data model and make sure that I am indeed reading the same state as the current state of the device?
I know that
Add/UpdateGroupInputBuilder has setTransactionUri()
DataBroker gives transaction to read/write
you should use transaction chaining
But I cannot figure out how to combine these.
Thank you
EDIT: Or do I have to use write transactions in stead of RPC calls?
I dropped using RPC calls for writing flows and switched to using writes to the config datastore. It will still take some time to see the changes appear in the actual device and in the operational datastore but that is ok as long as I use the config datastore for both reads and writes.
However, I have to keep in mind that it is not guaranteed that changes to the config datastore will always make it to the actual device. My flows are not that complicated in the sense that conflicts are unlikely to happen. Still, I will probably check consistency between operational and configuration datastore.
Let's assume we have an Aggregate User which has a UserPortraitImage and a Contract as a PDF file. I want to store files in a dedicated document-based store and just hold process-relevant data in the event (with a link to the BLOB data).
But how do I avoid a two-phase commit when I have to store the files and store the new event?
At first I'd store the documents and then the event; if the first transaction fails it doesn't matter, the command failed. If the second transaction fails it also doesn't matter even if we generated some dead files in the store, the command fails; we could even apply a rollback.
But could there be an additional problem?
The next question is how to design the aggregate and the event. If the aggregate only holds a reference to the BLOB storage, what is the process after a SignUp command got called?
SignUpCommand ==> Store documents (UserPortraitImage and Contract) ==> Create new User aggregate with the given BLOB storage references and store it?
Is there a better design which unburdens the aggregate of knowing that BLOB data is saved in another store? And who is responsible for storing BLOB data and forwarding the reference to the aggregate?
Sounds like you are working with something analogous to an AtomPub media-entry/media-link-entry pair. The blob is going into your data store, the meta data gets copied into the aggregate history
But how do I avoid a two-phase commit when I have to store the files and store the new event?
In practice, you probably don't.
That is to say, if the blob store and the aggregate store happen to be the same database, then you can update both in the same transaction. That couples the two stores, and adds some pretty strong constraints to your choice of storage, but it is doable.
Another possibility is that you accept that the two changes that you are making are isolated from one another, and therefore that for some period of time the two stores are not consistent with each other.
In this second case, the saga pattern is what you are looking for, and it is exactly what you describe; you pair the first action with a compensating action to take if the second action fails. So "manual" rollback.
Or not - in a sense, the git object database uses a two phase commit; an object gets copied into the object store, and then the trees get updated, and then the commit... garbage collection comes along later to discard the objects that you don't need.
who is responsible for storing BLOB data and forwarding the reference to the aggregate?
Well, ultimately it is an infrastructure concern; does your model actually need to interact with the document, or is it just carrying a claim check that can be redeemed later?
At first I'd store the documents and then the event; if the first
transaction fails it doesn't matter, the command failed. If the second
transaction fails it also doesn't matter even if we generated some
dead files in the store, the command fails; we could even apply a
rollback. But could there be an additional problem?
Not that I can think of, aside from wasted disk space. That's what I typically do when I want to avoid distributed transactions or when they're not available across the two types of data stores. Oftentimes, one of the two operations is less important and you can afford to let it complete even if the master operation fails later.
Cleaning up botched attempts can be done during exception handling, as an out-of-band process or as part of a Saga as #VoiceOfUnreason explained.
SignUpCommand ==> Store documents (UserPortraitImage and Contract) ==>
Create new User aggregate with the given BLOB storage references and
store it?
Yes. Usually the Application layer component (Command handler in your case) acts as a coordinator betweeen the different data stores and gets back all it needs to know from one store before talking to the other or to the Domain.
I am using iCloud to store sync user preferences between devices. On the device, these are stored in an array of 'Favorite Teams' in NSUserDefaults, and I am using MKiCloudSync to mirror them to the NSUbiquitousKeyValueStore. Changes on one device are propagating to the second device well.
But I am not sure how to prevent the cloud data from being wiped on the first launch after a new install. Here is what's happening:
Device A launches for the first time. App finds nothing in the cloud. User adds multiple items to the array in NSUserDefaults. Changes are synced immediately to cloud.
Device B launches for the first time, but is offline. User adds a single item to the NSUserDefaults array, then remembers the app supports iCloud, so finds some wifi instead.
Device B pushes its version of the defaults to the cloud (with only one item). Device A pulls it, effectively wiping out all of the teams added on Device A.
Is this a limitation of iCloud or is my implementation naive? The docs address a similar issue where a 'highest level' is synced, and adds application logic to never overwrite this value with a smaller value. That's fine when there is some clear business logic to adhere to (higher level is always the one to keep), but when data is more arbitrary, I don't see how I can determine what to do.
Or is it because I am using an array in NSUserDefaults for 'Favorite Teams' and replacing it wholesale? If I used separate keys for each team, perhaps they will be synced independently, based on time code?
Any time you sync a value for a specific key, you run the risk that it will be changed by a different device using the same account. The iCloud service chooses the winning value for you, makes updates, and notifies you when it's done. This is a limitation of iCloud and of your app, and is a simple example of why syncing is hard. What if your step 2 above looked like this:
Device B launches for the first time, and iCloud is available. The app downloads the current data from device A. The user changes their mind and deletes all the data they created on device A. Then they a single new item.
Well, what then? Step 3 still happens exactly as you describe it, except that this time the incoming data is what the user wants. You could refactor your data but the same kind of situation will still be possible.
One option is to keep a non-syncing local copy of the data, so that you can compare incoming changes with the previous local state. What to do when they're different is up to you. Just don't forget that even dramatic changes might well be exactly what the user wants, and not a syncing issue that needs to be fixed. Or, they might be something that would lose data the user wants to keep. Resolving this conflict is your job.
Just after some advice and recommendations here if I may.
I'm creating an iPad app (IOS6) that will write data to the local database on the device and then either straight away or later on replicate that record to a web service (so a Cloud service basically).
What is the best way to go about this you think?
I was thinking of just having a column in the local DB called "synced" and set the flag to '0' right away when the record is created, then sync records with a '0' either right away or during regular intervals. Then obviously set the flag to '1' when each record is replicated.
I want the app to work offline and then sync when the device has an available connection to my web service.
Ideally every record should be replicated right away or seconds later, but in the event of no network connectivity I want to be able to queue the replication to occur.
So what's the best way or achieving this you think? Thanks in advance :)
The solution is going to depend a lot on how complex your total solution is.
For example, if the records are only being created on the local device and then uploaded, without ever being modified, then your solution will be more than adequate.
However, if you allow update of the records on the local device or the records can be updated once they get into your web service, then you need to start managing conflict resolution. The way that we address this situation is to record a timestamp in the "master" database (the one updated by the web service) and synchronize that timestamp when a record is uploaded either as a new record or as an update. When the user updates a record, we send the timestamp and if the value in the database is different than the sent database, the update request is rejected. Of course there are different approaches to this conflict resolution, this is just one that works for our application and users.
Can I share same sqlite database from 2 different iOS devices or update one from another?
Not easily. You can sync the db over iCloud but there is a high chance that data will be overwritten.
I recommend writing some sort of syncing tool that can detect and merge changes originating on either device.
A Common method is to use a timestamp column and get all rows modified since the last sync, then update the other database.
You can take a look at OpenMobster's Sync Service. It supports multi-device sync operations with conflict management.
Besides this it supports the following sync modes:
two-way
one-way device
one-way server
bootup
You can run in complete offline mode and the changes will be auto-tracked and synchronized with the network returns. At this time sync happens with the Cloud and other devices that hold the same data that you are holding locally.
The project is located at: http://openmobster.googlecode.com
You can look at the following iOS sync tutorial to get an idea of how this thing works:
http://code.google.com/p/openmobster/wiki/iPhoneSyncApp