I have a launchd daemon that every so often uploads some data via a web service using NSOperationQueue.
I need to be able to persist this data, both so that it can later be re-uploaded in the event of failure, even between sessions (in case of computer shut down, for example).
This is not a high load application, it probably receives items intermittently no more than 1 or 2 every minute, often with several hour gaps in between.
My initial implementation without this persistence in place is as follows:
Daemon receives data.
Daemon parses data into an object of type MyDataObject.
Daemon creates instance of NSOperation subclass with MyDataObject as the object to upload and adds it to its NSOperationQueue.
NSOperationQueue goes through and uploads MyDataObject via web service as it is able.
This part all functions just fine. The part I now want to add is the persistence in case of web service failure, computer shut down, etc.
It seems like I could use an NSMutableArray of MyDataObjects along with NSKeyed(Un)archiver containing all the items which had not yet been uploaded and observation of the -isFinished key of all the operations to remove items from the array, but it seems like there should be a simpler way to do is, with less room for things to go wrong, especially as far as thread safety goes.
Can somebody point me in the right direction?
You could add two operations per item. The first would store the item to local storage, and the second would depend on the first and would remove the item from local storage on success.
Then, when you want to restore any items from local storage, you create only the store-to-the-cloud operations, not the store-locally operations. As before, they remove the items from local storage only if they succeed, and if they don't succeed, they leave the items in local storage for the next attempt.
Related
I'm currently working on an app with a reasonably complex Core Data model. The data model currently has 10 tables in it, with a bunch of relationships set between them. The data for the model is obtained piecemeal from a remote server. In order to minimize the amount of traffic to/from the server, the server API passes object ID's first, giving me a chance to discover if I already have stored the objects. If not, then I can ask the server for the full objects and store them. However, those objects can have references to other objects, for which I will need to check follow the same process: check if I have the object(s) and, if not, grab the objects from the server. The Core Data model includes fields for the server IDs which I use to validate and construct Core Data's object graph.
This creates a situation where objects will have been instantiated in Core Data, but won't have been completely constructed as they may be waiting for referenced objects to be returned by the server (which may, in turn, need to wait for their own reference objects).
So my first attempt to deal with this was to create a semaphore that would not allow the object context to be saved (I only save the context in one place) until all objects are downloaded and the object graph is constructed. The problem I ran into was that the context was being saved anyway, without me asking. This results in a ton of changes propagating through NSFetchedResultsController as objects are downloaded from the server and the object graph is being constructed. Moreover, the propagated objects may not be complete.
Has any dealt with anything like this? I think this could all work if I could explicitly control when Core Data saves, but that does not appear to be possible. Or am I missing something?
UPDATE
I was missing something. I was under the impression that NSFetchedResultsController received updates when the Context is saved. This is not true. It receives updates whenever processPendingChanges is called in the context, which occurs at the end of an event cycle. In the past, I've always used two contexts to keep updates separate from the UI, but this project had a deadline and existing code that kept me from refactoring. Given this new information, I think the separate context will fix my problem.
That is an extremely expensive way to sync with a server. Is there a reason your server can't respond to "changed since X" calls and give you everything? In your current design you are spending more time opening and closing sockets than you are receiving data.
Be that as it may, you want to do all of this processing in a secondary context that is connected directly to the NSPersistentStoreCoordinator. When it saves you want to capture the NSManagedObjectContextDidSaveNotification and then have your UI context consume that notification. That will update your UI when your server sync is complete.
This will keep your syncing 100% isolated from the UI and allow the UI to save or do whatever else it needs to do while you are working with the server. I would not use a parent/child design here. There is no reason to.
You access a core data database via the NSManagedObjectContext class.
Each context object must belong to a single thread, and any NSManagedObjects that context creates belong to the same thread.
Do not read or write any managed object from a thread other than the one that created it. If you do, you'll end up with unpredictable and impossible to debug data corruption problems.
However, you can have multiple NSManagedObjectContext instances for a single core data database, each one on a different thread, and you can merge any changes made to the context in one thread over to a context on another thread.
So, basically, you have a "main" NSManagedObjectContext which is used on the main thread, and used for almost all your operations. And then when you need to do something on another thread you create a "child" context for that thread, make all your changes, then merge those changes back to the main context on the main thread.
You can find specific details how to implement this from Apple's official documentation. Start reading here:
https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/CoreData/Articles/cdConcurrency.html#//apple_ref/doc/uid/TP40003385-SW1
I am using iCloud to store sync user preferences between devices. On the device, these are stored in an array of 'Favorite Teams' in NSUserDefaults, and I am using MKiCloudSync to mirror them to the NSUbiquitousKeyValueStore. Changes on one device are propagating to the second device well.
But I am not sure how to prevent the cloud data from being wiped on the first launch after a new install. Here is what's happening:
Device A launches for the first time. App finds nothing in the cloud. User adds multiple items to the array in NSUserDefaults. Changes are synced immediately to cloud.
Device B launches for the first time, but is offline. User adds a single item to the NSUserDefaults array, then remembers the app supports iCloud, so finds some wifi instead.
Device B pushes its version of the defaults to the cloud (with only one item). Device A pulls it, effectively wiping out all of the teams added on Device A.
Is this a limitation of iCloud or is my implementation naive? The docs address a similar issue where a 'highest level' is synced, and adds application logic to never overwrite this value with a smaller value. That's fine when there is some clear business logic to adhere to (higher level is always the one to keep), but when data is more arbitrary, I don't see how I can determine what to do.
Or is it because I am using an array in NSUserDefaults for 'Favorite Teams' and replacing it wholesale? If I used separate keys for each team, perhaps they will be synced independently, based on time code?
Any time you sync a value for a specific key, you run the risk that it will be changed by a different device using the same account. The iCloud service chooses the winning value for you, makes updates, and notifies you when it's done. This is a limitation of iCloud and of your app, and is a simple example of why syncing is hard. What if your step 2 above looked like this:
Device B launches for the first time, and iCloud is available. The app downloads the current data from device A. The user changes their mind and deletes all the data they created on device A. Then they a single new item.
Well, what then? Step 3 still happens exactly as you describe it, except that this time the incoming data is what the user wants. You could refactor your data but the same kind of situation will still be possible.
One option is to keep a non-syncing local copy of the data, so that you can compare incoming changes with the previous local state. What to do when they're different is up to you. Just don't forget that even dramatic changes might well be exactly what the user wants, and not a syncing issue that needs to be fixed. Or, they might be something that would lose data the user wants to keep. Resolving this conflict is your job.
I'm curious how AX 2009 handles code propagation when operating in a load balanced environment.
We have recently converted our AX server infrastructure from a single AOS instance to 3 AOS instances, one of which is a dedicated load balancer (effectively 2 user-facing servers). All share the same application files and database. Since then, we have had one user who has been having trouble receiving code updates made to the system. The changes generally take a few days before they can see it, and the changes don't seem to update all at once.
For example, a value was added to an ENUM field, and they were not able to see it on a form where it was used (though others connected to the same instance were). Now, this user can see the field in the dropdown as expected, but when connected to one of the instances it will not flow onto a report as it should. When connected to the other instance it works fine, and for any other user connected to either instance it works properly.
I'm not certain if this is related to the infrastructure changes, but it does seem odd that only one user is experiencing it. My understanding was that with this setup, code changes would propagate across the servers either immediately (due to sharing the Application Files), or at least in a reasonable amount of time (<1 day). Is this correct or have I been misinformed?
As your cache problems seems to be per user, then go learn about AUC files.
The files are store on the client computer and can be tricky to keep in sync. There are other problems as well.
Start AX by a script, delete the AUC file before starting AX.
There is no cache coherency between AOS instances: import an XPO on one AOS server, and it is not visible on the other. You will either have to flush the cache manually or restart the other AOS. The simplest thing is to import on each server, this is especially true for labels, as this is the only way to bring labels in sync to my knowledge.
I am sort of curious to this as well, but what I do know, is that if a user has access to the AOT (member of admin or a group with developer access), the client will cache AOT-elements more aggressively than if not having developer access.
Elements (like an Enum) might be cached at client level, but also at AOS-level. Restarting the AOS (service) would flush out memory for that service, forcing it to reload elements upon restart.
I guess what I am suggesting is that you make sure the element is not cached client side. Either restart the client, or run the "Refresh AOD" from the developer tools menu. If that doesn't help, try restaring the AOS the client connects to, and see if that helps.
I think it is safe to say, if you want to be absolutely sure every user has the most recent "copy" of any element, you should not develop on the application files shared by all of these services, but rather develop in an environment with 1 AOS. And when you need to move things to production, you need to take down all AOSes in production and move the chances over while the system is down.
In such cases it is often difficult to find the exact cause for a specific case.
I try to follow some best practices to avoid such situations:
- Use separate environment for developing
- Deploy code changes using layer files, not XPOs
- When deploying, stop all AOSs, deploy files, delete index files in the application directory, start one AOSs, compile, sync DB, start other AOS (or even shut down all and start again)
- Try to have latest kernel versions for AOSs and client
I have two processes that are talking to the same persistent store. I save the context on one process, and I post a distributed notification. The other process sees the distributed notification, and fetches its data again, but still receives the old data. Is there some kind of "flushing" I need to do to get the other process to get the correct data from the store?
EDIT: So, it turns out that I was flushing the data correctly. NSManagedObjects have a "refreshObject:mergeChanges" method that you use to do this. The issue appears to be timing related. Let's say I have two processes, A and B. Process A is the main process and does a save to the database. Then Process B does a save to the database and sends a notification to Process A that it has done so, and Process A fetches the new data. I've found that if Process A's save and Process B's save are too close together, the old data is fetched by Process A even if I refresh. If I force there to be some time between the two saves, then it works out correctly.
Obviously this seems like some kind of race condition, where perhaps the notification is getting sent before the data is actually getting saved to the database, however, the notification gets sent in the didSave method of the managed object, at which point the context has already saved.
You should check the merge policy concept, to manage, get and communicate the correct values of a persistent store coordinator between different contexts.
Here -> http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreData/Articles/cdChangeManagement.html#//apple_ref/doc/uid/TP30001201-CJBDBHCB
That should resolve the problem.
Hope this can help.
If I push something onto a list in Redis, then pop from that list, is it guaranteed that I will get the item I pushed earlier or is it possible for the read to happen before the write?
Redis runs in a single thread (with the exception of forking when doing background saves, but that doesn't matter), so any request that you send later will necessarily run later. Thus, you will see the value that you pushed.
(Though, on a second thought, it is probably possible to provoke a failure, if you are ill inclined and dedicated to make it fail on purpose. But that would require sending your requests via separate connections, which doesn't happen accidentially in normal operation.)