Me and a friend are going to be working on an iPhone app that is going to be sending and requesting a lot of information from a 24/7 server (a 2009 Mac Mini). There's going to be a lot of realtime information floating around, and we're not sure what a robust database solution for that info would be. We're essentially looking for a db that's really fast and powerful, but still pretty simple to use, especially since we'll be accessing it from Objective-C, which is still pretty new for us. Is Sqlite what we're looking for? A colleague of ours suggested using PostGres, but isn't PostGres pretty dated?
SQLite is great for the iPhone side. Are you asking about that or the server?
Take a look at the Core Data, Apple's native solution for object graph management and persistence. It uses SQLite internally (though you can set different storage type, including in-memory one), and it is very flexible and optimized.
Related
Can I somehow work on two computers with the same Core Data store? This could, I presume, lead to some incompatibilities during saving. What is the best way to deal with this?
Also, let's say I want to avoid the pain of having to worry about this. How would I make sure that only one computer can work on a particular Core Data store at the same time?
Incidentally, you can work on multiple devices on the same store with one of Apple's own core technologies. It's called iCloud.
Sure, technically speaking there are several copies of the store on the devices as well as the logs in iCloud, but the effect would be the same.
Fortunately, iCloud syncing includes some clever mechanism to merge multiple versions if possible (if not, you have to decide which one to give preference).
Only caveat: in my experience iCloud with Core Data has been far from reliable when using the published information for implementation.
From my own experience with Core Data I do not believe that the framework was designed to be used in a multi user (or distributed) environment. I found this interesting post on CocoaBuilder which might help you shape your thoughts on the matter. It's dated July 2012, so it's pretty recent and also discusses some interesting other technologies that are available.
I am making an iPhone application and Website simultaneously and I want a shared database between the two. I know about some of the options although none of them seem to work perfectly. Like Core Data which will be very nice on the iPhone side, although I havent found a way to access the same information in something other than Obj-c. SQLite is another option, I might not be right, but it supposedly does not work well for servers and being accessed simultaneously from different places. Or I could do XML which seems easy on both sides but it seems like thats the slowest option and will have a huge drag on performance if im going to be reading it from a server all the time.
Any help would be appreciated, and if you know of any other solutions I will be glad to give them a try.
Thanks in advance,
Jordan
Have you thought of using a MySQL backend on your server, and then providing your mobile application with an API?
This takes the hassle out of data synchronisation and also provides you with a good level of sustainability and scalability.
I hope this helps.
Josh.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Okay, I will shortly be starting down the path of windows mobile development. I know nothing about the subject really and I am looking for people with experience to let me know of any gottchas you may know of.
Right now I dont even have a breif of what is requied but the assumption is that the application will be very little more than a bunch of CRUD forms for updating data. The only other requirment knowladge I have is that the application will need to support offline storage when there is no signal avaliable. This in turn will obviously require some kind of syncronization when signal returns.
My initial thoughts are that the application will primarily be a front end to interact with a web service layer. Im assuming that WCF will be an appropriate technology for building these services? I also thought that SQL Server CE would be a good route to go down with regards to the offline storage issues.
Any knowlage that you feel is useful within this domain would be appreciated. Advice, links, books anything appreciated.
EDIT: It has been noted that there are two ways to go with off-line synchronization. To either use some form of message queuing or to use SQL synchronization tools. Could anyone offer a good comparison and introduction to these?
EDIT 2: After a little more digging I get the impression that there are basically 3 different approaches I can use here:
Emmbeded Database to query against then syncronization online, when able
MSMQ along with .NET remoting
WCF with ExchangeWebServiceMailTransport bindings using Exchange Server.
Now, there has been a nice few points raised on the first issue, and I think I understand at some level the issues I would face. But I'd like to get a little more information regarding MSMQ implementations and using WCFs new bindings.
Here a few words from my experience so far (about 9 months) of .net Windows Mobile development.
Well you are occasionally connected. (Or more likely occasionally disconnected). You have to choose whether you are going to use messaging with queues (i.e. WCF/SOAP/XML or something like it) or database synchronisation. I choose the SQL synchronisation route so I can't really comment on messaging. The SQL synchronisation route is not hassle free!
If you go down the sync route with SQL compact like me you basically have two choices. SQL Server merge replication or the newer ADO.NET Synchronisation services. If you choose the former you need to be really careful with your DB design to ensure it can be easily partitioned between mobile subscribers and the publisher. You really need to think about conflicts, and splitting tables that wouldn't normally be split in a normalised DB design is one way of doing that. You have to consider situations where a device goes offline for some time and the publisher DB (i.e. main DB) and/or a subscriber alters the same data. What happens when the device comes back online? It might mean resolving conflicts even if you have partitioned things well. This is where I got burnt. But SQL Merge Replication can work well and reduces the amount of code you have to write.
Roll your own DAL. Don't attempt to use datareaders etc. directly from UI code and don't use typed datasets either. There may be third party DALs that work with Windows Mobile (i.e. I know LLBLGEN does, might be worth a look) but Linq-to-SQL is not supported and anyway you need something lightweight. The chances are the DAL won't be too big so roll it yourself.
If you are using .net you'll probably end up wanting some unimplemented platform features. I recommend using this inexpensive framework to give you what your missing (especially as related to connectivity and power management) - http://www.opennetcf.com/Products/SmartDeviceFramework/tabid/65/Default.aspx
Windows Mobile devices partially switch off to save power when not in use. If you are doing a polling type design you'll need to wake them up every x mins. A normal .net timer class won't do this. You'll need to use a platform feature which can be used from OpenNetCF (above). The timer class is called LargeIntervalTimer and is in the OpenNetCF.WindowsCE assembly/namespace (I think).
Good Luck!
SqlCE is only one of the options available for local data storage on a Windows Mobile device, and although it's an excellent database it has limitations. For one thing, SqlCE will not work (period) under encryption (in other words, if your user encrypts the location where your SDF file is, you will no longer be able to access the data).
The second (and most critical) weakness of SqlCE lies in the RDA/Merge Replication tools. SqlCE Merge Replication is not 100% reliable in situations where the network connection can drop during replication (obviously very common in Windows Mobile devices). If you enjoy trying to explain missing or corrupted data to your clients, go ahead and use SqlCE and merge replication.
Oracle Lite is a good alternative to SqlCE, although it too doesn't work properly under encryption. If encryption is a potential problem, you need to find a database engine that works under encryption (I don't know of one) or else write your own persistence component using XML or something.
Writing a WM application as a front end that primarily interacts with a web service in real time will only work in an always-connected environment. A better approach is to write your application as a front end that primarily interacts with local data (SqlCE, Oracle Lite, XML or whatever), and then create a separate Synchronization component that handles pushing and pulling data.
Again, SqlCE merge replication does this pushing and pulling beautifully and elegantly - it just doesn't work all the time. If you want a replication mechanism that works reliably, you'll have to write your own. Oracle Lite has something called a snapshot table that works very well for this purpose. A snapshot table in Olite tracks changes (like adds, updates and deletes) and allows you to query the changes separately and update the central database (through a web service) to match.
This thread I just posted on SO a few days ago has proven to be a great resource for me thus far.
Also the Windows Mobile MSDN WebCasts are a wealth of information on everything from just getting started up to advanced development.
I would suggest Sqlite for local storage. From the last benchmark I ran it was much better than SqlCe and you don't have to do stupid things like retain an open connection for performance improvements.
Trade-offs being that the toolset is less rich and the integration with other MSSql products is nil. :(
you might want to refer to this:
getting-started-with-windows-mobile-development
You shouldn't be intimidated for windows mobile development. It isn't much different from desktop development. I strongly recommend that you use .NET Compact Framework for development and not C++/MFC.
Some useful links:
Mobile section at the Code
Project. You would find a lot of
articles, a little digging is needed
to find the appropriate one.
Smart
Device Framework from OpenNetCF
offer valuable extensions to the
compact framework.
When you install
the Mobile SDK, you will find under the
Community folder links for the
Windows Mobile and CF framework
blogs. These are also valuable
resources.
Regarding your application, you are right about the WCF and the SQL Server CE. These are the proper ways for handling communication and storage.
Some hints for people coming from a desktop world:
You need to have some sort of power management. The device may automatically go to suspend state. Also, you shouldn't consume power when you don't have to.
Network connectivity is a difficult issue. You can register notifications for when a specific network (Wi-Fi, GPRS) becomes available or unavailable. You can also set the preferred means of communication.
Make the UI as simple as possible. The user uses his thumb and/or a pen and he is probably on the move.
Test in a real device as early as possible.
"24 Hours of Windows Mobile Application Development" from the Windows Mobile Team Blog has some good resources
If you can, try to start from the user use cases and work back to the code, rather than vice versa. It's really easy to spend a lot more time working on the tools than working on the business problem. And thinking through user requirements will help you consider alternate strategies, because a lot of the patterns you know from normal .NET don't apply.
I've done lots of intermittent application development of exactly the type you are describing, and an on-board database works just fine. The MSMQ/WCF stuff just adds conceptual overhead without adding much value. You need a logical datastore locally anyway, and replication at this level is a simple concept that you want to keep simple, so the audit trail is easily monitored and debugged. MSMQ and WCF tend to hide things in unfamiliar places.
I upvoted the SqlLite suggestion BTW. MS doesn't have their persistence story stabilized yet for CE.
For the database replication bit I highly recommend Sybase Ultralite. In terms of flexibility and performance it knocks the socks off SQL CE
I had to do this once. Weird setup with Macs for development, and we were all Java programmers. And a short deadline. PowerPC macs too, so no chance to install Windows for Visual Studio development, never mind that the money for this would never have appeared.
We ended up writing applications using Java, running on the IBM J9 virtual machine, with SWT for a user interface. Entirely free development stack. Easy to deploy. Code ran on any platform we desired, not just PocketPC/WinMob.
Most of the work was on the server side anyway - the database, the web service server. The logic. The reporting engine. The client side wasn't totally simple however - would get the form templates from the server (because they changed frequently), the site details (multi-site deployment), generate a UI from the form template (using some SWT GUI components that are wonderful for PocketPC development, like the ExpandBar), gather data with a point and click interface (minimising keyboard entry where possible), and then submit it back to the server.
For offline storage we used XML files on the device itself. More than enough for our needs, but yours may differ. Maybe consider SQLite?
There are a couple links you can check out to start with:
http://developer.windowsmobile.com
http://msdn.microsoft.com/en-us/windowsmobile/default.aspx
If you have a sticking point while developing, there are also Windows Mobile dedicated chats on MSDN that you can attend and ask your questions. The calendar hasn't been updated yet, but the next ones should be in January. You can find the schedule here: http://msdn.microsoft.com/en-us/chats/default.aspx
I am going to add an additional question to this post, as its been active enough and hopefully will be helpful to others as well as me. Ok, so after playing around I now realize that standard class libraries cannot be included in windows mobile applications.
Now the overwhelming advice here seems to be use an embedded database, though I now do have use cases and it appears that I will need to have document synchronization as well as relational data. With this in mind service layer interaction seems inevitable. So my question is how would I share common domain objects and interfaces between the layers?
"Document synchronization" - does that mean bidirectional? Or cumulative write-only? I can think of mobile architectures that would mainly collect and submit transactions for a shared document - if that's your requirement, then we should discuss offline - it's a long (and interesting) conversation.
Owen you can share code from Compact Framework -> Desktop, it's only Desktop -> Compact Framework that has compatability issues if you use certain objects that are not supported by the CF.
While a desktop lib doesn't work on CF a CF lib WILL work on the desktop, you can also run CF.exes on the desktop!
Just create a CF library as the project that defines your base objects / interfaces etc.
This book sshould e essential reading for all Windows Mobile developers: http://www.microsoft.com/learning/en/us/books/10294.aspx
For developing windows mobile applications you must have the basic tools like silverlight, visual studio, windows phone emulator and sqlite as your database storage.
I am developing a cocoa application that will be making heavy use of both web services and a standard dbms (most likely MySQL) and I am wondering if anyone has a good option for a database library or ORM solution they have used. CoreData is not an option due to the need to support a standard DBMS and to be able to modify the data outside of the normal application operation.
I have found a number of possible options from new open source libraries:
http://github.com/aptiva/activerecord/tree/master
To writing my own wrapper for the C MySQL api.
Any advice is welcome,
Thanks!
Paul
We faced a similar question when we first started work on Checkout, our solution was to code the entire app in Python, using PyObjC. Checkout 1 had an sqlite backend, Checkout 2 has a postgres backend.
There are a couple of really mature and powerful ORMs on the Pyton side, such as SQLObject, which is pretty simple to work with (we used it for Checkout 1.0) and SQLAlchemy, which is more powerful but a bit harder to wrap your brain around (we used it for Checkout 2.0).
One approach you could evaluate, is building the application in Objective-C, but writing the data model and database connectivity/adminstration code in Python. You can use PyObjC to create a plugin bundle from this code, that you then load into your app That's more or less the approach we took for Checkout Server, which uses a Foundation command-line tool to administer a postgres server and the databases in it, this CLI tool in turn loads in a Python plugin bundle that has all of the actual database code in it. End-users mostly interact with the database through a System Preferences pane, that has no clue what the database looks like, but instead uses the command-line tool to interact with it.
Loading a plugin is simple:
NSBundle *pluginBundle = [NSBundle bundleWithPath:pluginPath];
[pluginBundle load];
You will probably need to create .h files for the classes in your bundle that you want to have access to from your Obj-C code.
You might also want to check out the BaseTen framework. It is a Core Data-like framework (in fact, it can import Core Data models), but works with PostgreSQL (though not MySQL, as far as I know). It includes some very nice features such as schema discovery at run time. It also includes an NSArrayController subclass that automatically handles locking and synchronizing across multiple users, so you can continue to make use of Apples Key-value Binding in your UI.
I have personal experience with this particular problem. I even started down the road of writing my own wrapper for the C MySQL API.
The eventual conclusion was: Don't!
The solution that worked in my case was to communicate with the MySQL server via PHP. If you are familiar with web services, chances are that you know about PHP, so I don't won't go into loads of detail about that.
To read from the database:
The cocoa app sends a request for a URL on the server: http://theserver.com/app/get_values.php
The get_values.php script handles the database query, and returns the data in xml format
The cocoa app loads and parses the xml
To write to the database:
The cocoa app sends a more complex request to the server: http://theserver.com/app/put_values.php?name="john doe"&age=21&address=...
The put_values.php script parses the input and writes to the database
The beauty of this solution is that PHP is great for working with MySQL, and cocoa has some handy built-in classes for working with XML data.
edit: one more thing:
One of the key things you have to figure out with this approach is how much processing should be done on the server, and how much should be done in the app itself. Let cocoa do the things that cocoa is good at, and let PHP and MySQL do the things that they are good at.
You could write a generic PHP script to handle all queries: perform_query.php?querystring="SELECT * FROM .....", but that is hardly an optimal solution. Your best bet is several smaller PHP scripts that handle individual datasets for you. In my case, there was one to get the list of users, one to get the list of transactions, etc. Again, it all depends on what your app is going to do.
GDL2 is a nice example, based on EOF.
Instead of reinventing the wheel by writing your own communication wrapper to deal with MySQL from Cocoa, you could try the SMySQL framework (a.k.a. MCPKit), it was part of the CocoaMySQL application that evolved into the Sequel Pro project. It works with varying versions of MySQL, and seems to be quite robust.
If you need to understand how to incorporate it into your application, there's not much documentation around, but it has an easy to understand interface and you can see it working by looking at the source of Sequel Pro, which is downloadable from Google code.
There is also the CocoaMySQL-SBG fork of the CocoaMySQL project, but that seems to be out of date and I couldn't get it to build properly.
I've also implemented a simple object persistence framework based on sqlite, but it certainly wasn't trivial to do. I agree with eJames' conclusion- don't implement one yourself if you don't have to.
If you aren't committed to programming in Objective-C you might want to take a look at PyObjC which would allow you to program the database portion in Python. You can use the MySQLdb module for DB access and there are plenty of tutorials online for its use. It isn't hard to stuff the data back into Cocoa/CF classes and pass them back to your app.
The main caveat with PyObjC is that at the moment it doesn't work with Tiger.
Has anyone considered using something along the lines of the Amazon SimpleDB data store as their backend database?
SQL Server hosting (at least in the UK) is expensive so could something like this along with cloud file storage (S3) be used for building apps that could grow with your application.
Great in theory but would anyone consider using it. In fact is anyone actually using it now for real production software as I would love to read your comments.
This is a good analysis of Amazon services from Dare.
S3 handled what I've typically heard described as "blob storage". A typical Web application typically has media files and other resources (images, CSS stylesheets, scripts, video files, etc) that is simply accessed by name/path. However a lot of these resources also have metadata (e.g. a video file on YouTube has metadata about it's rating, who uploaded it, number of views, etc) which need to be stored as well. This need for queryable, schematized storage is where SimpleDB comes in. EC2 provides a virtual server that can be used for computation complete with a local file system instance which isn't persistent if the virtual server goes down for any reason. With SimpleDB and S3 you have the building blocks to build a large class of "Web 2.0" style applications when you throw in the computational capabilities provided by EC2.
However neither S3 nor SimpleDB provides a solution for a developer who simply wants the typical LAMP or WISC developer experience of building a database driven Web application or for applications that may have custom storage needs that don't fit neatly into the buckets of blob storage or schematized storage. Without access to a persistent filesystem, developers on Amazon's cloud computing platform have had to come up with sophisticated solutions involving backing data up manually from EC2 to S3 to get the desired experience.
I just finished writing a library to make porting an app to simpledb in Perl easy, Net::Amazon::SimpleDB::Simple because I found the Amazon client libraries painful. The library isn't on CPAN yet, but it is at http://rjurneyopen.s3.amazonaws.com/SimpleDB/Simple.pm The idea was to make it trivial to stuff hashes in and out of SimpleDB.
I just ported an app to use it. Overall I am impressed with SimpleDB... even inefficient queries take only 2-3 seconds to return. SimpleDB doesn't seem to care about the size of your table, owing to its Erlang/parallel nature. Tablescans are easy for it.
The pain comes from the fact that you can't count, sum or group by. If you plan on doing any of those things... then SimpleDB probably isn't for you. At the moment in terms of functionality it exists somewhere in between memcached and MySQL. You can SELECT ORDER BY LIMIT, which is nice. Its also nice that you don't have to scale it yourself, and its nice that it doesn't care how much you stuff into it. But more advanced operations like analytics are painful at best. You'll have to do your own calculations server side. Its also a big plus that on any computer I can use the simpledb CLI http://code.google.com/p/amazon-simpledb-cli/ to query my data.
There are some confusing 'gotchas.' For instance, attributes can have more than one value, and you have to explicitly set 'replace' when storing items. Also, storing undef or null string results in a library error, instead of deleting that attribute name/value pair or setting it null/empty string.
Learning to think in terms of a largely un-normalized way is a little strange too, which is why I would second the suggestion above that says it is best for new applications. Porting from a SQL app to SimpleDB would be painful because your application logic would have to change. The way you do things is a bit different. The amazon docs are pretty good at explaining this.
All of this is extractable in a library that sits atop SimpleDB, so for your use of SimpleDB you will want to pick a good library... you probably don't want to deal with it directly. There is some work on the PHP side to make things easy, and there is my library. There is a RAILS activesource, but it doesn't seem to do much for you.
All in all its still early in the game, but compared to other APIs (twitter comes to mind), I have to say that the SimpleDB REST API is pretty simple (especially considering that it is XML) and polite to work with. I would recommend it... depending on the requirements of your application and the economics of your use of it. If you're looking to rapidly scale a service that doesn't put a great load on the DB and don't want to bother with a scalable MySQL/memcache combo... then SimpleDB can offer a 'simple' solution for you.
I expect that its features will continue to grow and it will be a good choice for more and more applications that do more complex and interesting things. But right now it is targeted at and appropriate for your typical Web 2.0 service.
We are using SimpleDB almost exclusively for our new projects. The zero maintenance, high availability, no install aspects are just too good. And for your Ruby developers, check out SimpleRecord, an ActiveRecord like interface for SimpleDB which makes it super easy to use.
But do you really need SQL Server? Can't you live with PostgreSQL or MySQL? Both have proven to be ok for most tasks.
Now if you need SQL Server features then you're out of luck.
Another option is to rent a server. How expensive is expensive?
(I've used Amazon S3 to store images for an application, it's ok and works fine, at least for that)
I haven't used SimpleDB, but have been using combination of S3, EC2, and MySQL for our application.
As long as you are willing to use SimpleDB, then you might as well consider using MySQL (which is very scalable, and not that expensive).
On the S3 and EC2 side, it is great in practice as well.
SimpleDB works great for many applications.... if your project will require a lot of analytic reporting, joining, etc, you may consider MySQL or a hybrid-model.
If you go SimpleDB, we've developed Radquery.com for our internal use and opened it up to the public.