Related
I am a composer by profession and my computer science skills are limited though I program quite a bit of the software that I use.
What are the most reasonable ways to approach SQLite integration as a file format and database in an iOS app (it also needs to run on windows, but that is a secondary concern)?
I have been researching Hiberlite, which looks fantastic, but it seems to be little used and apparently it doesn't run well on embedded systems (iOS?) and chokes up when thousands of objects are in play. I haven't been able to get a sense of how severe those bottle necks are when running under those conditions.
The settings of thousands of objects (~50,000 though that number could expand) would be read every 1-10 seconds and written periodically. Read performance is more critical as write operations can stutter with out effecting the core operation of the app.
Given those conditions, how should I approach SQLite? My understanding is that without something like Hiberlite the entire database (many millions of entries) must be read and rewritten for every entry, is that less efficient. If that is the best approach is there a good resource to follow for implementing it?
Any advice would be greatly appreciated. My current software that I rely on is beyond buggy and needs refactoring, but due to my inexperience I am having a difficult time finding information about a reasonable approach.
I'm guessing you've probably found a solution for this by now, but I've been interested myself in embedding SQLite on Android and IOS, and I came across many C++-based ORM solutions.
Hiberlite looked possibly not fully mature (I didn't readily see a method of returning subsets of data, which is fairly standard). A framework which did draw my attention was the POCO:Data ORM library. It's based on the stream-based mechanism used in SOCI ORM. The POCO library is modular and optimised for embedded environments (I believe it also has a minimal external dependencies). Wikipedia has an article here, they outline some of its users, of which OpenFrameworks is one.
The WT ORM also looked pretty interesting.
I'm listing some of the other C++ ORM frameworks I found here, in no particular order:
http://soci.sourceforge.net
webtoolkit WT DBO ORM
http://debea.net
http://www.qxorm.com
http://sourceforge.net/apps/trac/litesql
http://otl.sourceforge.net
http://cppcms.com/sql/cppdb
http://dtemplatelib.sourceforge.net
http://code.google.com/p/qdjango
Me and a friend are going to be working on an iPhone app that is going to be sending and requesting a lot of information from a 24/7 server (a 2009 Mac Mini). There's going to be a lot of realtime information floating around, and we're not sure what a robust database solution for that info would be. We're essentially looking for a db that's really fast and powerful, but still pretty simple to use, especially since we'll be accessing it from Objective-C, which is still pretty new for us. Is Sqlite what we're looking for? A colleague of ours suggested using PostGres, but isn't PostGres pretty dated?
SQLite is great for the iPhone side. Are you asking about that or the server?
Take a look at the Core Data, Apple's native solution for object graph management and persistence. It uses SQLite internally (though you can set different storage type, including in-memory one), and it is very flexible and optimized.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Okay, I will shortly be starting down the path of windows mobile development. I know nothing about the subject really and I am looking for people with experience to let me know of any gottchas you may know of.
Right now I dont even have a breif of what is requied but the assumption is that the application will be very little more than a bunch of CRUD forms for updating data. The only other requirment knowladge I have is that the application will need to support offline storage when there is no signal avaliable. This in turn will obviously require some kind of syncronization when signal returns.
My initial thoughts are that the application will primarily be a front end to interact with a web service layer. Im assuming that WCF will be an appropriate technology for building these services? I also thought that SQL Server CE would be a good route to go down with regards to the offline storage issues.
Any knowlage that you feel is useful within this domain would be appreciated. Advice, links, books anything appreciated.
EDIT: It has been noted that there are two ways to go with off-line synchronization. To either use some form of message queuing or to use SQL synchronization tools. Could anyone offer a good comparison and introduction to these?
EDIT 2: After a little more digging I get the impression that there are basically 3 different approaches I can use here:
Emmbeded Database to query against then syncronization online, when able
MSMQ along with .NET remoting
WCF with ExchangeWebServiceMailTransport bindings using Exchange Server.
Now, there has been a nice few points raised on the first issue, and I think I understand at some level the issues I would face. But I'd like to get a little more information regarding MSMQ implementations and using WCFs new bindings.
Here a few words from my experience so far (about 9 months) of .net Windows Mobile development.
Well you are occasionally connected. (Or more likely occasionally disconnected). You have to choose whether you are going to use messaging with queues (i.e. WCF/SOAP/XML or something like it) or database synchronisation. I choose the SQL synchronisation route so I can't really comment on messaging. The SQL synchronisation route is not hassle free!
If you go down the sync route with SQL compact like me you basically have two choices. SQL Server merge replication or the newer ADO.NET Synchronisation services. If you choose the former you need to be really careful with your DB design to ensure it can be easily partitioned between mobile subscribers and the publisher. You really need to think about conflicts, and splitting tables that wouldn't normally be split in a normalised DB design is one way of doing that. You have to consider situations where a device goes offline for some time and the publisher DB (i.e. main DB) and/or a subscriber alters the same data. What happens when the device comes back online? It might mean resolving conflicts even if you have partitioned things well. This is where I got burnt. But SQL Merge Replication can work well and reduces the amount of code you have to write.
Roll your own DAL. Don't attempt to use datareaders etc. directly from UI code and don't use typed datasets either. There may be third party DALs that work with Windows Mobile (i.e. I know LLBLGEN does, might be worth a look) but Linq-to-SQL is not supported and anyway you need something lightweight. The chances are the DAL won't be too big so roll it yourself.
If you are using .net you'll probably end up wanting some unimplemented platform features. I recommend using this inexpensive framework to give you what your missing (especially as related to connectivity and power management) - http://www.opennetcf.com/Products/SmartDeviceFramework/tabid/65/Default.aspx
Windows Mobile devices partially switch off to save power when not in use. If you are doing a polling type design you'll need to wake them up every x mins. A normal .net timer class won't do this. You'll need to use a platform feature which can be used from OpenNetCF (above). The timer class is called LargeIntervalTimer and is in the OpenNetCF.WindowsCE assembly/namespace (I think).
Good Luck!
SqlCE is only one of the options available for local data storage on a Windows Mobile device, and although it's an excellent database it has limitations. For one thing, SqlCE will not work (period) under encryption (in other words, if your user encrypts the location where your SDF file is, you will no longer be able to access the data).
The second (and most critical) weakness of SqlCE lies in the RDA/Merge Replication tools. SqlCE Merge Replication is not 100% reliable in situations where the network connection can drop during replication (obviously very common in Windows Mobile devices). If you enjoy trying to explain missing or corrupted data to your clients, go ahead and use SqlCE and merge replication.
Oracle Lite is a good alternative to SqlCE, although it too doesn't work properly under encryption. If encryption is a potential problem, you need to find a database engine that works under encryption (I don't know of one) or else write your own persistence component using XML or something.
Writing a WM application as a front end that primarily interacts with a web service in real time will only work in an always-connected environment. A better approach is to write your application as a front end that primarily interacts with local data (SqlCE, Oracle Lite, XML or whatever), and then create a separate Synchronization component that handles pushing and pulling data.
Again, SqlCE merge replication does this pushing and pulling beautifully and elegantly - it just doesn't work all the time. If you want a replication mechanism that works reliably, you'll have to write your own. Oracle Lite has something called a snapshot table that works very well for this purpose. A snapshot table in Olite tracks changes (like adds, updates and deletes) and allows you to query the changes separately and update the central database (through a web service) to match.
This thread I just posted on SO a few days ago has proven to be a great resource for me thus far.
Also the Windows Mobile MSDN WebCasts are a wealth of information on everything from just getting started up to advanced development.
I would suggest Sqlite for local storage. From the last benchmark I ran it was much better than SqlCe and you don't have to do stupid things like retain an open connection for performance improvements.
Trade-offs being that the toolset is less rich and the integration with other MSSql products is nil. :(
you might want to refer to this:
getting-started-with-windows-mobile-development
You shouldn't be intimidated for windows mobile development. It isn't much different from desktop development. I strongly recommend that you use .NET Compact Framework for development and not C++/MFC.
Some useful links:
Mobile section at the Code
Project. You would find a lot of
articles, a little digging is needed
to find the appropriate one.
Smart
Device Framework from OpenNetCF
offer valuable extensions to the
compact framework.
When you install
the Mobile SDK, you will find under the
Community folder links for the
Windows Mobile and CF framework
blogs. These are also valuable
resources.
Regarding your application, you are right about the WCF and the SQL Server CE. These are the proper ways for handling communication and storage.
Some hints for people coming from a desktop world:
You need to have some sort of power management. The device may automatically go to suspend state. Also, you shouldn't consume power when you don't have to.
Network connectivity is a difficult issue. You can register notifications for when a specific network (Wi-Fi, GPRS) becomes available or unavailable. You can also set the preferred means of communication.
Make the UI as simple as possible. The user uses his thumb and/or a pen and he is probably on the move.
Test in a real device as early as possible.
"24 Hours of Windows Mobile Application Development" from the Windows Mobile Team Blog has some good resources
If you can, try to start from the user use cases and work back to the code, rather than vice versa. It's really easy to spend a lot more time working on the tools than working on the business problem. And thinking through user requirements will help you consider alternate strategies, because a lot of the patterns you know from normal .NET don't apply.
I've done lots of intermittent application development of exactly the type you are describing, and an on-board database works just fine. The MSMQ/WCF stuff just adds conceptual overhead without adding much value. You need a logical datastore locally anyway, and replication at this level is a simple concept that you want to keep simple, so the audit trail is easily monitored and debugged. MSMQ and WCF tend to hide things in unfamiliar places.
I upvoted the SqlLite suggestion BTW. MS doesn't have their persistence story stabilized yet for CE.
For the database replication bit I highly recommend Sybase Ultralite. In terms of flexibility and performance it knocks the socks off SQL CE
I had to do this once. Weird setup with Macs for development, and we were all Java programmers. And a short deadline. PowerPC macs too, so no chance to install Windows for Visual Studio development, never mind that the money for this would never have appeared.
We ended up writing applications using Java, running on the IBM J9 virtual machine, with SWT for a user interface. Entirely free development stack. Easy to deploy. Code ran on any platform we desired, not just PocketPC/WinMob.
Most of the work was on the server side anyway - the database, the web service server. The logic. The reporting engine. The client side wasn't totally simple however - would get the form templates from the server (because they changed frequently), the site details (multi-site deployment), generate a UI from the form template (using some SWT GUI components that are wonderful for PocketPC development, like the ExpandBar), gather data with a point and click interface (minimising keyboard entry where possible), and then submit it back to the server.
For offline storage we used XML files on the device itself. More than enough for our needs, but yours may differ. Maybe consider SQLite?
There are a couple links you can check out to start with:
http://developer.windowsmobile.com
http://msdn.microsoft.com/en-us/windowsmobile/default.aspx
If you have a sticking point while developing, there are also Windows Mobile dedicated chats on MSDN that you can attend and ask your questions. The calendar hasn't been updated yet, but the next ones should be in January. You can find the schedule here: http://msdn.microsoft.com/en-us/chats/default.aspx
I am going to add an additional question to this post, as its been active enough and hopefully will be helpful to others as well as me. Ok, so after playing around I now realize that standard class libraries cannot be included in windows mobile applications.
Now the overwhelming advice here seems to be use an embedded database, though I now do have use cases and it appears that I will need to have document synchronization as well as relational data. With this in mind service layer interaction seems inevitable. So my question is how would I share common domain objects and interfaces between the layers?
"Document synchronization" - does that mean bidirectional? Or cumulative write-only? I can think of mobile architectures that would mainly collect and submit transactions for a shared document - if that's your requirement, then we should discuss offline - it's a long (and interesting) conversation.
Owen you can share code from Compact Framework -> Desktop, it's only Desktop -> Compact Framework that has compatability issues if you use certain objects that are not supported by the CF.
While a desktop lib doesn't work on CF a CF lib WILL work on the desktop, you can also run CF.exes on the desktop!
Just create a CF library as the project that defines your base objects / interfaces etc.
This book sshould e essential reading for all Windows Mobile developers: http://www.microsoft.com/learning/en/us/books/10294.aspx
For developing windows mobile applications you must have the basic tools like silverlight, visual studio, windows phone emulator and sqlite as your database storage.
A while ago another question referred to the (possibly urban tale) statistic that
... the average lifespan of software is about 3 years
At the time I came up with the following reasons (and I'm sure there are more possibly better ones):
A new major system (ERP, CRM, etc.) is implemented and it has an "integrated" module to replace the old app.
Same, but no integrated app - but the existing app is not adaptable (the people left, technology has changed, current IT policies have changed, users don't like the existing app.)
The company you acquired the basic app from, to customize it for your needs has disappeared.
Or you don't get along well with them any more.
The technology for the existing app is "obsolete" (according to the framework vendor/Microsoft/consultant/industry expert/new IT manager who has management's ear.)
"We're phasing out (Windows 95/Windows 98/Windows 2000/Windows XP/NT) and we need matching technology in our apps".
"We've learned a lot from (App Version n) and we'll do a lot better the second/third/fourth/n+1th time."
Job justification for developers/IT manager/Division VP/consulting company.
The users hate it.
We've merged/acquired a competitor/been acquired by a competitor and theirs is better.
Some of these are unavoidable (e.g. your company gets bought), but overall this is surely smething that needs to be avoided. Does your organization intentionally fight this syndrome? What effective strategies would you recommend?
That's why an application needs to be easy to expand, and you should be able to easily add-in all the buzzwords.
If you have a solid base code, most of the buzzwords are related to the UI (Vista Controls, Ajax, .net, ASP.net 3.5)...
You could be running COBOL in the back-end ( I wouldn't).
A new major system is implemented - There's nothing you can do.
current IT policies have changed, - The app should be adaptable.
users don't like/hate the existing app - why? cosmetic changes in the UI can fix this most of the time.
The company you acquired the basic app from, to customize it for your needs has disappeared. - I wouldn't do that, I'd prefer to write it myself.
The technology for the existing app is "obsolete" (according to the framework vendor/Microsoft/consultant/industry expert/new IT manager who has management's ear.) - same as the above, if the back-end is solid, you should follow these in the front-end.
"We're phasing out (Windows 95/Windows 98/Windows 2000/Windows XP/NT) and we need matching technology in our apps". - a simple compatibility test and minor UI elements solve this.
I'll also say that this is different when you compare in-house to commercial apps, if you're doing an in-house app, change guarantees your job (if you know what you're doing). If you're doing a commercial app, change is an opportunity to make more money, new features would get you upgrades from existing clients and new clients who are looking for the buzzwords, these buzzword could become your advantage when compared to a competitor.
The average lifetime of software I write at the moment is probably a few days. (I write a lot of scripts, so I might be an aberration. ;-) But the core system I work with is probably 15 to 20 years old now. The underlying OS is about 30 years old. There is nothing inherently wrong with either old or young software. In fact, software ages best when it's possible to adapt it to new uses.
Having layers of abstraction between functional parts make it easier to replace functionality in a system. For instance, we've gone through several different tape libraries on our system and now we are considering going to disk archives in the future. Since the "archive" portion of our system sits behind an abstraction layer, we can replace it fairly easily without replacing the rest of the system.
When possible, it's also best to use standard parts. That way, if you run into some limitation, it's likely others will have the same problems and more likely someone will come up with a fix.
Continuous improvement - add useful features at regular intervals
No show-stopping bugs in new versions - testing, testing, testing...
be nice to your clients and treat them with respect (most users really don't want to change their ERPs every three years so if you have a good realtions with them they'll be on your side)
Stay current with new technologies and integrate them in your application when needed
When gathering requirements and someone says "Situation X will always be the case, no exceptions", make it configurable. It will always change, no exceptions.
Most companies don't make it for 5 years. Their software implementations wouldn't be expected to last as long.
Everyone I work with is obsessed with the data-centric approach to enterprise development and hates the idea of using custom collections/objects. What is the best way to convince them otherwise?
Do it by example and tread lightly. Anything stronger will just alienate you from the rest of the team.
Remember to consider the possibility that they're onto something you've missed. Being part of a team means taking turns learning & teaching.
No single person has all the answers.
If you are working on legacy code (e.g., apps ported from .NET 1.x to 2.0 or 3.5) then it would be a bad idea to depart from datasets. Why change something that already works?
If you are, however, creating a new apps, there a few things that you can cite:
Appeal to experiencing pain in maintaining apps that stick with DataSets
Cite performance benefits for your new approach
Bait them with a good middle-ground. Move to .NET 3.5, and promote LINQ to SQL, for instance: while still sticking to data-driven architecture, is a huge, huge departure to string-indexed data sets, and enforces... voila! Custom collections -- in a manner that is hidden from them.
What is important is that whatever approach you use you remain consistent, and you are completely honest with the pros and cons of your approaches.
If all else fails (e.g., you have a development team that utterly refuses to budge from old practices and is skeptical of learning new things), this is a very, very clear sign that you've outgrown your team it's time to leave your company!
Remember to consider the possibility that they're onto something you've missed. Being part of a team means taking turns learning & teaching.
Seconded. The whole idea that "enterprise development" is somehow distinct from (and usually the implication is 'more important than') normal development really irks me.
If there really is a benefit for using some technology, then you'll need to come up with a considered list of all the pros and cons that would occur if you switched.
Present this list to your co workers along with explanations and examples for each one.
You have to be realistic when creating this list. You can't just say "Saves us lots of time!!! WIN!!" without addressing the fact that sometimes it is going to take MORE time, will require X months to come up to speed on the new tech, etc. You have to show concrete examples where it will save time, and exactly how.
Likewise you can't just skirt over the cons as if they don't matter, your co-workers will call you on it.
If you don't do these things, or come across as just pushing what you personally like, nobody is going to take you seriously, and you'll just get a reputation for being the guy who's full of enthusiasm and energy but has no idea about anything.
BTW. Look out for this particular con. It will trump everything, unless you have a lot of strong cases for all your other stuff:
Requires 12+ months work porting our existing code. You lose.
Of course, "it depends" on the situation. Sometimes DataSets or DataTables are more suited, like if it really is pretty light business logic, flat hierarchy of entities/records, or featuring some versioning capabilities.
Custom object collections shine when you want to implement a deep hierarchy/graph of objects that cannot be efficiently represented in flat 2D tables. What you can demonstrate is a large graph of objects and getting certain events to propagate down the correct branches without invoking inappropriate objects in other branches. That way it is not necessary to loop or Select through each and every DataTable just to get the child records.
For example, in a project I got involved in two and half years ago, there was a UI module that is supposed to display questions and answer controls in a single WinForms DataGrid (to be more specific, it was Infragistics' UltraGrid). Some more tricky requirements
The answer control for a question can be anything - text box, check box options, radio button options, drop-down lists, or even to pop up a custom dialog box that may pull more data from a web service.
Depending on what the user answered, it can trigger more sub-questions to appear directly under the parent question. If a different answer is given later, it should expose another set of sub-questions (if any) related to that answer.
The original implementation was written entirely in DataSets, DataTables, and arrays. The amount of looping through the hundreds of rows for multiple tables was purely mind-bending. It did not help the programmer came from a C++ background attempting to ref everything (hello, objects living in the heap use reference variables, like pointers!). Nobody, not even the originally programmer, could explain why the code is doing what it does. I came into the scene more than six months after this, and it was stil flooded with bugs. No wonder the 2nd-generation developer I took over from decided to quit.
Two months of tying to fix the chaotic mess, I took it upon myself to redesign the entire module into an object-oriented graph to solve this problem. yeap, complete with abstract classes (to render different answer control on a grid cell depending on question type), delegates and eventing. The end result was a 2D dataGrid binded to a deep hierarchy of questions, naturally sorted according to the parent-child arrangement. When a parent question's answer changed, it would raise an event to the children questions and they would automatically show/hide their rows in the grid according to the parent's answer. Only question objects down that path were affected. The UI responsiveness of this solution compared to the old method was by orders of magnitude.
Ironically, I wanted to post a question that was the exact opposite of this. Most of the programmers I've worked with have gone with the custom data objects/collections approach. It breaks my heart to watch someone with their SQL Server table definition open on one monitor, slowly typing up a matching row-wrapper class in Visual Studio in another monitor (complete with private properties and getters-setters for each column). It's especially painful if they're also prone to creating 60-column tables. I know there are ORM systems that can build these classes automagically, but I've seen the manual approach used much more frequently.
Engineering choices always involve trade-offs between the pros and cons of the available options. The DataSet-centric approach has its advantages (db-table-like in-memory representation of actual db data, classes written by people who know what they're doing, familiar to large pool of developers etc.), as do custom data objects (compile-type checking, users don't need to learn SQL etc.). If everyone else at your company is going the DataSet route, it's at least technically possible that DataSets are the best choice for what they're doing.
Datasets/tables aren't so bad are they?
Best advise I can give is to use it as much as you can in your own code, and hopefully through peer reviews and bugfixes, the other developers will see how code becomes more readable. (make sure to push the point when these occurrences happen).
Ultimately if the code works, then the rest is semantics is my view.
I guess you can trying selling the idea of O/R mapping and mapper tools. The benefit of treating rows as objects is pretty powerful.
I think you should focus on the performance. If you can create an application that shows the performance difference when using DataSets vs Custom Entities. Also, try to show them Domain Driven Design principles and how it fits with entity frameworks.
Don't make it a religion or faith discussion. Those are hard to win (and is not what you want anyway)
Don't frame it the way you just did in your question. The issue is not getting anyone to agree that this way or that way is the general way they should work. You should talk about how each one needs to think in order to make the right choice at any given time. give an example for when to use dataSet, and when not to.
I had developers using dataTables to store data they fetched from the database and then have business logic code using that dataTable... And I showed them how I reduced the time to load a page from taking 7 seconds of 100% CPU (on the web server) to not being able to see the CPU line move at all.. by changing the memory object from dataTable to Hash table.
So take an example or case that you thing is better implemented differently, and win that battle. Don't fight the a high level war...
If Interoperability is/will be a concern down the line, DataSet is definitely not the right direction to go in. You CAN expose DataSets/DataTables over a service but whether you SHOULD or is debatable. If you are talking .NET->.NET you're probably Ok, otherwise you are going to have a very unhappy client developer from the other side of the fence consuming your service
You can't convince them otherwise. Pick a smaller challenge or move to a different organization. If your manager respects you see if you can do a project in the domain-driven style as a sort of technology trial.
If you can profile, just Do it and profile. Datasets are heavier then a simple Collection<T>
DataReaders are faster then using Adapters...
Changing behavior in an objects is much easier than massaging a dataset
Anyway: Just Do It, ask for forgiveness not permission.
Most programmers don't like to stray out of their comfort zones (note that the intersection of the 'most programmers' set and the 'Stack Overflow' set is the probably the empty set). "If it worked before (or even just worked) then keep on doing it". The project I'm currently on required a lot of argument to get the older programmers to use XML/schemas/data sets instead of just CSV files (the previous version of the software used CSV's). It's not perfect, the schemas aren't robust enough at validating the data. But it's a step in the right direction. The code I develop uses OO abstractions on the data sets rather than passing data set objects around. Generally, it's best to teach by example, one small step at a time.
There is already some very good advice here but you'll still have a job to convince your colleagues if all you have to back you up is a few supportive comments on stackoverflow.
And, if they are as sceptical as they sound, you are going to need more ammo.
First, get a copy of Martin Fowler's "Patterns of Enterprise Architecture" which contains a detailed analysis of a variety of data access techniques.
Read it.
Then force them all to read it.
Job done.
data-centric means less code-complexity.
custom objects means potentially hundreds of additional objects to organize, maintain, and generally live with. It's also going to be a bit faster.
I think it's really a code-complexity vs performance question, which can be answered by the needs of your app.
Start small. Is there a utility app you can use to illustrate your point?
For instance, at a place where I worked, the main application had a complicated build process, involving changing config files, installing a service, etc.
So I wrote an app to automate the build process. It had a rudimentary WinForms UI. But since we were moving towards WPF, I changed it to a WPF UI, while keeping the WinForms UI as well, thanks to Model-View-Presenter. For those who weren't familiar with Model-View-Presenter, it was an easily-comprehensible example they could refer to.
Similarly, find something small where you can show them what a non-DataSet app would look like without having to make a major development investment.