We have been using Aeropsike exentsively as key-value store only. However now I couldn't find a way to actually save a lock in Aeropspike like the way we do in Redis.
I guess, I can always save native application lock as blobs, but that means I will be limited to a particular implementation in my application.
Don't believe you can "save" locks in Aerospike. Any form of locking has to be implemented at the Application level and then you have to deal with the locking client abandoning the lock.
I would suggest that you read Martin Kleppmann's Redlock Discussion. It covers the responses people in the distributed systems community had to Antirez's debating the topic, following Kleppmann's earlier article How to do distributed locking.
Building a DLM is not a trivial problem at all, and Redlock fails as one. If you are up for it, you can consider writing such a thing over the linearizable strong consistency mode of Aerospike Enterprise Edition 4.0.
As opposed to Redis and its variants, Aerospike EE 4.0 passes Jepsen.
Related
The last 6 months learning curve have been challenging with CQRS and DDD the main culprits.
It has been fun and we are 1/2 way through our project and the area I have not had time to delve into is a messaging framework.
Currently I don't use DTC so there is a very good likely hood that if my read model is not updated then I will have inconsistency between the read and write databases. Also my read and write database will be on the same machine. I doubt we will ever put them on separate machines.
I don't have a large volume of messages in my system so my concern is more to do with consistency and reliability of the system.
So, do I have to put in a messaging framework like NServiceBus (even though both read and write databases are on the same machine) or do I have other options? Yes there is learning curve but I suppose there would be a hell of a lot to learn if I don't use it.
Also, I don't want to put in a layer if it is not necessary
Thoughts?
Currently I don't use DTC so there is a very good likely hood that if
my read model is not updated then I will have inconsistency between
the read and write databases.
Personally, I dislike the DTC and try to avoid it. Instead, it is often possible to implement a compensation mechanism, especially for something like a read model where eventual consistency is already acceptable and updates are idempotent. For example, you could implement a version on entities and have a background task which ensures versions are in-sync. Having a DTC will provide transactional retry functionality, but it still won't solve cases where failure occurs after retries - you still have to watch the error log and have procedures in place to deal with errors.
So, do I have to put in a messaging framework like NServiceBus (even
though both read and write databases are on the same machine) or do I
have other options?
It depends on a few things. What you often encounter in a CQRS system is need for pub/sub where several sub-systems publish events to which the query/caching system subscribes to. If you see a need for pub/sub beyond basic point-to-point messaging, then go with something like NServiceBus. Also, I wouldn't immediately shy away from using NServiceBus even if you don't need it for scalability purposes because I think the logical partitioning is beneficial on its own. On the other hand, as you point out, adding layers of complexity is costly, therefore first try to see if the simplest possible thing will work.
Another question to ask is whether you need a separate query store at all. If all you have is a single machine, why bother? You could use something simpler like the read-model pattern and still reap a lot of the benefits of CQRS.
Does a CQRS project need a messaging framework like NServiceBus?
The short answer: no.
It is the first time I hear about the 'read-model pattern' mentioned by eulerfx. It is a nice enough name but there is a bit more to it:
The general idea behind the 'query' part is to query a denormalized view of your data. In the 'read-model pattern' link you will notice that the query used to populate the read-model is doing some lifting. In the mentioned example the required data manipulation is not that complex but what if it does become more complex? This is where the denomalization comes in. When you perform your 'command' part the next action is to denormalize the data and store the results for easy reading. All the heavy lifting should be done by your domain.
This is why you are asking about the messaging. There are several techniques here:
denormalized data in same database, same table, different columns
denormalized data in same database, different table
denormalized data in different database
That's the storage. How about the consistency?:
immediately consistent
eventually consistent
The simplest solution (quick win) is to denormalize your data in your domain and then after saving your domain objects through the repository you immediately save the denomarlized data to the same data store, same table(s), different columns. 100% consistent and you can start reading the denormalized data immediately.
If you really want to you can create a separate bunch of objects to transport that data but it is simpler to just write a simple query layer that returns some data carrying object provided by your data-access framework (in the case of .Net that would be a DataRow/DataTable). Absolutely no reason to get fancy. There will always be exceptions but then you can go ahead and write a data container.
For eventual consistency you will need some form of queuing and related processing. You can roll your own solution or your may opt for a service bus. That is up to you and your time / technical constraints :)
BTW: I have a free open-source service bus here:
Shuttle.Esb
documentation
Any feedback would be welcomed. But any old service bus will do (MassTransit / NServiceBus / etc.).
Hope that helps.
Hello All
I am planing to use Nhibernate in Dotnet 3.5 application.This application is like an enterprise application which will provide core services to all other application of my company. So my question is:
Is Nhibernate is right choice with this kind of application?
Are there any performance issues with Nhibernate?
NHibernate is good for any kind of data access applications.
As for performance its good enough for most of the apps. The only thing where I think it won't fit is synchronization tasks where really tons of data could be transferred. For that kind of work any ORM would suck.
I know of many places that use Nhibernate for Enterprise and mission critical application and are satisfied.
Nhibernate supplies an object oriented abstraction to data base entities. if you're comparing them to equivalent SQL operations, the performance penalty is negligible.
However, when using more complex operations, mainly on large sets of data, native SQL may perform significantly better.
NHibernate is the probably the right choice. The library is very mature (currently v3), based on java Hibernate which is there for even longer.
It depends. NHibernate is designed for OLTP scenarios. That means you load some small chunk of data, process it and you save it back. Critical part is: "small chunk".
If you find yourself in any OLAP-like scneario where you must batch process large chunks, NH is not a good choice. In such situation you most likely don't want to use any ORM at all.
As always: Right tool for the right job.
If your application is a typical enterprise like, you will be happy with NH. The library is extremely flexible with many fine-tuning options. Also remember that you can still use plain old ADO.NET in OLAP parts of your app if there are any.
If you run into any problems, NH community is very supportive plus there are companies and individuals offering consulting and paid support (in case your company policy requires it).
I'm using NH for 3 years and I can recommend it - the tool does its job.
I am developing AMF Flash gateway on FlourineFx application for deployment on Windows Azure and I want to use Azure SQL.
I use NHibernate 2.1 + NHibernate.Linq 1.0 + FluentNHibernate 1.1
There will be two or more instances of this FlourineFx gateway and only 1 database.
I am planning on implementing memcached as 2nd level cache later (as Windows Azure WorkerRole), but is it necessary?
(I don't mind performance, but I do mind consistency)
I don't know if 2nd level cache solves some transaction-related problems or just makes it faster
The main point of the L2 cache is to avoid database hits and I wouldn't say that the L2 cache solves transactions-related problems; It might just be involved (and thus make the whole process a bit more complicated), if fully transactional caches are supported by NHibernate.
Personally, I tend to limit the use of L2 caching to read-only (or mostly read) objects, that's where the L2 cache gives all its power. Caching read-write entities is trickier, especially in a clustered environment, and the cache provider must support the Cache Concurrency Strategy required by your application for a given entity (read-only, non-strict-read-write, read-write).
I'm not sure this really answers the question, but at least it might give you some hints.
References
17.2. The Second Level Cache
Chapter 23, NHibernate.Caches
The cache won't help you with consistency. Of course it will help with performance, and you should use a distributed one, like memcached, if running multiple instances, as you correctly inferred.
That said, NHibernate does have features to help with consistency. Check:
5.1.7. version
10.4. Optimistic concurrency control
No, you don't.
But, how you guys helped me point to the right direction, it will help with performance.
So I will definitely run some instances of memcached and investigate concurrency control further.
Thanks.
as opposed to writing your own library.
We're working on a project here that will be a self-dividing server pool, if one section grows too heavy, the manager would divide it and put it on another machine as a separate process. It would also alert all connected clients this affects to connect to the new server.
I am curious about using ZeroMQ for inter-server and inter-process communication. My partner would prefer to roll his own. I'm looking to the community to answer this question.
I'm a fairly novice programmer myself and just learned about messaging queues. As i've googled and read, it seems everyone is using messaging queues for all sorts of things, but why? What makes them better than writing your own library? Why are they so common and why are there so many?
what makes them better than writing your own library?
When rolling out the first version of your app, probably nothing: your needs are well defined and you will develop a messaging system that will fit your needs: small feature list, small source code etc.
Those tools are very useful after the first release, when you actually have to extend your application and add more features to it.
Let me give you a few use cases:
your app will have to talk to a big endian machine (sparc/powerpc) from a little endian machine (x86, intel/amd). Your messaging system had some endian ordering assumption: go and fix it
you designed your app so it is not a binary protocol/messaging system and now it is very slow because you spend most of your time parsing it (the number of messages increased and parsing became a bottleneck): adapt it so it can transport binary/fixed encoding
at the beginning you had 3 machine inside a lan, no noticeable delays everything gets to every machine. your client/boss/pointy-haired-devil-boss shows up and tell you that you will install the app on WAN you do not manage - and then you start having connection failures, bad latency etc. you need to store message and retry sending them later on: go back to the code and plug this stuff in (and enjoy)
messages sent need to have replies, but not all of them: you send some parameters in and expect a spreadsheet as a result instead of just sending and acknowledges, go back to code and plug this stuff in (and enjoy.)
some messages are critical and there reception/sending needs proper backup/persistence/. Why you ask ? auditing purposes
And many other use cases that I forgot ...
You can implement it yourself, but do not spend much time doing so: you will probably replace it later on anyway.
That's very much like asking: why use a database when you can write your own?
The answer is that using a tool that has been around for a while and is well understood in lots of different use cases, pays off more and more over time and as your requirements evolve. This is especially true if more than one developer is involved in a project. Do you want to become support staff for a queueing system if you change to a new project? Using a tool prevents that from happening. It becomes someone else's problem.
Case in point: persistence. Writing a tool to store one message on disk is easy. Writing a persistor that scales and performs well and stably, in many different use cases, and is manageable, and cheap to support, is hard. If you want to see someone complaining about how hard it is then look at this: http://www.lshift.net/blog/2009/12/07/rabbitmq-at-the-skills-matter-functional-programming-exchange
Anyway, I hope this helps. By all means write your own tool. Many many people have done so. Whatever solves your problem, is good.
I'm considering using ZeroMQ myself - hence I stumbled across this question.
Let's assume for the moment that you have the ability to implement a message queuing system that meets all of your requirements. Why would you adopt ZeroMQ (or other third party library) over the roll-your-own approach? Simple - cost.
Let's assume for a moment that ZeroMQ already meets all of your requirements. All that needs to be done is integrating it into your build, read some doco and then start using it. That's got to be far less effort than rolling your own. Plus, the maintenance burden has been shifted to another company. Since ZeroMQ is free, it's like you've just grown your development team to include (part of) the ZeroMQ team.
If you ran a Software Development business, then I think that you would balance the cost/risk of using third party libraries against rolling your own, and in this case, using ZeroMQ would win hands down.
Perhaps you (or rather, your partner) suffer, as so many developers do, from the "Not Invented Here" syndrome? If so, adjust your attitude and reassess the use of ZeroMQ. Personally, I much prefer the benefits of Proudly Found Elsewhere attitude. I'm hoping I can proud of finding ZeroMQ... time will tell.
EDIT: I came across this video from the ZeroMQ developers that talks about why you should use ZeroMQ.
what makes them better than writing your own library?
Message queuing systems are transactional, which is conceptually easy to use as a client, but hard to get right as an implementor, especially considering persistent queues. You might think you can get away with writing a quick messaging library, but without transactions and persistence, you'd not have the full benefits of a messaging system.
Persistence in this context means that the messaging middleware keeps unhandled messages in permanent storage (on disk) in case the server goes down; after a restart, the messages can be handled and no retransmit is necessary (the sender does not even know there was a problem). Transactional means that you can read messages from different queues and write messages to different queues in a transactional manner, meaning that either all reads and writes succeed or (if one or more fail) none succeeds. This is not really much different from the transactionality known from interfacing with databases and has the same benefits (it simplifies error handling; without transactions, you would have to assure that each individual read/write succeeds, and if one or more fail, you have to roll back those changes that did succeed).
Before writing your own library, read the 0MQ Guide here: http://zguide.zeromq.org/page:all
Chances are that you will either decide to install RabbitMQ, or else you will make your library on top of ZeroMQ since they have already done all the hard parts.
If you have a little time give it a try and roll out your own implemntation! The learnings of this excercise will convince you about the wisdom of using an already tested library.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Okay, I will shortly be starting down the path of windows mobile development. I know nothing about the subject really and I am looking for people with experience to let me know of any gottchas you may know of.
Right now I dont even have a breif of what is requied but the assumption is that the application will be very little more than a bunch of CRUD forms for updating data. The only other requirment knowladge I have is that the application will need to support offline storage when there is no signal avaliable. This in turn will obviously require some kind of syncronization when signal returns.
My initial thoughts are that the application will primarily be a front end to interact with a web service layer. Im assuming that WCF will be an appropriate technology for building these services? I also thought that SQL Server CE would be a good route to go down with regards to the offline storage issues.
Any knowlage that you feel is useful within this domain would be appreciated. Advice, links, books anything appreciated.
EDIT: It has been noted that there are two ways to go with off-line synchronization. To either use some form of message queuing or to use SQL synchronization tools. Could anyone offer a good comparison and introduction to these?
EDIT 2: After a little more digging I get the impression that there are basically 3 different approaches I can use here:
Emmbeded Database to query against then syncronization online, when able
MSMQ along with .NET remoting
WCF with ExchangeWebServiceMailTransport bindings using Exchange Server.
Now, there has been a nice few points raised on the first issue, and I think I understand at some level the issues I would face. But I'd like to get a little more information regarding MSMQ implementations and using WCFs new bindings.
Here a few words from my experience so far (about 9 months) of .net Windows Mobile development.
Well you are occasionally connected. (Or more likely occasionally disconnected). You have to choose whether you are going to use messaging with queues (i.e. WCF/SOAP/XML or something like it) or database synchronisation. I choose the SQL synchronisation route so I can't really comment on messaging. The SQL synchronisation route is not hassle free!
If you go down the sync route with SQL compact like me you basically have two choices. SQL Server merge replication or the newer ADO.NET Synchronisation services. If you choose the former you need to be really careful with your DB design to ensure it can be easily partitioned between mobile subscribers and the publisher. You really need to think about conflicts, and splitting tables that wouldn't normally be split in a normalised DB design is one way of doing that. You have to consider situations where a device goes offline for some time and the publisher DB (i.e. main DB) and/or a subscriber alters the same data. What happens when the device comes back online? It might mean resolving conflicts even if you have partitioned things well. This is where I got burnt. But SQL Merge Replication can work well and reduces the amount of code you have to write.
Roll your own DAL. Don't attempt to use datareaders etc. directly from UI code and don't use typed datasets either. There may be third party DALs that work with Windows Mobile (i.e. I know LLBLGEN does, might be worth a look) but Linq-to-SQL is not supported and anyway you need something lightweight. The chances are the DAL won't be too big so roll it yourself.
If you are using .net you'll probably end up wanting some unimplemented platform features. I recommend using this inexpensive framework to give you what your missing (especially as related to connectivity and power management) - http://www.opennetcf.com/Products/SmartDeviceFramework/tabid/65/Default.aspx
Windows Mobile devices partially switch off to save power when not in use. If you are doing a polling type design you'll need to wake them up every x mins. A normal .net timer class won't do this. You'll need to use a platform feature which can be used from OpenNetCF (above). The timer class is called LargeIntervalTimer and is in the OpenNetCF.WindowsCE assembly/namespace (I think).
Good Luck!
SqlCE is only one of the options available for local data storage on a Windows Mobile device, and although it's an excellent database it has limitations. For one thing, SqlCE will not work (period) under encryption (in other words, if your user encrypts the location where your SDF file is, you will no longer be able to access the data).
The second (and most critical) weakness of SqlCE lies in the RDA/Merge Replication tools. SqlCE Merge Replication is not 100% reliable in situations where the network connection can drop during replication (obviously very common in Windows Mobile devices). If you enjoy trying to explain missing or corrupted data to your clients, go ahead and use SqlCE and merge replication.
Oracle Lite is a good alternative to SqlCE, although it too doesn't work properly under encryption. If encryption is a potential problem, you need to find a database engine that works under encryption (I don't know of one) or else write your own persistence component using XML or something.
Writing a WM application as a front end that primarily interacts with a web service in real time will only work in an always-connected environment. A better approach is to write your application as a front end that primarily interacts with local data (SqlCE, Oracle Lite, XML or whatever), and then create a separate Synchronization component that handles pushing and pulling data.
Again, SqlCE merge replication does this pushing and pulling beautifully and elegantly - it just doesn't work all the time. If you want a replication mechanism that works reliably, you'll have to write your own. Oracle Lite has something called a snapshot table that works very well for this purpose. A snapshot table in Olite tracks changes (like adds, updates and deletes) and allows you to query the changes separately and update the central database (through a web service) to match.
This thread I just posted on SO a few days ago has proven to be a great resource for me thus far.
Also the Windows Mobile MSDN WebCasts are a wealth of information on everything from just getting started up to advanced development.
I would suggest Sqlite for local storage. From the last benchmark I ran it was much better than SqlCe and you don't have to do stupid things like retain an open connection for performance improvements.
Trade-offs being that the toolset is less rich and the integration with other MSSql products is nil. :(
you might want to refer to this:
getting-started-with-windows-mobile-development
You shouldn't be intimidated for windows mobile development. It isn't much different from desktop development. I strongly recommend that you use .NET Compact Framework for development and not C++/MFC.
Some useful links:
Mobile section at the Code
Project. You would find a lot of
articles, a little digging is needed
to find the appropriate one.
Smart
Device Framework from OpenNetCF
offer valuable extensions to the
compact framework.
When you install
the Mobile SDK, you will find under the
Community folder links for the
Windows Mobile and CF framework
blogs. These are also valuable
resources.
Regarding your application, you are right about the WCF and the SQL Server CE. These are the proper ways for handling communication and storage.
Some hints for people coming from a desktop world:
You need to have some sort of power management. The device may automatically go to suspend state. Also, you shouldn't consume power when you don't have to.
Network connectivity is a difficult issue. You can register notifications for when a specific network (Wi-Fi, GPRS) becomes available or unavailable. You can also set the preferred means of communication.
Make the UI as simple as possible. The user uses his thumb and/or a pen and he is probably on the move.
Test in a real device as early as possible.
"24 Hours of Windows Mobile Application Development" from the Windows Mobile Team Blog has some good resources
If you can, try to start from the user use cases and work back to the code, rather than vice versa. It's really easy to spend a lot more time working on the tools than working on the business problem. And thinking through user requirements will help you consider alternate strategies, because a lot of the patterns you know from normal .NET don't apply.
I've done lots of intermittent application development of exactly the type you are describing, and an on-board database works just fine. The MSMQ/WCF stuff just adds conceptual overhead without adding much value. You need a logical datastore locally anyway, and replication at this level is a simple concept that you want to keep simple, so the audit trail is easily monitored and debugged. MSMQ and WCF tend to hide things in unfamiliar places.
I upvoted the SqlLite suggestion BTW. MS doesn't have their persistence story stabilized yet for CE.
For the database replication bit I highly recommend Sybase Ultralite. In terms of flexibility and performance it knocks the socks off SQL CE
I had to do this once. Weird setup with Macs for development, and we were all Java programmers. And a short deadline. PowerPC macs too, so no chance to install Windows for Visual Studio development, never mind that the money for this would never have appeared.
We ended up writing applications using Java, running on the IBM J9 virtual machine, with SWT for a user interface. Entirely free development stack. Easy to deploy. Code ran on any platform we desired, not just PocketPC/WinMob.
Most of the work was on the server side anyway - the database, the web service server. The logic. The reporting engine. The client side wasn't totally simple however - would get the form templates from the server (because they changed frequently), the site details (multi-site deployment), generate a UI from the form template (using some SWT GUI components that are wonderful for PocketPC development, like the ExpandBar), gather data with a point and click interface (minimising keyboard entry where possible), and then submit it back to the server.
For offline storage we used XML files on the device itself. More than enough for our needs, but yours may differ. Maybe consider SQLite?
There are a couple links you can check out to start with:
http://developer.windowsmobile.com
http://msdn.microsoft.com/en-us/windowsmobile/default.aspx
If you have a sticking point while developing, there are also Windows Mobile dedicated chats on MSDN that you can attend and ask your questions. The calendar hasn't been updated yet, but the next ones should be in January. You can find the schedule here: http://msdn.microsoft.com/en-us/chats/default.aspx
I am going to add an additional question to this post, as its been active enough and hopefully will be helpful to others as well as me. Ok, so after playing around I now realize that standard class libraries cannot be included in windows mobile applications.
Now the overwhelming advice here seems to be use an embedded database, though I now do have use cases and it appears that I will need to have document synchronization as well as relational data. With this in mind service layer interaction seems inevitable. So my question is how would I share common domain objects and interfaces between the layers?
"Document synchronization" - does that mean bidirectional? Or cumulative write-only? I can think of mobile architectures that would mainly collect and submit transactions for a shared document - if that's your requirement, then we should discuss offline - it's a long (and interesting) conversation.
Owen you can share code from Compact Framework -> Desktop, it's only Desktop -> Compact Framework that has compatability issues if you use certain objects that are not supported by the CF.
While a desktop lib doesn't work on CF a CF lib WILL work on the desktop, you can also run CF.exes on the desktop!
Just create a CF library as the project that defines your base objects / interfaces etc.
This book sshould e essential reading for all Windows Mobile developers: http://www.microsoft.com/learning/en/us/books/10294.aspx
For developing windows mobile applications you must have the basic tools like silverlight, visual studio, windows phone emulator and sqlite as your database storage.