I am trying to learn messaging system. I have found that RabbitMq and NServiceBus are using together in few places. My questions are
If I am using the RabbitMQ then why do i need NServiceBus? and vice versa
What NServiceBus can do but RabbitMQ or Kafka cannot?
Can I use NServiceBus and kafka together? Or Apache-Kafka does not require NServiceBus
Years ago, I asked myself the same question. I was looking at NServiceBus to work with a different message queue, but the question was the same.
I decided not to use NServiceBus.
6 Month later, I realized I had re-built half of what NServiceBus did... only much more poorly.
The equivalent question of why would you need NServiceBus with RabbitMQ, is to ask why you would need the .NET Framework with ASP.NET MVC, or WinForms, or XAML, or any of the built-in libraries that .NET ships with, when you have the Common Language Runtime.
Shouldn't the CLR be enough, after all?
Of course not. Having the runtime on which code can execute - the MSIL interpreter and execution engine - is not nearly enough to be productive.
Sure, you can write command-line applications that take input and produce output. But try to build a real application without the common libraries - without the built-in SQL Server drivers; without any 3rd party controls or libraries. Build a Windows Desktop app without the System.Windows namespace.
You need those libraries to give you collections, and database access, and window objects and UI controls.
Similarly, RabbitMQ gives you everything you need to get started and working, but not enough to maintain productivity.
Sure, you can grab the .NET driver for RabbitMQ and start producing and consuming messages.
For a while, this will work just fine.
Pretty soon, you'll find yourself creating a wrapper around the driver, so you can reduce the amount of code you need to write.
Then you'll find yourself needing to deal with ack vs nack, and you'll create a simple API for that.
Then the need for dead-letter queues will pop up with nack calls, and you'll wrap that up in your API - simplified compared to the rabbitmq driver, of course.
Eventually, you'll want to deal with poison messages - messages that are malformed and causing exceptions. Once again, you don't want to write one-off code for this, so you'll write a library to handle it.
The list goes on and on.
6 months from now, you'll find yourself working with a half-written, barely specified, untestable library that only mimics the value and capabilities of NServiceBus (or MassTransit or whatever other service bus library you choose).
I won't say you have to use NServiceBus. And I would say you should learn how RabbitMQ works, without it. But once you get beyond the basics of sending and receiving messages, the value of NServiceBus and other service bus implementations, becomes very apparent, very quickly.
It seems there is community support for Kafka transport in NServiceBus now: https://docs.particular.net/nservicebus/kafka/
(haven't tried it myself yet).
My team is currently in the initial stages of designing implementations using NServiceBus (v4, possibly v5) in a number of different contexts to facilitate integration between a number of our custom applications. However we would also like to utilize NServiceBus to raise business events triggered from some of our off-the-shelf third-party systems. These systems do not provide inherent messaging or eventing apis, so our current thinking is to hook into their underlying databases using triggers and potentially SQL Service Broker as a bridge to NServiceBus.
I've looked at ServiceBroker.net but that seems to use NServiceBus v2 or v3 api's, interfaces, etc., by creating a totally new ITransport. We're planning on using more recent versions of NServiceBus though, so this doesn't seem to be a solid option. Other somewhat similar questions here on SO (all from a few years ago) seem to be answered with guidance to simply use the SQL Transport. That uses table-based pseudo-queues instead of MSMQ, but what's not clear is if it is then advisable to have SQL triggers hand-craft NServiceBus message records and manually INSERT them into the pseudo-queue tables directly, or whether there would still be some usage of SQL Service Broker in the middle that somehow more natively pops the NServiceBus messages onto the bus. And if somehow using the SQLTransport is the answer, what would be best practice to bridge the messages over to the main MSMQTransport-based bus?
It seemed like there was some concerted movement on SQL Service Broker bridging over to NServiceBus several years ago, but was deprecated once the native NServiceBus SQLTransport was introduced. I feel like maybe I'm missing something in terms of the modern NServiceBus approach to generating data-driven events in a design that is more real-time than a looped polling design.
You may want to take a look at the Gateway feature. You should be able to run 2 different transports and use the Gateway feature to bridge the two via HTTP.
We have a similar system, although it's slightly easier in that we control the underlying databases and applications (i.e. not 3rd party) and the current proof of concept uses the ServiceBroker / SQLDependency / ServiceBus as part of its architecture.
If you go this route, I would also advise using triggers to populate a common table, then monitoring that.
I didn't know about ServiceBroker.Net until today, so can't comment. I also haven't looked at CLR stored procs / triggers; whether there's any possibilities there.
Somebody else asked a question about nServiceBus and ServiceBroker which I answered here which may be useful for anyone looking to implement this.
We are designing a distributed .NET application on W2008R2 servers and need to send a low volume of messages to multiple known nodes. We looked at RabbitMQ and liked it, but found that we hardly need any of its numerous features that would justify the additional complexity of a separate product (and let's not forget Erlang). It appears that good old MSMQ would actually satisfy our needs.
In looking around, there are signs that maybe MSMQ is no longer being used that much. Many postings about it are old and there don't appear to be any new books about it on Amazon, etc.
What we like about MSMQ is the fact that it's already part of the OS. However, when embarking on a major new system, is MSMQ really still a viable technology choice?
We have been using MSMQ with Win Server 2008 and now Win Server 2012. As you said it's part of a system and just works. If you are a Windows guy, use it without fear. It is not obsolete.
Just other frameworks/technologies build on top of mqmq. If you happy with the features, then why would you add an extra dependency on RabitMQ or ActiveMQ, apart from more stuff to learn, and more things to fail?
I'm trying to figure out how to profile a WCF service so I can identify any bottlenecks.
I have found a bit of information on line, but nothing that assumes no prior knowlege which is where I'm at.
What are recomended FREE tools?
- visual studio tools
- clrprofiler
Here is information I found using vsperfcmd.exe to profile wcf service and according to this it is very simple, but I need to fill in the gaps on where to start. My assumptions are to copy VsPerfCLREnv and VsPerfCmd to the server that hosts my wcf service and perform some configuraiton steps that I'm not quite sure on. I'm also not quite sure how I would be able to see the call stack to evaluate the performance of each call.
clrprofiler seems a bit simpler. I assume I would copy clrprofiler.exe to the server, File->Profile Service and add the name and start/stop commands. (is this a friendly name or filename or the service display name?) I assume I would then run my tests against the service and I could see the call stack in clrprofiler. Does that sound correct?
[edit]
I'm not so interested in testing the network since this is on a test server, and this is a large wcf project with multiple devs on it and I am unable to make changes to the project for the sole purpose of monitoring the performance. I want to focus on the performance of the actual methods within it.
Any assistance on getting started is greatly appreciated.
For WCF it is not enough to profile your code only as bunch of things happen on the channel stack (security, deserialization, formatting etc). A good way to visualise that is by using WCF Tracing at verbose level and then using the service trace viewer to see how long it is taking at each step of message processing. Read here on how to configure and use WCF tracing. This is the single most thing that has hepled me with diagnosing WCF issues.
Of course all other code profiling, DB profiling etc. are valid approach as well. You may even use a tool like SoapUI to test your network communication and client side performance overhead for a more end-to-end benchmark.
some things I've learned that someone might find helpful:
you cannot remote profile a service, even over your local network. The profiler must be running on the same machine as the service. (This actually took me quite a while to figure out. Maybe obvious to you, but it was never spelled out so I kept trying to do it)
Visual Studio didn't work for me to profile my WCF service. I was able to get a bit of help from the VS profiler team, but never came out of it with a working solution.
VS was slow to connect and disconnect the profiler and often instrumented my binaries and left them in a corrupted state.
.net binaries do not need to be instrumented since they contain the metadata of the methods which is odd that visual studio kept hosing my binaries trying to instrument them.
I also tried the VS standalone profiler but this is very complex to use and requires reboots of my server.
I ended up getting an internal profiler to work (after getting a private build from the team) so I'm not sure how many profilers out there are designed to work with a WCF service.
I actually set the profiler to watch the WAS service and then added my additional binaries to the profiler.
process explorer is useful when troubleshooting if the profiler is connected or not. Use it to look at inetinfo.exe environment
Can you run it under a debugger?
Can you stand a simple, old-fashioned, method that just works? Here's one.
In addition to Mike's comments, you can use the built-in WCF performance counters to see a number of performance-related metrics and you can also see call times on a WCF trace. Once you know which operations are 'slow' it's usually easier to add some custom timing/logging code to those operations than using a general purpose profiler for something like this. This coming from someone who used to work on commercial profilers.
Tools you should look into: svctracelogviewer (and turn on tracing in both your service and clients). SoapUI for simulating load (and do analysis) and Fiddler, an excellent HTTP sniffer/diagnostics tool.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Okay, I will shortly be starting down the path of windows mobile development. I know nothing about the subject really and I am looking for people with experience to let me know of any gottchas you may know of.
Right now I dont even have a breif of what is requied but the assumption is that the application will be very little more than a bunch of CRUD forms for updating data. The only other requirment knowladge I have is that the application will need to support offline storage when there is no signal avaliable. This in turn will obviously require some kind of syncronization when signal returns.
My initial thoughts are that the application will primarily be a front end to interact with a web service layer. Im assuming that WCF will be an appropriate technology for building these services? I also thought that SQL Server CE would be a good route to go down with regards to the offline storage issues.
Any knowlage that you feel is useful within this domain would be appreciated. Advice, links, books anything appreciated.
EDIT: It has been noted that there are two ways to go with off-line synchronization. To either use some form of message queuing or to use SQL synchronization tools. Could anyone offer a good comparison and introduction to these?
EDIT 2: After a little more digging I get the impression that there are basically 3 different approaches I can use here:
Emmbeded Database to query against then syncronization online, when able
MSMQ along with .NET remoting
WCF with ExchangeWebServiceMailTransport bindings using Exchange Server.
Now, there has been a nice few points raised on the first issue, and I think I understand at some level the issues I would face. But I'd like to get a little more information regarding MSMQ implementations and using WCFs new bindings.
Here a few words from my experience so far (about 9 months) of .net Windows Mobile development.
Well you are occasionally connected. (Or more likely occasionally disconnected). You have to choose whether you are going to use messaging with queues (i.e. WCF/SOAP/XML or something like it) or database synchronisation. I choose the SQL synchronisation route so I can't really comment on messaging. The SQL synchronisation route is not hassle free!
If you go down the sync route with SQL compact like me you basically have two choices. SQL Server merge replication or the newer ADO.NET Synchronisation services. If you choose the former you need to be really careful with your DB design to ensure it can be easily partitioned between mobile subscribers and the publisher. You really need to think about conflicts, and splitting tables that wouldn't normally be split in a normalised DB design is one way of doing that. You have to consider situations where a device goes offline for some time and the publisher DB (i.e. main DB) and/or a subscriber alters the same data. What happens when the device comes back online? It might mean resolving conflicts even if you have partitioned things well. This is where I got burnt. But SQL Merge Replication can work well and reduces the amount of code you have to write.
Roll your own DAL. Don't attempt to use datareaders etc. directly from UI code and don't use typed datasets either. There may be third party DALs that work with Windows Mobile (i.e. I know LLBLGEN does, might be worth a look) but Linq-to-SQL is not supported and anyway you need something lightweight. The chances are the DAL won't be too big so roll it yourself.
If you are using .net you'll probably end up wanting some unimplemented platform features. I recommend using this inexpensive framework to give you what your missing (especially as related to connectivity and power management) - http://www.opennetcf.com/Products/SmartDeviceFramework/tabid/65/Default.aspx
Windows Mobile devices partially switch off to save power when not in use. If you are doing a polling type design you'll need to wake them up every x mins. A normal .net timer class won't do this. You'll need to use a platform feature which can be used from OpenNetCF (above). The timer class is called LargeIntervalTimer and is in the OpenNetCF.WindowsCE assembly/namespace (I think).
Good Luck!
SqlCE is only one of the options available for local data storage on a Windows Mobile device, and although it's an excellent database it has limitations. For one thing, SqlCE will not work (period) under encryption (in other words, if your user encrypts the location where your SDF file is, you will no longer be able to access the data).
The second (and most critical) weakness of SqlCE lies in the RDA/Merge Replication tools. SqlCE Merge Replication is not 100% reliable in situations where the network connection can drop during replication (obviously very common in Windows Mobile devices). If you enjoy trying to explain missing or corrupted data to your clients, go ahead and use SqlCE and merge replication.
Oracle Lite is a good alternative to SqlCE, although it too doesn't work properly under encryption. If encryption is a potential problem, you need to find a database engine that works under encryption (I don't know of one) or else write your own persistence component using XML or something.
Writing a WM application as a front end that primarily interacts with a web service in real time will only work in an always-connected environment. A better approach is to write your application as a front end that primarily interacts with local data (SqlCE, Oracle Lite, XML or whatever), and then create a separate Synchronization component that handles pushing and pulling data.
Again, SqlCE merge replication does this pushing and pulling beautifully and elegantly - it just doesn't work all the time. If you want a replication mechanism that works reliably, you'll have to write your own. Oracle Lite has something called a snapshot table that works very well for this purpose. A snapshot table in Olite tracks changes (like adds, updates and deletes) and allows you to query the changes separately and update the central database (through a web service) to match.
This thread I just posted on SO a few days ago has proven to be a great resource for me thus far.
Also the Windows Mobile MSDN WebCasts are a wealth of information on everything from just getting started up to advanced development.
I would suggest Sqlite for local storage. From the last benchmark I ran it was much better than SqlCe and you don't have to do stupid things like retain an open connection for performance improvements.
Trade-offs being that the toolset is less rich and the integration with other MSSql products is nil. :(
you might want to refer to this:
getting-started-with-windows-mobile-development
You shouldn't be intimidated for windows mobile development. It isn't much different from desktop development. I strongly recommend that you use .NET Compact Framework for development and not C++/MFC.
Some useful links:
Mobile section at the Code
Project. You would find a lot of
articles, a little digging is needed
to find the appropriate one.
Smart
Device Framework from OpenNetCF
offer valuable extensions to the
compact framework.
When you install
the Mobile SDK, you will find under the
Community folder links for the
Windows Mobile and CF framework
blogs. These are also valuable
resources.
Regarding your application, you are right about the WCF and the SQL Server CE. These are the proper ways for handling communication and storage.
Some hints for people coming from a desktop world:
You need to have some sort of power management. The device may automatically go to suspend state. Also, you shouldn't consume power when you don't have to.
Network connectivity is a difficult issue. You can register notifications for when a specific network (Wi-Fi, GPRS) becomes available or unavailable. You can also set the preferred means of communication.
Make the UI as simple as possible. The user uses his thumb and/or a pen and he is probably on the move.
Test in a real device as early as possible.
"24 Hours of Windows Mobile Application Development" from the Windows Mobile Team Blog has some good resources
If you can, try to start from the user use cases and work back to the code, rather than vice versa. It's really easy to spend a lot more time working on the tools than working on the business problem. And thinking through user requirements will help you consider alternate strategies, because a lot of the patterns you know from normal .NET don't apply.
I've done lots of intermittent application development of exactly the type you are describing, and an on-board database works just fine. The MSMQ/WCF stuff just adds conceptual overhead without adding much value. You need a logical datastore locally anyway, and replication at this level is a simple concept that you want to keep simple, so the audit trail is easily monitored and debugged. MSMQ and WCF tend to hide things in unfamiliar places.
I upvoted the SqlLite suggestion BTW. MS doesn't have their persistence story stabilized yet for CE.
For the database replication bit I highly recommend Sybase Ultralite. In terms of flexibility and performance it knocks the socks off SQL CE
I had to do this once. Weird setup with Macs for development, and we were all Java programmers. And a short deadline. PowerPC macs too, so no chance to install Windows for Visual Studio development, never mind that the money for this would never have appeared.
We ended up writing applications using Java, running on the IBM J9 virtual machine, with SWT for a user interface. Entirely free development stack. Easy to deploy. Code ran on any platform we desired, not just PocketPC/WinMob.
Most of the work was on the server side anyway - the database, the web service server. The logic. The reporting engine. The client side wasn't totally simple however - would get the form templates from the server (because they changed frequently), the site details (multi-site deployment), generate a UI from the form template (using some SWT GUI components that are wonderful for PocketPC development, like the ExpandBar), gather data with a point and click interface (minimising keyboard entry where possible), and then submit it back to the server.
For offline storage we used XML files on the device itself. More than enough for our needs, but yours may differ. Maybe consider SQLite?
There are a couple links you can check out to start with:
http://developer.windowsmobile.com
http://msdn.microsoft.com/en-us/windowsmobile/default.aspx
If you have a sticking point while developing, there are also Windows Mobile dedicated chats on MSDN that you can attend and ask your questions. The calendar hasn't been updated yet, but the next ones should be in January. You can find the schedule here: http://msdn.microsoft.com/en-us/chats/default.aspx
I am going to add an additional question to this post, as its been active enough and hopefully will be helpful to others as well as me. Ok, so after playing around I now realize that standard class libraries cannot be included in windows mobile applications.
Now the overwhelming advice here seems to be use an embedded database, though I now do have use cases and it appears that I will need to have document synchronization as well as relational data. With this in mind service layer interaction seems inevitable. So my question is how would I share common domain objects and interfaces between the layers?
"Document synchronization" - does that mean bidirectional? Or cumulative write-only? I can think of mobile architectures that would mainly collect and submit transactions for a shared document - if that's your requirement, then we should discuss offline - it's a long (and interesting) conversation.
Owen you can share code from Compact Framework -> Desktop, it's only Desktop -> Compact Framework that has compatability issues if you use certain objects that are not supported by the CF.
While a desktop lib doesn't work on CF a CF lib WILL work on the desktop, you can also run CF.exes on the desktop!
Just create a CF library as the project that defines your base objects / interfaces etc.
This book sshould e essential reading for all Windows Mobile developers: http://www.microsoft.com/learning/en/us/books/10294.aspx
For developing windows mobile applications you must have the basic tools like silverlight, visual studio, windows phone emulator and sqlite as your database storage.