I am new to NServiceBus, trying to introduce messaging into a WCF/RPC solution.
Because of architectural constraints and overhead (memory and cpu usage already high) IT Operations will not allow MSDTC. (I'm also keen to avoid 2PC fwiw). I also require messages over http so the NSB bridge looks like a great solution.
Based on these posts (how-i-avoid-two-phase-commit and extending-nservicebus-avoiding-two-phase-commits) it looks to me as though it's possible to use NSB with the DTC disabled.
It sounds like EventStore does manage to avoid 2PC in the same way that I want to setup NSB but at the moment I just want to get NSB to work rather than adding event sourcing into the mix.
Questions:
Are there any examples of configuring NSB to work this way? I'm quite happy to add the extra complexity (custom message handler with local message state storage) - without 2PC there isn't really another option. I already know of this example (IdempotentConsumer) but the test projects for this repo contain no code. It would be even more helpful if there was an example using nosql storage.
Will I need to alter the NSB bridge to deal with no DTC? I'm guessing no - bridge transactions are only against the local queue but the process that consumes the local queue will obviously need to coded to avoid 2PC. Correct?
Are there any other useful resources/posts around using NSB without MSDTC? The solution (how-i-avoid-two-phase-commit) seems not too complex but given I'm just starting out with NSB it would be great to find a quickstart for this...
I would have thought this would be a common scenario - but there doesn't seem to be much written about avoiding MSDTC while still using NSB. Surely there are others who are using a message bus but aren't allowed to use MSDTC... Is there another obvious way that I've missed?
thanks
2) Yes you should be fine. Since you're doing the deduplication your self you don't need the gateway to do it for you. Just configure it to use the InMemory persistence and you should be fine.
Related
I'm using NServiceBus as an abstraction layer for Azure Service Bus (in case we move away from Azure). I find that when working with multiple subscribers (who subscribe to the same events) the number of duplicate messages increases. I know Azure Service Bus (ASB) has a way of detecting these duplicates and I can see that the feature is configurable through NServiceBus (according to documentation). However, I can only get a sample of of achieving duplication detection by means of configuration section. What I require is a sample of how to achieve this with code.
Thanks
Suraj
You can specify configuration using code-based approach as well. NServiceBus has to contracts that can help with that IConfigurationSource and IProvideConfiguration<T>. Here's an example how you can take a configuration file section (UnicastBusConfig) and specify values via code.
Specifically to what you've asked, implementing IProvideConfiguration<AzureServiceBusQueueConfig> will allow you configure ASB transport, specifying duplicates and such.
The observation about number of duplicates increasing as a result of increasing subscribers feels as a symptom, not the problem. That is probably a different question, not related to the configuration. Saying that, I'd look into it prior to enabling the native de-dupplication. While you can specify RequiresDuplicateDetection and DuplicateDetectionHistoryTimeWindow be aware that ASB performing duplicate detection on the ID property only. Also, it is better to build your handlers as idempotent, rather than relying on the native de-duplication.
We have multiple web and windows applications which were deployed to different servers that we are planning to integrate using NservierBus to let all apps can pub/sub message between them, I think we using pub/sub pattern and using MSMQ transport will be good for it. but one thing I am not clear if it is a way to avoid hard code to set sub endpoint to MSMQ QueueName#ServerName which has server name in it directly if pub is on another server. on 6-pre I saw idea to set endpoint name then using routing to delegate to transport-level address, is that a solution to do that? or only gateway is the solution? is a broker a good idea? what is the best practice for this scenario?
When using pub/sub, the subscriber currently needs to know the location of the queue of the publisher. The subscriber then sends a subscription-message to that queue, every single time it starts up. It cannot know if it subscribed already and if it subscribed for all the messages, since you might have added/configured some new ones.
The publisher reads these subscriptions messages and stores the subscription in storage. NServiceBus does this for you, so there's no need to write code for this. The only thing you need is configuration in the subscriber as to where the (queue of the) publisher is.
I wrote a tutorial myself which you can find here : http://dennis.bloggingabout.net/2015/10/28/nservicebus-publish-subscribe-tutorial/
That being said, you should take special care related to issues regarding websites that publish messages. More information on that can be found here : http://docs.particular.net/nservicebus/hosting/publishing-from-web-applications
In a scale out situation with MSMQ, you can also use the distributor : http://docs.particular.net/nservicebus/scalability-and-ha/distributor/
As a final note: It depends on the situation, but I would not worry too much about knowing locations of endpoints (or their queues). I would most likely not use pub/sub just for this 'technical issue'. But again, it completely depends on the situation. I can understand that rich-clients which spawn randomly might want this. But there are other solutions as well, with a more centralized storage and an API that is accessed by all the rich clients.
I am a newbie to real-time application development and am trying to wrap my head around the myriad options out there. I have read as many blog posts, notes and essays out there that people have been kind enough to share. Yet, a simple problem seems unanswered in my tiny brain. I thought a number of other people might have the same issues, so I might as well sign up and post here on SO. Here goes:
I am building a tiny real-time app which is asynchronous chat + another fun feature. I boiled my choices down to the following two options:
LAMP + RabbitMQ
Node.JS + Redis + Pub-Sub
I believe that I get the basics to start learning and building this out. However, my (seriously n00b) questions are:
How do I communicate with the end-user -> Client to/from Server in both of those? Would that be simple Javascript long/infinite polling?
Of the two, which might more efficient to build out and manage from a single Slice (assuming 100 - 1,000 users)?
Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )
I hope this isn't a crazy question and won't get flamed right away. Would love some constructive feedback, love this community!
Thank you.
Architecturally, both of your choices are the same as storing data in an Oracle database server for another application to retrieve.
Both the RabbitMQ and the Redis solution require your apps to connect to an intermediary server that handles the data communications. Redis is most like Oracle, because it can be used simply as a persistent database with a network API. But RabbitMQ is a little different because the MQ Broker is not really responsible for persisting data. If you configure it right and use the right options when publishing a message, then RabbitMQ will actually persist the data for you but you can't get the data out except as part of the normal message queueing process. In other words, RabbitMQ is for communicating messages and only offers persistence as a way of recovering from network problems or system crashes.
I would suggest using RabbitMQ and whatever programming languages you are already familiar with. Since the M in LAMP is usually interpreted as MySQL, this means that you would either not use MySQL at all, or only use it for long term storage of data, not for the realtime communications.
The RabbitMQ site has a huge amount of documentation about building apps with AMQP. I suggest that after you install RabbitMQ, you read through the docs for rabbitmqctl and then create a vhost to experiment in. That way it is easy to clean up your experiments without resetting everything. I also suggest using only topic exchanges because you can emulate the behavior of direct and fanout exchanges by using wildcards in the routing_key.
Remember, you only publish messages to exchanges, and you only receive messages from queues. The exchange is responsible for pattern matching the message's routing_key to the queue's binding_key to determine which queues should receive a copy of the message. It is worthwhile learning the whole AMQP model even if you only plan to send messages to one queue with the same name as the routing_key.
If you are building your client in the browser, and you want to build a prototype, then you should consider just using XHR today, and then move to something like Kamaloka-js which is a pure Javascript implementation of AMQP (the AMQ Protocol) which is the standard protocol used to communicate to a RabbitMQ message broker. In other words, build it with what you know today, and then speed it up later which something (AMQP) that has a long term future in your toolbox.
Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )
This is usually called RAD (rapid application design/development) and it is what I would recommend right now. This lets you build the proof of concept that you can use to work off of later to get what you want to happen.
As for how to talk to the clients from the server, and vice versa, have you read at all on websockets?
Given the choice between LAMP or event based programming, for what you're suggesting, I would tell you to go with the event based programming, so nodejs. But that's just one man's opinion.
Well,
LAMP - Apache create new process for every request. RabbitMQ can be useful with many features.
Node.js - Uses single process to handle all request asynchronously with help of event looping. So, no extra overhead process creation like apache.
For asynchronous chat application,
socket.io + Node.js + redis pub-sup is best stack.
I have already implemented real-time notification using above stack.
I'm currently playing around a little with WCF, during this I stepped on a question where I'm not sure if I'm on the right track.
Let's assume a simple setup that looks like this: client -> service1 -> service2.
The communication is tcp-based.
So where I'm not sure is, if it makes sense that the service1 caches the client proxy for service2. So I might get a multi-threaded access to that proxy, and I have to deal with it.
I'd like to take advantage of the tcp session to get better performance, but I'm not sure if this "architecture" is supported by WCF/network/whatever at all. The problem I see is that all the communication goes over the same channel, if I'm not using locks or another sync.
I guess the better idea is to cache the proxy in a threadstatic variable.
But before I do that, I wanted to confirm that it's really not a good idea to have only one proxy instance.
tia
Martin
If you don't know that you have a performance problem, then why worry about caching? You're opening yourself to the risk of improperly implementing multithreading code, and without any clear, measurable benefit.
Have you measured performance yet, or profiled the application to see where it's spending its time? If not, then when you do, you may well find that the overhead of multiple TCP sessions is not where your performance problems lie. You may wish you had the time to optimize some other part of your application, but you will have spent that time optimizing something that didn't need to be optimized.
I am already using such a structure. I have one service that collaborates with some other services and realise the implementation. Of course, in my case the client calls some one-way method of the first service. I am getting very good benifit. Of course, I also have configured it to limit the number of concurrent calls in some of the cases.
Yes, that architecture is supported by WCF. I deal with applications every day that use similar structures, using NetTCPBinding.
The biggest thing to worry about is the ConcurrencyMode of the various services involved, and making sure that they do not block unnecessarily. It is very easy to get into a scenario where you will be guaranteed timeouts, or at the least have poor performance due to multiple, synchronous calls across service boundaries. Even OneWay calls are not guaranteed to immediately return.
careful with threadstatic, .net changes the thread so the variable can get null.
For session...perhaps you could use session enabled calls:
http://msdn.microsoft.com/en-us/library/ms733040.aspx
But i would not recomend using if you do not have any performance issue. I would use the normal way, or if service 1 is just for forwarding you could use that functionality easily with 4.0:
http://www.sdn.nl/SDN/Artikelen/tabid/58/view/View/ArticleID/2979/Whats-New-in-WCF-40.aspx
Regards
Firstly, make sure you know about the behaviour of ThreadStatic in ASP.NET applications:
http://piers7.blogspot.com/2005/11/threadstatic-callcontext-and_02.html
The same thread that started your request may not be the same thread that finishes it. Basically the only safe way of storing Thread local storage in ASP.NET applications is inside HttpContext. The next obvious approach would be to creat a wrapper client to manage your WCF client proxy and ensure each IO request is thread safe using locks.
Although my personal preference would be to use a pool of proxy clients. Whenever you need one pop it off the pool queue and when you're finished with it put it back on.
Currently im developing a server for satelite monitoring of objects.
In its current state it is very efficient and stable in hi load scenarios.
Server must be able to handle 50+mln messages per day, or more if load balancing is used.
It consists of gps data gateway(singleton), "databroker"(singleton), which is responsible for persisting data, providing it on request, and alerting subscribers about new data, login service(per call) and client service(per session), responsible for subscription and working with web interface and rich client.
At this state i can forsee those problems:
Growing complexity if new services are added.
Tight coupling.
Hard to configure.
Lots of low level code for handling subscriptions etc in future bl services(reporting for example).
To solve those problems i want to use something like ServiceBus.
MS solutions are to expensive for our customers, NServiceBus is frightening me a bit, due to its open source origins(and Class1 in source code=D) and CIO asked to avoid using it.
So i decided to write my own simple bus, and encountered problems with adressing and subscribing diffirent types of services(singleton, per session, per call) and also there is a requirement for load balancing.
I found quite elegant solution for this: use "adapters" for bus - wcf services for incapsulating some specific issues of working with services - like loadbalancing. So bus will only see adapters and route messages between them and they will forward messages further. But im concerned about perfomance and whole idea..
Will be very grateful to hear thoughts about all of this stuff=)
PS Bus and adapters use MSMQ for communication between them, but other services can use http,tcp bindings.
PS2 Sorry for my english, its not my native language=)
I'm probably just bringing the dead back but if you'd still like to implement your own bus these links may come in handy (on design level):
http://msdn.microsoft.com/en-us/magazine/cc500646.aspx
http://msdn.microsoft.com/en-us/magazine/cc546553.aspx
I found answers for some my questions about how dynamically register subservices and route messages from client to them by central router service
You could try using "sql service broker" as a service bus
http://javiercrespoalvez.com/2009/03/using-sql-service-broker-in-net.html
The blog post also has links to other service buses you could try.
Thanks for all replyes=) I was able to convince bosses to use NServiceBus. (After creating a working bus prototype on weekend =). Now it fits quite well into the system=)