I have created publisher and subscribers in RTI_CONNEXT which are able to communicate concurrently over a particular domain. I want publishers with multiple writers running concurrently. Do I need to use threads or RTI has provision of the same.
Also later I want to include publisher and writer into same participant. How to do that ?
Code will be appreciated. Thanks
Related
I need to run multiple instances of the same service, with the same database, for redundancy reason.
I found some question about "Hangfire multiple instances" but for a differenct purpose then mine: usually about running multiple instances for different tasks on the same database, or similar to this.
I need to know if there are problems of concurrency when 2 or more instances of Hangfire use the same Database (we want to use MongoDB) and if this is the solution to make the service resilient.
The goal is to have instance that take care of all the jobs when another instance goes down.
Any suggestion wellcome for covering this scenario.
In our environment, we have a replica set used by about 10 Hangfire servers. If there are multiple Hangfire servers servicing the same queue, it means they will share the load and whichever Hangfire server checks the queue first, picks up the job and continues. If you remove all but 1 server, the jobs will continue (as long as there are enough workers otherwise they will remain queued until a worker is available).
To answer your question, yes, you can have 2 or more Hangfire servers using the same MongoDB. MongoDB provides multi-threading support so its safe to have various servers accessing the same database backend. If you have two servers, both will be active and if one instance goes off line, other instance (based on queues) will continue to process the jobs in queue.
Keep in mind, Hangfire servers processes the jobs in Specific Queues. If both servers are part of the same queue then you are load balancing the jobs among the two servers. If they are part of different queues, then you read about that scenario where each Hangfire instance processes different jobs (because they are part of different queues).
Read about configuring Job Queues here
I am making one session per connection per thread to activeMQ cluster. But I want to consume from hundreds of destinations. I do understand that if I only have one thread ( one session ), I can't consume messages from these destinations concurrently. I don't want to do that either. But I want to have hundreds of consumers per session which will in-turn be associated to hundreds of different destinations, is this a viable approach? Please also provide the reason of viability or non-viability.
PS : I don't want to do any heavy processing on the messages, so that's why only 1 thread.
A session is not bound to a single thread - threading is a separate chapter. You can use a session in multiple threads (not recommended) and multiple sessions in a single thread. The session construct is more a thing to control transactions - i.e. commit and rollback messages in a transaction.
Anyway, you can use a single consumer to read multiple destinations. Simply put the destinations in a list like: "my.first.queue,my.other.queue,my.last.queue". You can also read a queue using wildcards - "my.>". would use all queues above.
This way, you can use a single thread and a single session to read from a large number of queues.
We're a licensed product using NServiceBus as the messaging framework in our federated system.
Looking for opportunities for using it in a new feature-
Is there a way to build a multi-site system (scaled-out), at which each site/node produces and distributes messages to workers located on several nodes/sites?
Each distributor and worker can be hosted at it's own site (same LAN), and each site can go down at any point. All distributors and workers should be symmetric.
At first it looked like a classical "many producers to many competing consumers" problem. but I can't find a built-in way to achieve it with NServiceBus as, from what I saw, each worker can send health sings to only one distributor (I might be wrong about that).
Another issue I came against is with having a centralized RavenDB instance holding the distributor subscriptions. Having the RavenDB in it's own "availability group" requires additional resources. Is there a way to host the RavenDB instance under the same sites, having their data replicated while each site is using it's local DB instance? This will also bind the HA of the distributors to their subscription DB's.
Reading this discussion- http://tech.groups.yahoo.com/group/nservicebus/message/18412
It seems that NServiceBus requires a cluster to keep HA of the published data. But why can’t the distributor wait for an acknowledge that the msg was successfully consumed & processed by the worker, and keep retrying to publish it to the same or a different worker? This way, even if the VM went down with data in the queues, the data will be sent to another node which is available at that time.
Edit: Tried to ask the same at NServiceBus official Yahoo group but keep getting pythonError within the yahoo group.
Thanks in advance,
Rami Prilutsky, dbMotion
after reading and searching about OS and process and threads, I checked on wiki and it said,
A computer program is a passive
collection of instructions, a process
is the actual execution of those
instructions. Several processes may be
associated with the same program; for
example, opening up several instances
of the same program often means more
than one process is being executed.
Now is it possible for a program to have more than one process and I am not including the possibility of running more than one instance of the same program. I mean one instance of one program is running, is it possible for a program to have more than one process?
If yes, how? If no, why not?
I am a newbie in this, but damn curious :)
Thanks for all your help..
Yes, fairly obviously - you can run two or more copies of most programs - I routinely have about 5 copies of vim running, and each of those is a separate process. As to how, the OS loads the executable file, creates a process and then tells that process to start executing the file contents.
It is most definitely possible but a desktop application might not be a good example and I think this is the source of your confusion.
Consider a webserver instead (NginX or Apache). There is one master process and multiple worker processes at work. The master process "accpets" the work , so to speak, and delegates it to the workers. Both NginX and Apache could be configured to any number of worker processes.
At our company we are in the business of delivering a SaaS that helps businesses have an online chat with their visitors via their websites. The back-end part of our system has multiple "service"es communicating with each other to accomplish the task. Each service has multiple instances running.
Please consider the following questions in the context of multiple publications from a scaled out publisher (using DB subscription storage) and multiple subscriptions with scaled out subscribers (using distributors) where installs and uninstalls happen regularly for initial deployments, upgrades, etc. using automated MSI's.
Using DB subscription storage, what happens if the DB goes down? If access to the Subscription DB is required in order to Publish a message, how will it be delivered? Will it get lost? Will the call to Bus.Publish throw an exception?
Assuming you need to have no down-time deployments: What if you want to move your subscription DB for a particular publication to a different server? How do you manage a transition like this?
Same question goes for a distributor on the subscriber side: What if you want to move your distributor endpoint? One scenario I can think of is if you have multiple subscriptions utilizing a single distributor machine, it might be hard if you want to move some of them to another distributor server to reduce load.
What would the install/uninstall scenarios look like for a setup like this (both initially, and for continuous upgrades)? It seems like you would want to have some special install/uninstall scripts for deployment of the "logical publication" and subscription DB, as well as for the "logical subscriptions" and the distributors. The publisher instances wouldn't need any special install/uninstall logic (since they just start publishing messages using the configured subscription DB, and then stop when they are uninstalled). The subscriber worker nodes wouldn't need anything special on install other than the correct configuration of the distributor endpoint, but would need uninstall logic to make sure they are removed from the distributors list of worker nodes.
Eventually the publisher will fail and the messages will build up in the internal queue. You will have to plan the size of disk you need to handle this based on the message size and how long you want to wait for a DB to come up. From there it is based how much downtime you can handle. You can use DB mirroring or clustering to make the DB have less downtime.
Mirroring and clustering technologies can also help with this. Depends on if you want to do manual or automatic failover and where your doing it(remote sites?).
Clustering MSMQ could help you here. If you want to drop a distributor and move it within a cluster you'd be ok. Another possibility is to expose your distributors via HTTP and load balance them behind either a software or hardware load balancing solution. Behind the load balancer you'd be more free to move things around.
Sounds like you have a good grasp on this one already :)
To your first question, about the high availability of the subscription DB, you can use a cluster for failover. If the DB is down, then the Bus.Publish will throw an exception, yes. It is recommended to keep the subscription DB separate from your applicative DB to avoid having to bring it down when upgrading your app. This doesn't have to be a separate DB server, a separate DB on the same DB server will be fine.
About moving servers, this is usually managed at a DNS level where for a certain period of time you'll have both running, until communication moves over.
On your third question about distributors - don't share a distributor between different publishers or subscribers.
As a rule of thumb, it is recommended to not add/remove subscribers when doing these kinds of maintainenance activities. This usually simplifies things quite a bit.