Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am a newbie who just started reading about distributed system.
I am wondering what are some use cases for a distributed queue as opposed to queues on each machine.
For example, how RabbitMQ is used among different web servers. How is it used for example in load balancing?
We typically use distributed queues when the up front cost of processing some task is too expensive or otherwise unnecessary. For example, when you upload a video to YouTube, typically there's some processing of the video that occurs before it's displayed on the site. In the modern web, it can be unacceptable for users to have to wait while that processing occurs. So, the video can be stored and a task put on a queue so that processing can take place later. Then, other machines that are polling the queue can process the video at their leisure. This means the user doesn't have to wait for their video to be processed before they can continue on doing other things on the site. It also critically allows for a buffer for periods of high throughput. If users are uploading videos faster than they can be processed by YouTube's servers, the queue grows independently of the back end's ability to process items.
Another consideration is that the distributed nature of the queue allows for fault tolerance. In the YouTube example, that allows the website to respond to the user, assuring the user that their video will eventually be processed. Typically distributed queues have configurable replication levels, where once an item is put on the queue it's guaranteed to be replicated on n nodes and therefore is unlikely to be lost.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I'm researching with a friend about the idea of using an event queue/stream system, such as kafka or rabbitmq, as a way to store the adverts in a queue instead of a traditional database.
The required system would need to provide a stream of events holding various fields, be filterable and searchable. Allow the stream to hold the events indefinitely or for a certain amount of time (for example to let the adverts expire). We are just not sure whether to go for a message queue/event stream, or whether the traditional database is the way to go.
Has anybody experience with this, would you recommend to investigate one system over another?
Kafka would support the usecase, as you can treat it not as messaging queue, but as a transaction log:
you can re-read the same message multiple times with different consumers
the messages would be persisted until they expire (configurable on server) or until they are compacted
(data propagation) there are tools such as mirror maker (or own streaming applications) to replicate data among data centres (or some part of them, e.g. if you decide to put some attributes in one topic, and other in another)
I do not know if generic messaging solution (like RabbitMQ) would suit you, as messages consumed would disappear, so you would need to re-publish them to keep them (and in case of multiple consumers, you'd need to use RMQ logic like fanout exchanges that messages get multicasted to multiple queues, each per consumer).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I know that a
Multithreaded processing can simultaneously run multiple procedures. For example, a word processor application can check spelling as a separate task while you continue to work on the document. Because multithreaded applications divide programs into independent tasks, they can substantially improve performance in the following ways:
• Multithreaded techniques can make your program more responsive because the user interface can remain active while other work continues.
• Tasks that are not currently busy can yield processor time to other tasks.
• Tasks that use a lot of processing time can periodically yield to other tasks.
• Tasks can be stopped at any time.
• You can set the priority of individual tasks higher or lower to optimize performance.
But i want to know
when i can use multithreading in vb.net ? any examples or an usual app use it
and what is the different if we use it in desktop apps or web applications
thanks .
I think that multithreading is often used in image processing and can often be done in parallel along with the rendering of an animation also, in GUI programming it often helps to have at least two threads when doing something slow, e.g. processing large number of files.
I thought it was the same in desktop apps and web applications though.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I am new to a desktop application and seeing that there are threads used everywhere.
I don't understand why we are using it here and not in webapplications.
We can directly call the method instead of threads.
'Threads used everywhere' is probably an ominous sign of problems to come and possibly past incompetence, but the main reason people use multiple threads in desktop applications is to allow long-running things to happen without freezing the user-interface of the application.
A conventional Windows GUI app has a 'main thread' which is responsible for processing messages received from the OS. These messages are used for all mouse/keyboard interaction, so the moment you stop processing them, the application stops responding (after a few seconds of this, Windows itself will grey-out the app and put 'stopped responding' in the title bar.)
To avoid this ugliness, people move activities which take a long time (either because they need a lot of CPU, or because the may be waiting for I/O (disk, network, etc)) onto other threads. This means that the main thread is free to keep pumping messages.
You do not start the main thread explicitly - Windows gives it to you when the app starts.
You need to be very careful about interaction between background threads and the main thread - there are a bunch of rules and restrictions, and failing to follow them will give you the kind of problems which may not show up immediately, consistently or on your machine at all. That doesn't mean they're not there...
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
So I have tinkered with WCF in the past, and I know that you can use it not only for web services but also for persistent TCP client/server applications, but I never really put it through significant use.
I am looking into creating a client/server application for .NET that will run locally, that is both the server and multiple clients will run on the same machine. There will not be a huge number of simultaneous connections, maybe 50 at the most. The amount of data that is transferred per request will not be very much, mostly small objects around 40-100 bytes. There will be a lot of different object types, though, so I'd prefer a solution that makes the serialization simple.
The biggest concern is that I will need to send many packets back and forth in quick succession, and I can't afford much latency. Ideally I will need a round-trip of a 100 byte packet to be less than 1ms (assuming no processing time is used processing the data on the server).
I know this is possible with a native Winsock/IOCP-based solution as I have a solution that works for another project, but I have no idea if this is something that WCF can handle, as I have no idea how much processing overhead is involved in each request/response, or what kind of asynchronous IO it uses.
So, do you think WCF could work for this, or will I need to create a lower-level socket-based solution?
Yes, use NamedPipes for inter-process full duplex communication. Very fast and does not go over tcp. Only works if everything is running on the same machine.
This article should help.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
We have to implement a queuing system for our ROR Application.We have evaluated the following options:
Amazon SQS: High Availability but relatively slow performance.Requires constant poling.
CloudAmqp: Looks promising but doubtful about the support.
RabbitMq setup on EC2: Needs user bandwidth to manage the setup,may result in downtime if some issue arises in the setup.
Right now there won't be any dedicated team/person to manage the setup full-time so implementing our own RabbitMq setup on Ec2 may result in downtime in case something goes wrong.
I want to know considering the situation which is our best option?
I use SQS and I am happy with it; I don't worry about the support aspect, because I also don't have time to deal with setting up my own server and supporting myself when I can pay AWS pennies to do it for me.
If you don't want to poll constantly, considering pairing up your SQS queue with an SNS topic and it can do push notifications to your application instead. Don't know the nature of your application, but its something to look into. http://aws.amazon.com/sns/
ALso keep in mind the slow performance of SQS (relative to Rabbit) is not apples to apples. SQS is redundant and distributed, a single instance of RabbitMQ on a single box is not; can your application deal with the queue not being available for a period of time?
At CloudAMQP all our servers are redundant, each cluster has at least two instances in different availability zones. For support we have email support 24/7, as we have staff in different timezones. We do have phone support for our largest plans too.