How to estimate resources (cpu, memory, for rabbitmq? On the manufacturer's website I found the short information https://www.rabbitmq.com/production-checklist.html
Use PerfTest to simulate your expected and potential future workloads.
Related
If I were to design a huge distributed system whose throughput should scale linearly with the number of subscribers and number of channels in the system, which would be better ?
1) Redis Cluster (only for Redis 3.0 alpha, if its in cluster mode, you can publish in one node and subscribe in another completely different node, and the messages will propagate and reach you). The complexity of Publish is O(N+M), where N is the number of subscribed clients and M is the number of subscribed patterns in the system, but how does it scale when in a Redis Cluster ? I accept educated guesses on this.
2) ZeroMQ since 3.x, it does server-side filtering, so it also has some time complexity there, but I have not seen anything about it in the documentation. If I wanted to scale it, I could just have lots of servers publishing to whatever channels, and each subscriber would connect to all the servers, and subscribe for the desired channel. That seems nice.
So which of those is better for horizontal scaling of a huge publisher system ? What are other solutions I should look into ? Remember, I want to minimize latency and throughput, but being able to scale horizontally.
You want to minimize latency, I guess. The number of channels is irrelevant. The key factors are the number of publishers and number of subscribers, message size, number of messages per second per publisher, number of messages received by each subscriber, roughly. ZeroMQ can do several million small messages per second from one node to another; your bottleneck will be the network long before it's the software. Most high-volume pubsub architectures therefore use something like PGM multicast, which ZeroMQ supports.
In Redis, like in ZeroMQ, the bottleneck will be the network. Redis can reach millions of messages per second, at least as much if not more than ZeroMQ.
You should be aware that the current implementation of Redis Cluster distributes PUBLISH messages across all cluster nodes using the inter-node bus. This approach assumes that PUBLISH is extremely cheap on Redis (as explained in this issue on Github).
However, there is a small overhead involved which is inter-node communication. As you scale up this overhead will be more significant. There is another Redis Cluster implementation I'm aware of - please note it's a commercial one - in which channels or patterns are distributed across cluster nodes in a similar fashion to the way Redis keys are distributed. At least according to the vendor, this should save the overhead of inter-node communication and increase performance, but I have not benchmarked it myself.
I would like to measure the memory consumption for one active Apache connection(=Thread) under Ubuntu.
Is there a monitoring tool which is capable of doing this?
If not, does anyone knows how much memory an Apache connection roughly needs?
Activate the mod_status module, you'll get a report on /server-status page, there is a more parseable version on /server-status?q=auto. If you enable ExtendedStatus On you will have a lot of information on processes and threads.
This is the page used by monitoring tools to track a lot of stats parameters, so you will certainly find the one you need (edit: if it is not memory...) . Be careful with security/access settings of this file, it's a nice tool to check how your server respond to DOS :-)
About memory you must note that Apache loves memory, how much memory per process depends on a lot of things (number of modules loaded - check that you need all the ones you have, number of virtualHosts, etc). But on a stable configuration it does not move a lot (except if you use PHP scripts with high memory limit usage...). If you find memory leaks try to limit the number of requests per process MaxRequests (apache will kill him and put a new one).
edit: in fact not a lot of memory info in the server-status. About monitoring tools, any tool using SNMP MIB-II can track memory usage per process, with average/top/low values for the different childs (Cacti, Nagios, Munin, etc) if you had a snmpd daemon. Check this excellent Munin example. It's not a tracking of each apache child but it will give you an idea of what you can track with these tools. If you do not need a complete monitoring system such as Nagios or Centreon, with alerts, user managmenent, big networks (and if you do not have a lot of days for books reading) Munin is, IMHO, a pretty tool to get monitoring reports quite fast.
I'm not sure if there are any tools for doing this. But you could estimate it yourself. Start apache and check how much memory it uses without any sessions. Than create a big number of sessions and check again how much memory it uses.
You could use JMeter to create different workloads.
Not a glassfish specialist here but in a project where GlassFish is to be used as enterprise service bus, and I am sort of on the receigving end and a little no agreeing with the architecture team.
What are practical limits on throughput (messages per second) and message size for GlassFish? Just as house numbers for a decent modern dedicated server. Asking because the architectural proposal on my desk is IMHO ridiculous - but I have no idea about the limits of GlassFish.
You really should run your own benchmark yourself (on your hardware, with your own settings) but to my experience, GlassFish ESB is pretty efficient, and reliable.
Just in case, here are some resources that might give you some ideas:
Benchmarking BPEL Service Engine
BPEL SE Scalability Experiments/Evaluations
Benchmark Glassfish ESB April 2010
But if you need to convince someone, you should IMO come back with demonstrable facts.
How would you test a ‘Mobile’ version of IM? What are some key differences between testing this version versus a desktop application?
First of all you need to account for unstable connectivity - the program should be ready to deal with temporary and permanent network coverage losses.
Resources consumption (memory footprint, network bandwidth and CPU time) is also to be considered. Noone wants a program that drains the battery in an hour, occupies all memory or causes a giant bill to the owner.
Both memory requirement and network bandwidth has to be kept to as low as possible...
adding to sharptooth's answer, notifications on sending/receiving a message. Support for voice chat/video chat via available hardware :)
we are evaluating ActiveMQ and I whould like to know what you think about it. Any feedback is welcome.
EDIT:
actually, I need to know things like: how does it scale? What about stability? Is it stable enough for production environments? Memory consumption, etc.
Its been used in thousands of production environments, some examples are Sabre Holdings (travelocity,lastminute.com,etc),FAA,CERN,JPL, etc - so its proven to be robust and perform:
Sabre -32k transactions per second -
that's reliable performance
A large retailer (can't name them) -
10k+ stores, all connected with
real-time updates over ActiveMQ - that's scalability
Here's my opinion & experience, for what its worth.
We have been using ActiveMQ in production for almost 2 years. We did have a couple of issues with ActiveMQ 5.1 that we had work arounds for, but since ActiveMQ 5.3 (we are using SNAPSHOT from July) we have not had any issues and have been happy. Our system pushes an average of 60 msgs/sec through the system with peak volumes > 700 msgs/sec. We run a single active/passive broker pair with persistence journaling via NFS mount to a dedicated NAS device.
--Steven
We have had customers move from ActiveMQ to FioranoMQ - they had the following problems that they reported - found Active MQ configuring very cumbersome/difficult to the point of giving up; and stability could not be achieved; support and documentation are a big issue as he said he can't get a working answer from them how to solve the problem which means even if they paid for support it might not help. Data loss prevention configs for non-persistant/persistant and stability are big concerns and ofcourse limitations on speed. Failure under peak conditions - eg. when message-rate
spikes.