I have the following problem to solve and I wanted to know what options I have to do it. Basically I have a feed that comes to me via a socket and I need to retransmit it to multiple computers.
First of all I want to make a beta version as fast and simple as possible, for a maximum of 250 connections. Then in the future I will build a full version with an architecture that supports scaling in number of connections and perhaps a little more in the size of packages.
Some more detailed data:
The packages have a weight of 2KB approximately.
At peak moments we will send about 50 packets per second. (1 pack every 20 ms)
In the beta version if a package does not reach one of the consumers its not a problem but in the full version it is.
I would like to push the messages and not have the consumers pull them, but for the beta version its not mandatory.
In the beta version I do not need authentication but in the full version I do.
I was researching and I found that I could use:
Message Queue, ie RabbitMQ
An Api Stream like Twitter
Are there other alternatives? What technology and tools do you recommend me to use for beta version and full version?
Thank you very much
AMQP is a good choice, as you've mentioned.
You also can check out DDS: http://opendds.org/about/dds_overview.html
Related
Please be aware that I am a relative newbie to ActiveMQ.
I am currently working with a small cluster of ActiveMQ (version 5.15.x) nodes (< 5). I recently experimented with setting up the configuration to use "Shared File System Master Slave" with kahadb, in order to provide high availability to the cluster.
Having done so and seeing how it operates, I'm now considering whether this configuration provides the level of throughput required for both consumers/producers, as only one broker's ports are available at one time.
My question is basically two part. First, does it make sense to configure the cluster as highly available AND load balanced (through Network of Brokers)? Second, is the above even technically viable, or do I need to review my design consideration to favor one aspect over the other?
I had some discussions with the ActiveMQ maintainers in IRC on this topic a few months back.
It seems that they would recommend using ActiveMQ Artemis instead of ActiveMQ 5.
Artemis has a HA solution:
https://activemq.apache.org/artemis/docs/latest/clusters.html
https://activemq.apache.org/artemis/docs/latest/ha.html
The idea is to use Data Replication to allow for failover, etc:
When using replication, the live and the backup servers do not share the same data directories, all data synchronization is done over the network. Therefore all (persistent) data received by the live server will be duplicated to the backup.
And, I think you want to have at least 3 nodes (or some odd number) to avoid issues with split brain during network partitions.
It seems like Artemis can mostly be used as a drop-in replacement for ActiveMQ; it can still speak the OpenWire protocol, etc.
However, I haven't actually tried this yet, so YMMV.
I was planning to use out or process distributed caching solution and I am trying infinispan hot rod protocol for this purpose. It performs quite well compare to other caching solutions but I feel it is taking more in network communication than expected.
We have 1000Mbps ethernet network and round trip time between client and server is around 200ms but infinispan hot rod protocol is taking around 7 seconds in transferring an object of size 30 MB from server to client. I feel that I need to do tcp tuning to reduce this time, can someone please suggest me how can I tune tcp to get best performance? On googling I found that send-buffer-size and receive -buffer-size can help in this case but I don't know how and where to set these properties. Can someone please help me in this regard.
Any help in this regard is highly appreciated.
Thanks,
Abhinav
By default, Hot Rod client and server enable TCP-no-delay, which is good for small objects. For bigger objects, such as your case, you might wanna disable it so that client/server can buffer and then send instead. For the client, when you construct RemoteCacheManager, try passing infinispan.client.hotrod.tcp_no_delay=false, and the server needs a similar configuration option too. How the server is configured depends on your Infinispan versions. If using the latest Infinispan 6.0.0 version, you'll have to go to the standalone.xml file and change the endpoint subsystem configuration so that hotrod-connector has tcp-nodelay attribute set to false. Send/receive buffers only apply when TCP-no-delay are disabled. These are also configurable via similar methods, but I'd only do that if you're not happy with the result once TCP-no-delay has been disabled.
Can anyone tell me what is holding us back.
I tried every different php script in front end to send emails. Interspire, Oempro, PHP list, PHPknode. But we are only able to send 5 emails every 2 seconds.
We upgraded our server, Our H/W configuration is good. We have used EXIM, We even tried PMTA. Even though our sending speed does not improved.
Our requirement is to send 200,000 - 300,000 emails a day But we need to send this in peak hours i.e. between 9am to 1pm. We are only able to send 15000 emails in 6-7 hours.
I don't know what is the problem, Why are we not able to send emails quickly. Is it because of the PHP script, MTA or the server h/w configuration.
Can anyone please help me with this problem? Any help will be appreciated.
I can tell you directly that Interspire Emailmarketer is not especially high-performing. I had a similar situation as you do. We had a high-end server machine, with SAS disks, 16 CPU cores and lots of RAM. We had a highly fine-tuned Postfix MTA and MySQL server (spent a few days configuring those). The performance you get matches our experience. The load in our case was entirely in the PHP script, not the database and not in the MTA.
I suspect that the Interspire software is meant for very low-traffic newsletters (where receivers can be counted in the hundreds).
Interspire by default uses a single php process to process the email queue, and thus it's unable to use multi-core machines. There is a paid multi-processing script called MSH addon which takes IEM processing queue and distributes it across several processor cores for massive speed bonus. From the addons website:
MSH is built around "multi processing library", a multi platform
,shared nothing, multi processing framework inspired by Python
multiprocessing module(but very different from api level). It uses
"proc family functions" for process spawning and "soq" for IPC.
Disclaimer: I am one of the developers of MSH addon.
I've been trying to find out ways to improve our nservicebus code performance. I searched and stumbled on these profiles that you can set upon running/installing the nservicebus host.
Currently we're running the nservicebus host as-is, and I read that by default we are using the "Lite" version of the available profiles. I've also learnt from this link:
http://docs.particular.net/nservicebus/hosting/nservicebus-host/profiles
that there are Integrated and Production profiles. The documentation does not say much - has anyone tried the Production profiles and noticed an improvement in nservicebus performance? Specifically affecting the speed in consuming messages from the queues?
One major difference between the NSB profiles is how they handle storage of subscriptions.
The lite, integration and production profiles allow NSB to configure how reliable it is. For example, the lite profile uses in-memory subscription storage for all pub/sub registrations. This is a concern because in order to register a subscriber in the lite profile, the publisher has to already be running (so the publisher can store the subscriber list in memory). What this means is that if the publisher crashes for any reason (or is taken offline), all the subscription information is lost (until each subscriber is restarted).
So, the lite profile is good if you are running on a developer machine and want to quickly test how your services interact. However, it is just not suitable to other environments.
The integration profile stores subscription information on a local queue. This can be good for simple environments (like QA etc.). However, in a highly distributed environment holding the subscription information in a database is best, hence the production profile.
So, to answer your question, I don't think that by changing profiles you will see a performance gain. If anything, changing from the lite profile to one of the other profiles is likely to decrease performance (because you incur the cost of accessing queue or database storage).
Unless you tuned the logging yourself, we've seen large improvements based on reduced logging. The performance from reading off the queues is same all around. Since the queues are local, you won't gain much from the transport. I would take a look at tuning your handlers and the underlying infrastructure. You may want to check out tuning MSMQ and look at the disk you are using etc. Another spot would be to look at how distributed transactions are working assuming you are using a remote database that requires them.
Another option to increase processing time is to increase the number of threads consuming the queue. This will require a license. If a license is not an option you can have multiple instances of a single threaded endpoint running. This requires you shard your work based on message type or something else.
Continuing up the scale you can then get into using the Distributor to load balance work. Again this will require a license, but you'll be able to add more nodes as necessary. All of the opportunities above also apply to this topology.
I need to modify the number of threads available in my Weblogic 10.0 MP2 environment for some perf benchmarking but I cannot seem to be able to find where exactly that option lies.
Can anyone share this info please? thank you.
Weblogic 10 does not use execute thread queues like in previous versions (i.e. Weblogic 8.1 and older)
This concept is now replaced with Work Managers.
These are self-tuned, i.e. WLS will auto-tune the number of threads every 2 seconds based on how it sees the need to increase threads for the application load.
You can confirm this from the console, it will show the increasing number of execute threads as the load increases.
You can use the work manager and constraints to make sure your applications get certain criteria met.
Such as certain web apps or EJBs can get a higher share of threads and so on.
For a quick read see http://www.oracle.com/technetwork/articles/entarch/workload-management-088692.html
and
http://m-button.blogspot.com/2009/02/tuning-default-workmanager-on-weblogic.html
Secondly, are you running in dev mode or production mode.
If dev mode, you can try this cmd line parameter
-Dweblogic.threadpool.MinPoolSize=100
but I am not sure if it will work, so it's better to leave it to Work Managers