We are using a pair of load-balanced Worklight 6.1.0.02.20150520-1015 servers in a production environment to support a mobile app with about 15 ~ 20k queries per day coming through to the Worklight servers adapters.
These adapters calls are not really doing any processing. For the most part they are simply passing http requests along to internal servers located in the same zone as the Worklight server. The internal servers typically respond to requests within 100ms or less.
We are seeing an average of 12 errors per thousand requests in the Worklight logs. They are roughly 2/3 UNEXPECTED_ERROR, 1/3 REQUEST_TIMEOUT, and 1/3 UNRESPONSIVE_HOST. As far as we can see, these requests never even reach the internal servers.
It is as if these requests are queuing up or failing on the Worklight servers somehow.
The adapters typically have these settings;
<loadConstraints maxConcurrentConnectionsPerNode="50" />
<procedure name=... requestTimeoutInSeconds="60" />
What should we be doing to reduce this error rate?
Does it indicate the servers need more memory or processing speed? Do we need to experiment with changing the settings? Or what?
My suggestion would be to open an IBM PMR (support ticket) by your business/dev unit, since this question is not suitable for Stack Overflow (more about infrastructure handling than programming). The support/dev team could then investigate and possibly provide a solution.
Related
Hi I am new to Marklogic and Apache. I have been provided task to use apache as loadbalancer for our Marklogic cluster of 3 machines. Marklogic cluster is currently running on Linux servers.
How can we achieve this? Any information regarding this would be helpful.
You could use mod_proxy_balancer. How you configure it depends what MarkLogic client you would like to use. If you would like to use the Java Client API, please follow the second example here to allow apache to generate stickiness cookies. If you would like to use XCC, please configure it to use the ML-Server-generated or backend-generated "SessionID" cookie.
The difference here is that XCC uses sessions whereas the Java Client API builds on the REST API which is stateless, so there are no sessions. However, even in the Java Client API when you use multi-request transactions, that imposes state for the duration of that transaction so the load balancer needs a way to route requests during that transaction to the correct node in the MarkLogic cluster. The stickiness cookie will be resent by the Java Client API with every request that uses a Transaction so the load balancer can maintain that stickiness for requests related to that transaction.
As always, do some testing of your configuration to make sure you got it right. Properly configuring apache plugins is an advanced skill. Since you are new to apache, your best hope of ensuring you got it right is checking with an HTTP monitoring tool like WireShark to look at the HTTP traffic from your application to MarkLogic Server to make sure things are going to the correct node in the cluster as expected.
Note that even with the client APIs (Java, Node.js) its not always obvious or explicit at the language API layer what might cause a session to be created. Explicitly creating multi statement transactions definately will, but other operations may do so as well. If you are using the same connection for UI (browser) and API (REST or XCC) then the browser app is likely to be doing things that create session state.
The safest, but least flexable configuration is "TCP Session Affinity". If they are supported they will eliminate most concerns related to load balancing. Cookie Session Affinity relies on guarenteeing that the load balencer uses the correct cookie. Not all code is equal. I have had cases where it the load balancer didn't always use the cookie provided. Changing the configuration to "Load Balancer provided Cookie Affinity" fixed that.
None of this is needed if all your communications are stateless at the TCP layer, the HTTP layer and the app layer. The later cannot be inferred by the server.
Another conern is if your app or middle tier is co-resident with other apps or the same app connecting to the same load balancer and port. That can be difficult to make sure there are no 'crossed wires' . When ML gets a request it associates its identity with the client IP and port. Even without load balencers, most modern HTTP and TCP client libraries implement socket caching. A great perfomrance win, but a hidden source of subtle random severe errors if the library or app are sharing "cookie jars" (not uncomnon). A TCP and Cookie Jar cache used by different application contexts can end up sending state information from one unrelated app in the same process to another. Mostly this is in middle tier app servers that may simply pass on requests from the first tier without domain knowledge, presuming that relying on the low level TCP libraries to "do the right thing" ... They are doing the right thing -- for the use case the library programmers had in mind -- don't assume that your case is the one the library authors assumed. The symptoms tend to be very rare but catastrophic problems with transaction failures and possibly data corruption
and security problems (at an application layer) because the server cannot tell the difference between 2 connections from the same middle tier.
Sometimes a better strategy is to load balance between the first tier and the middle tier, and directly connect from the middle tier to MarkLogic.
Especially if caching is done at the load balancer. Its more common for caching to be useful between the middle tier and the client then the middle tier and the server. This is also more analogous to the classic 3 tier architecture used with RDBMS's .. where load balancing is between the client and business logic tiers not between business logic and database.
Let's assume I have a cluster of 2 worklight servers sharing the same WL runtime.
On that runtime, I've installed a application with a adapter that is a create event source function.
Just like this IBM article.
https://www.ibm.com/developerworks/community/blogs/worklight/entry/configuring_a_polling_event_source_to_send_push_notifications?lang=en
My question is, what will happen on a cluster environment.
Will repeated work ensue?
By other words, would my two WL Servers will be pooling for events?
Or perhaps that functionality is writing a task on the WL DB that the WL Servers poll regularly to check for work if no instance is taking care of it, so that only a server at a time would be "the event source"?
I'm working with IBM Worklight 6.2 and Websphere Liberty Profile 8.5.5
Thanks in advance!
Here's my attempt to answer this after some consultation:
My question is, what will happen on a cluster environment. Will
repeated work ensue? By other words, would my two WL Servers will be
pooling for events?
While the Worklight Servers share the same runtime, they are still considered as 2 instances. This means that each of them will attempt to perform a polling action. This is considered OK.
However, it is important to note that the backend system that is being polled should likely be smart enough to handle such a situation where 2 polling attempts are done for the same message.
If the backend doesn't know how to handle polling properly, the same message can be pulled more than once. This is true even of you have a single eventsource running. So this is something to keep in mind.
A decade and more in the past I studied and used IBM's MQSeries and Websphere MQ software. It seemed to be a great solution for connecting two applications in different companies at different locations: the app at each company could drop off a message with MQSeries on the local machine, MQSeries would transport it to the machine in the other company, and the app on that side would pick up the message locally.
Fast forward to today: I no longer work for IBM, but I'm trying to solve a similar problem. My app needs to send a few messages a day, each a few MB or less, to an app at a remote company, and receive a similar number of somewhat smaller replies.
Is message queuing middleware still a good solution to this architectural need? I've been trying to prototype this with RabbitMQ, but the above seems to be an abnormal thing to do with RabbitMQ. Am I barking down the wrong rabbit hole?
sure it can be - if the remote company intends to provide the same service for others and not just you.
perhaps WMQ low latency is what you need, since there is no server required.
- WebSphere MQ Low Latency Messaging varies from conventional WebSphere products such as WebSphere MQ, WebSphere Message Broker, and WebSphere Application Server in that there is no installed and configured product infrastructure such as queue managers, message brokers, or application servers. Thus, there are no specific product components to be monitored, measured, and managed.
http://www-03.ibm.com/software/products/en/wmq-llm
http://pic.dhe.ibm.com/infocenter/wllm/v2r5/index.jsp?topic=%2Fcom.ibm.wllm.doc%2Fintroductiontowebspheremqlowlatencymessaging.html
Of course, a simple RESTful web interface might be able to provide the same functionality.
I do not recommend writing TCP socket applications - why do all that mid-weight lifting when there are so many products out there that will do the heavy and mid-weight lifting for you? You want to only do lightweight lifting - Send the request, get the response - 6 2 and even, over and out.
You need to draw up your list of requirements reagrding:
- reliability - how critical is it if a request or response gets lost?
- recoverability - can in-flight messages be recovered and resent if the application(s) crash?
- round trip time - one side of latency
- 1:1 service? many-to-one?
I hope this helps.
Dave
WebSphere MQ can still solve your problem... I think the scenario is point to point communication with Request-Response scenario. You can use some relatively new stuff like JMS which integrate well with your application.
But if you are very sure that its only 2 applications that will communicate with each other and there is no network issue that will crop up, you can go in for simple socket communication.
The other way to solve the problem is to share a common database between the 2 applications.
Even if this question is a year old, I am still searching a good answer for this question. I appreciate any information that will lead me to fully understand this issue regarding low performances of communicating web services hosted on the same machine.
I am currently developing a system with several WCF Web Services that communicate intensively.
They are running under IIS7, on the same machine, each service being in a different Application Pool, with multiple workers in the Web Garden.
During the individual evaluation of each Web Service, I can serve 10000-20000 requests per minute, quickly and without any issues for the resource consumption (processor and memory).
When I test the whole system or just a subsystem formed by two Web Services I can't serve more than 2000 requests/minute.
I also observed that communication time between Web Service is a big issue (sometimes more than 10 seconds). But when testing with only 1000 requests per minute everything goes smoothly (connection time of no more than 60 ms).
I have tested the system both with SOAPUI and JMETER, but the times were computed based on system logs, not from the testing tools.
Memory and network aren't an issue (they are used very little).
Later on, I have tested the performance of 2 communicating WCF web services, hosted on two server and on the same server. It again seems that there is a bottleneck when the services are on the same machine, lowering the number of connection with from ten thousands to thousands; again, no memory or processor limiting.
As a note, I am working with quite big data in some cases and some of the operations needed are long ones.
I used perf.mon to see what's going on, for memory, processes, webservice, aspnet, etc. but I didn't see anything that could indicate what it's going wrong.
I also tried all the performance settings and tuning options I could find on the Internet.
Does someone know what can be wrong? Why the communication between Web Services could last so long? Why the Web Service which serves as an entry point in the system can accept 10000 requests/minute when is tested alone, but when communicating with another Web Service barely accepts 2000?
It's an IIS7 problem? Could my system perform better if each Web Service will be deployed on a different server?
I want to understand better how things internally function (IIS and WCF services) to improve performances for current and future systems.
You could try to collect data from WCF performance counters : concurrent calls, instances, duration, ... In addition, WCF throttling provides some properties that you can use to limit how many instances or sessions are created at the application level. Performance of the WCF service can be improved by creating proper instance.
Finally, in load testing, there are many configuations to apply to different component : max concurrent http connection, IIS limits, having many load clients... You load test is invalidated because of this.
Recently I gone through a an article explaining potentiality of YAWS server and the number of requests it processes per second. It was mentioned that YAWS can handle 80K requests per second and it also run in multi threaded environment to improve request processing limit.
How can we compare IIS, Apache with YAWS? Which one will process maximum requests? Can I find any comparisons somewhere?
Check this link out:http://www.sics.se/~joe/apachevsyaws.html Link to Yaws vs apache
You see that Yaws handles 80000 concurrent requests (and continuing) while apache fails at around 4000 connections. This is because Yaws runs on the Erlang/OTP VM. Processes belong to this machine and not the operating system. Erlang has been highly customised for concurrent programming. Infact, other erlang web applications like:mochiweb,webmachine, e.t.c are much more powerful than apache when it comes to handling many concurrent requests. Yaws web server scales better than any web server i know of today. With the ability to create appmods, you can create Erlang Applications that communicate over http protocol, making use of the power of yaws.
Yaws home page is: http://yaws.hyber.org/. Actually, Yaws gets its power from OTP (Open Telecom Platform). This set of powerful libraries found at http://erlang.org/, has the most advanced design patterns such as fail over systems, supervision trees, finite state machines, event handlers, e.t.c, You should actually start using erlang for your next web application!!!!