We are currently using JAX-RS 2.0 jersey on WebLogic for hosting restful web services . We are observing very high heap memory utilization in the benchmarks that keep increasing with time. Even after benchmark is over the heap memory allocated does not get released even after I hit perform GC on jconsole. When I analyze the heap dump with MAT I see ~99% of the heap is consumed by oracle.j2ee.ws.server.jaxrs.dms.monitoring.internal.DmsApplication. I un-targetted DMS from the managed server but still the same behavior.
A Little bit of analysis of dominator tree in heap dump shows that every request is being tracked by the listener. The weblogic.jaxrs.monitoring.JaxRsRequestEventListener is mapped to oracle.j2ee.ws.server.jaxrs.dms.monitoring.DmsApplicationEventListener.
Am I understanding this correctly? Does JAX-RS jersey maps to DMS request event listener internally. How this can be configured correctly so we don't face this memory issue.
I think you need to look at your diagnostic module in weblogic. Look at watches & notifications
Related
I have three pods ActiveMQ 5.16.3 pods (xlarge) running inside a Amazon MQ. I am encountering some memory issues where the pods are consuming a lot of memory in general like the traffic is comparable to large (instance type) only but it is still hitting 60-70% heap usage sometimes.
To debug the issue I need to take the heap dump from Amazon MQ. Any idea on how to do that?
I have a question concerning the scalability within a microservice architecture:
Independent from the inter service communication style (REST HTTP or
messsage based), if a service scales, which means several replicas of
the service are going to be launched, how is a shared main memory
realized? To be more precise, how can instance1 access the memory of
instance2?
I am asking this question because a shared non in-memory database between all instances of a service can be way to slow in read and write processes.
Could some expert in designing scalable system architecture explain,
what exactly is the difference in using the (open source) Redis
solution or using the (open source) Hazlecast solution to this
problem?
And as another possible solution: Designing scalable systems with Rabbitmq:
Is it feasible to use message queues as a shared memory solution, by
sending large/medium size objects within messages to a worker queue?
Thanks for your help.
several instances of the service are going to be launched, how is a shared main memory realized? To be more precise, how can instance1 access the memory of instance2?
You don't. Stateless workload scales by adding more replicas. It is important that those replicas are in fact stateless and loosely coupled - shared nothing. All replicas can still communicate with an in-memory service, or database, but that stateful service is it's own independent service (in a microservice architecture).
what exactly is the difference in using the (open source) Redis solution or using the (open source) Hazelcast solution to this problem?
Both is a valid solution. Which is best for you depends on what libraries, protocols, or integration patterns is best for you.
Is it feasible to use message queues as a shared memory solution, by sending large/medium size objects within messages to a worker queue?
Yes, that is perfectly fine. Alternatively you can use a distributed pub-sub messaging platform like Apache Kafka or Apache Pulsar
I'm struggling with using jMeter to test my .NET web app. I'm running jMeter locally against a staging environment in Azure for my app. Hitting some endpoints, I get:
java.net.ConnectException: Connection timed out: connect
Which tells me it's something happening on my end, not caused by my app. My app shows no errors and is serving requests at this magnitude with ease.
In this particular test, I have 300 threads with a ramp-up of 10 seconds, repeated 3 times.
What can I do to diagnose further? Is there some kind of limit being imposed client-side?
JMeter default configuration is not suitable for producing high loads, you may use it for tests development and debugging only. When it comes to running the load test you need to increase at least Java Heap Space allocated to JMeter (it is only 512Mb by default, I think browser which you're using to read this page consumes twice as more).
You should be also running your test in non-GUI mode as JMeter GUI is not designed to correctly display information when more or less immense load is being generated, you can use it up to ~50 threads maybe.
More information:
JMeter Best Practices
9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
I would also suggest keeping an eye on your load generator health during the test and collect information regarding CPU, RAM, Swap, Disk, Network usage as well as some JVM metrics like Heap usage, Garbage collections, etc. so you could see whether your JMeter instance is configured good enough, if there is enough headroom w.r.t. hadrware resources, etc. Of course doing it on server side is a must. You can use PerfMon JMeter Plugin to collect this information and plot results along with other test metrics so you could correlate the values and identify the cause. See How to Monitor Your Server Health & Performance During a JMeter Load Test for plugin configuration and usage instructions
I am new in
Apache ZooKeeper : ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
Apache Mesos : Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers.
Apache Helix : Apache Helix is a generic cluster management framework used for the automatic management of partitioned, replicated and distributed resources hosted on a cluster of nodes.
Erlang Langauge : Erlang is a programming language used to build massively scalable soft real-time systems with requirements on high availability.
It sounds to me that Helix and Mesos both are useful for Clustering management System. How they are related to ZooKeeper? It'd better if someone give me a real world example for their usage.
I am curious to know How [BOINC][1] are distributing tasks to their clients? Are they using any of the above technologies? (Forget about Erlang).
I just need a brief view on it :)
Erlang was built by Ericsson, designed for use in phone systems. By design, it runs hundreds, thousands, or even 10s of thousands of small processes to handle tasks by sending information between them instead of sharing memory or state. This enables all sorts of interesting features that are great for high availability distributed systems such as:
hot code reloading. Each process is paused, it's relevant module code is swapped out, and it is resumed where it left off, so deploys can happen without restarting or causing significant interruption.
Easy distributed messaging and clustering. Sending a message to a local process or a remote one is fairly seamless in most instances.
Process-local GC. Garbage collection happens in each process independently instead of a global stop-the-world even like java, aiding in low-latency results.
Supervision trees and complex process hierarchy and monitoring/managing.
A few concrete real-world examples that makes great use of Erlang would be:
MongooseIM A highly performant and incredibly scalable, distributed XMPP / Chat server
Riak A distributed key/value store.
Mesos, on the other hand, you can sort of think of as a platform effectively for turning a datacenter of servers into a platform for teams and developers. If I, say as a company, own a datacenter with 10,000 physical servers, and I have 1,000 engineers developing hundreds of services, a good way to allow the engineers to deploy and manage services across that hardware without them needing to worry about the servers directly. It's an abstraction layer over-top of the physical servers to that allows you to share and intelligently allocate resources.
As a user of Mesos, I might say that I have Service X. It's an executable bundle that lives in location Y. Each instance of Service X needs 4 GB of RAM and 2 cores. And I need 8 instances which will be attached to a load balancer. You can specify this in configuration and deploy based on that config. Mesos will find hardware that has enough ram and CPU capacity available to handle each instance of that service and start it running in each of those locations.
It can handle a lot of other more complex topics about the orchestration of them as well, but that's probably a bit in-depth for this :)
Zookeepers most common use cases are Service Discover and configuration management. You can think of it, fundamentally, a bit like a nested key value store, where services can look at pre-defined paths to see where other services currently live.
A simple example is that I have a web service using a shared database cluster. I know a simple name for that database cluster and where the configuration for it lives in zookeeper. I can look up (or repeatedly poll) that path in zookeeper to check what the addresses of the active database hosts are. And on the other side, if I take a database node out of rotation and replace it with a new one, the config in zookeeper gets updated with the new address, and anything continually looking at it will detect this change and change where it's connected to.
A more complex use case for zookeeper is how Kafka uses it (or did at the time that I last used Kafka). Kafka has streams, and streams have many shards. Each consumer of each stream use zookeeper to save checkpoints in each shard after they have read and processed up to a certain point in the stream. That way if the consumer crashes or is restarted, it knows where to pick up in the stream.
I dont know about Meos and Earlang language. But this article might help you with Helix and Zookeeper.
This article tells us:
Zookeeper is responsible for gluing all parts together where Helix is cluster management component that registers all cluster details (cluster itself, nodes, resources).
The article is related to clustering in JBPM using helix and zookeeper.But with this you will get a basic idea on what helix and zookeeper is used for.
And from most of the articles i read online it seems like zookeeper and helix are used together.
Apache Zookeeper can be installed on a single machine or on a cluster.
It can be used to keep track of logs. It can provide various services on a distributed platform.
Storm and Kafka rely on Zookeeper.
Storm uses Zookeeper to store all state so that it can recover from an outage in any of its (distributed) component services.
Kafka queue consumers can use Zookeeper to store information on what has been consumed from the queue.
We noticed that CPU usage went up from 5% TO 50% after adding NServicebus to our ASP.net MVC app. This was on a server that was not under any load. We noticed the same behavior on another server that hosted a WCF app. After trying out different things, we figured out that if we configured the bus as send only, the CPU usage dropped to 5%. Does anybody know why the cpu usage was so high when the bus is not configured as send only?
I've experienced this before.
What happened to me was I set up an application pool, and it started out running as Network Service. Before I had the chance to set the application pool identity to a domain-level user (for access to file shares, etc.) the pages had already been hit, and so the NServiceBus installers had already created a queue with the Network Service credentials.
When I set the application pool user, all of a sudden it didn't have the proper permissions to the queue.
Normally NServiceBus checks for messages with a timeout if none are available to be received, but in this instance, it goes into a very tight loop of "Are there messages? I don't have permission. Are there messages? I don't have permission." and so you get the very high CPU.
I fixed the problem by deleting the queue and allowing NServiceBus to recreate it with the proper permissions.
It's possible that the cause of the high CPU was the NServiceBus code that looks for a message in the queue, though I find that a bit hard to believe. Send-only mode prevents NServiceBus from looking for messages in the queue.