We are using AWS Fargate ECS Tasks for our spring webflux java 11 microservice.We are using a FROM gcr.io/distroless/java:11 java image. When our application is dockerised locally and deployed as a image inside a docker container the memory utilization is quite efficient and we can see the heap usage never crosses 50%
However when we deploy the same image using the same dockerfile in AWS Fargate as a ECS task the AWS Dashbaord shows a completely different picture.The memory utilization never comes down and Cloudwatch logs show no OutOfMemory issues at all. In AWS ECS, once deployed we have done a Peak load test, a stress test after which the memory utilization reached 94% and then did a soak test for 6 hrs. The memory utilization was still 94% without any OOM errors.Memory the garbage collection is happening constantly and not letting the application go OOM.But it stays at 94%
For testing the application's memory utilization locally we are using Visual VM. We are also trying to connect to the remote ECS task in AWS Fargate using Amazon ECS Exec but that is work in progress
We have seen the same issue with other microservices in our and other clusters as well.Once it reaches a maximum number it never comes down.Kindly help if someone has faced the same issue earlier
Edit on 10/10/2022:
We connected to AWS Fargate ECS task using the Amazon ECS Exec and below were the findings
We analysed the GC logs of the AWS ECS Fargate Task and could see the messages.It uses the default GC i.e Simple GC. We keep getting "Pause Young Allocation Failure" which means that the memory assigned to the Young Generation is not enough and hence the GC fails.
[2022-10-09T13:33:45.401+0000][1120.447s][info][gc] GC(1417) Pause Full (Allocation Failure) 793M->196M(1093M) 410.170ms
[2022-10-09T13:33:45.403+0000][1120.449s][info][gc] GC(1416) Pause Young (Allocation Failure) 1052M->196M(1067M) 460.286ms
We made some code changes associated to byteArray getting copied in memory twice and the memory did come down but not by much
/app # ps -o pid,rss
PID RSS
1 1.4g
16 16m
30 27m
515 23m
524 688
1655 4
/app # ps -o pid,rss
PID RSS
1 1.4g
16 15m
30 27m
515 22m
524 688
1710 4
Even after a full gc like below the memory does not come down:
2022-10-09T13:39:13.460+0000][1448.505s][info][gc] GC(1961) Pause Full (Allocation Failure) 797M->243M(1097M) 502.836ms
One important observation was that after running inspect heap , a full gc got trigerred and even that didnt clear up the memory.It shows 679M->149M but the ps -o pid,rss command does not show the drop neither does the AWS Container Insights graph
2022-10-09T13:54:50.424+0000][2385.469s][info][gc] GC(1967) Pause Full (Heap Inspection Initiated GC) 679M->149M(1047M) 448.686ms
[2022-10-09T13:56:20.344+0000][2475.390s][info][gc] GC(1968) Pause Full (Heap Inspection Initiated GC) 181M->119M(999M) 448.699ms
How are you running it locally do you set any parameters (cpu/memory) for the container you launch? On Fargate there are multiple levels of resource configurations (size of the task and amount of resources you assign to the container - check out this blog for more details). Also the other thing to consider is that, with Fargate, you may land on an instance with >> capacity than the task size you configured. Fargate will create a cgroup that will box your container(s) to that size but some old programs (and java versions) are not cgroup-aware and they may assume the amount of memory you have is the memory available on the instance (that you don't see) and not the task size (and cgroup) that was configured.
I don't have an exact answer (and this did not fit into a comment) but this may be an area you can explore (being able to exec into the container should help - ECS exec is great for that).
Related
We have a Keycloak deployment running on Kubernetes. Our containers need to be periodically restarted because of high memory consumption. I want to analyze what is causing high memory consumption. How can I take JVM Heap dumps without modifying the Keycloak container image?
First, you can dump heap on demand with jmap command outside container.
You can also enable automatic heap dump on out of memory condition with -XX:+HeapDumpOnOutOfMemoryError JVM flag. Add -XX:HeapDumpPath to specify the path where to store heap dumps. JVM options can be added without modifying container image; just add the following environment variable:
JAVA_TOOL_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/storage/path"
Finally, since these JVM options are manageable, you can set them in runtime with jcmd:
jcmd <PID> VM.set_flag HeapDumpOnOutOfMemoryError true
jcmd <PID> VM.set_flag HeapDumpPath /storage/path
Cluster goes into deadlock state and stops allocating containers even when GBs of RAM and Vcores are available.
This was happening only when we start a lot of jobs in parallel most of which were Oozie jobs with many forked actions.
After a lot of search and reading related questions and articles, we came across a property called maxAMShare for YARN job scheduler (we are using Fair Scheduler).
What it means?
Percentage of memory and vcores from user's queue share that can be allotted to Application Masters. Default value: 0.5 (50%). Source
How it caused the deadlock?
When we will start multiple oozie jobs in parallel, each oozie job and the forked actions require couple of ApplicationMaster containers to be allotted first for oozie launchers which then start the other containers to do the actual action task.
In our case, we were actually starting around 20-30 oozie jobs in parallel, each with close to 20 forked actions. And with each action requiring 2 ApplicationMasters, close to 800 containers were getting blocked only by the Oozie ApplicationMasters.
Due to this, we were hitting the 50% default maxAMShare limit for our user queue. And YARN was not allowing to create new ApplicationMasters to run the actual job.
Solution?
One instant suggestion could be to disable the check by setting this property to -1.0. But this is not recommended. You can again end up allocating all or most of the resources to AMs and the real job that will get done will be very less.
Other option (which we went ahead with) is to specify a separate queue for AMs in the oozie configuration and then set maxAMShare property to 1.0. This way you can control how much resources can be allocated to AMs without affecting the other jobs. Reference
<global>
<configuration>
<property>
<name>oozie.launcher.mapred.job.queue.name</name>
<value>root.users.oozie_am_queue</value>
</property>
</configuration>
</global>
Hope this will be a major time saver for people facing the same issue. There could be many others reasons for deadlock too which are already discussed in other questions on SO.
Perhaps I am missing it, but I see no method to control the the hard memory limit for any given build (I have builds being murdered because of it). Is the build memory limit based on the build params supplied by the client (this means a single client can bring down everything) or is there someplace I can configure the service to only allow 512mb (for example) per build?
You can limit the max amount of memory per-container by setting the global DRONE_LIMIT_MEM variable (with the server). This should be set to the amount of memory in bytes, for example:
DRONE_LIMIT_MEM_SWAP=512000000
DRONE_LIMIT_MEM=512000000
These limits are passed to Docker when Drone starts a container [1]. It is the equivalent to the following Docker command:
docker run --memory=512000000 <image>
[1] https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory
I have the strange behaviour that yarn cannot allocate all containers properly.
I have maximizeResourceAllocation to true and it tries if I start an m4.4xlarge slave instance to request for each executor container all 32 cores from yarn.
However that fails for one container because the applicationmaster master process uses 1 core.
1 container 1vcore for the master process
32 cores container executor 1
32 cores container executor 2
32 cores container executor 3 fails because yarn has only 31 cores left to give.
my Step should execute in client mode. It is the same with the Zeppelin instances.
HadoopJarStepConfig sparkStepConf = new HadoopJarStepConfig()
.withJar("command-runner.jar")
.withArgs("spark-submit",
"--class", ".....",
"--deploy-mode", "client",
"/home/hadoop/jars/.....jar");
I use a bootstrap step to get the jars into /home/hadoop/jars, because if you want to use s3 paths only deploy cluster is allowed which would block one of my executors with the sparkContext process.
All this means if I only have one slave nothing happens at all. And if I have 3, 1 does not do any work. Which is a waste of money.
I could in theory calculate executor cores - 1 and force that setting in spark submit. But this is supposed to work.
How can I tell yarn to put this applicationMaster 1 core process on the master or not create it, or start executors with different core counts?
I only use 1-4 instances and effectively sitting idle is really ok.
I am using H2O to develp model. After initiated H2O instance I got an IP and port for opening H2O flow in web browser. I used below command in HDFS to initiate the H2O instance. The problem is when I run hyperparameter search, the job takes multiple hours and my shell session got inactive and will automatically log me out. This will kill the console session and H2O instance will be killed as well. I am using Rstudio interface with H2O. Is there any way to keep H2O instance longer without auto log out/shut down due to inactivity in
start h20 cluster
hadoop jar /dsap/devl/h2o/h2o-3.10.4.1-hdp2.4/h2odriver.jar -nodes 30 -mapperXmx 8g -output /user/userid1/h2o1 -baseport 6335
Yes.
Add the -disown flag to do exactly that.