memory leak in weblogic rmi driver - weblogic

We are having a memory leak issue after we started to use our existing Weblogic Datasource from our standalone apps. Standalone apps and WL Datasource does not lay in the same server. We are retrieving datasource with JNDI via t3 protocol.
We have used the approach down here.
https://docs.oracle.com/cd/E13222_01/wls/docs51/classdocs/API_jndi.html
87% of heap is allocated with weblogic/jdbc/rmi/internal/StructImpl_oracle_sql_STRUCT objects. This objects are in package RMI(Remote Method Invocation) and the problem itself started to appear after this remote usage. Since these two things happened at the same time we are suspicious that if we have done something wrong in the process of remote Datasource usage.
Everything works fine but we are restarting the nodes every 2-3 days and started to became a burden to us.
What can be the causes of this memory leak? Should we change anything in datasource properties or do we need to pass extra JVM parameters?
Here is a snapshot of our heap
Here is the weblogic properties.
Weblogic major version 10.3.6.0
Patch level: 10.3.6.0.181016 (October 2018 PSU - GENM)
Java version: Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode) - 1.6.0_45
MEM_ARGS="-Xms4096m -Xmx6144m -Xmn1024m -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:PermSize=256m -XX:MaxPermSi
ze=512m -verbosegc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:${LOGDIR}/GC_logs.${SERVER_NAME} -XX:+UseGCLogFileRotation -XX
:+PrintGCDateStamps -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=512M"
--- EXTRA JAVA OPTIONS
-DASA_MARK=ASA_VERSION_2016.02.17"
-Dweblogic.management.server=${ADMIN_URL}"
-DHostName_WLSName=${value_hostname}-${WLS_NAME}"
-Dweblogic.system.BootIdentityFile=${DOMAIN_HOME}/boot.properties"
-Dweblogic.rjvm.enableprotocolswitch=true"
-Djava.net.preferIPv4Stack=true"
Here is our standalone app properties
[wlsadmin#machine1 cfg]$ /usr/java/jdk1.6.0_26/bin/java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
CLASSPATH=$CLASSPATH:$HOME/lib/wlfullclient.jar
[wlsadmin#machine1 META-INF]$ more MANIFEST.MF
Manifest-Version: 1.0
Implementation-Vendor: Oracle Corp.
Implementation-Title: JDK 1.6 Client jar for WebLogic Server 10.3 Tue
Nov 15 08:52:36 PST 2011 1441050
Implementation-Version: 10.3.6.0
CLASSPATH=$CLASSPATH:$HOME/lib/ojdbc5.jar
[wlsadmin#machine1 META-INF]$ more MANIFEST.MF
Manifest-Version: 1.0
Specification-Title: "Oracle Globalization Development Kit"
Class-Path: orai18n-mapping.jar orai18n-utility.jar orai18n-collation.
jar orai18n-translation.jar orai18n-net.jar orai18n-servlet.jar orai1
8n-lcsd.jar orai18n-tools.jar gdk_custom.jar
Created-By: 1.5.0_07 (Sun Microsystems Inc.)
Main-Class: oracle.i18n.util.GDKOracleMetaData
Implementation-Title: "orai18n.jar archives"
Specification-Vendor: "Oracle Corporation"
Specification-Version: "Oracle Globalization Development Kit - 11.2.0.
1.0 - Production, Build : 11"
Implementation-Version: "Oracle Globalization Development Kit -
-Xms64m -Xmx256m

I am not able to guess what changes you can make in the relevant code, but based on the memory leak description - this is related to use of non thread safe hashmap.
It may help to try using ConcurrentHashMap.
Please check these links for more details:
https://support.oracle.com/knowledge/Middleware/1966075_1.html
https://javaeesupportpatterns.blogspot.com/2011/11/hashmap-thread-safe-problem-case-study.html

Related

How to run Ignite with defined JVM heap

Currently Ignite is taking the default installed OpenJDK 11 JVM heap size. Now, I want to define the JVM initial and max heap size for Ignite Instance. Where exactly I need to define so that it will take the defined JVM heap config to run.
You can use -Xms, -Xmx to set JVM memory parameters.
Here is more information: https://apacheignite.readme.io/docs/jvm-and-system-tuning
and: https://apacheignite.readme.io/docs/preparing-for-production
JDK 11 specific: https://docs.oracle.com/cd/E15523_01/web.1111/e13814/jvm_tuning.htm#PERFM160
In the Ignite logs you could see how much memory was allocated along with other parameters: look for these lines:
[IgniteKernal] Language runtime: Java Platform API Specification ver. 11
[IgniteKernal] VM information: Java(TM) SE Runtime Environment 11.0.4+10-LTS Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 11.0.4+10-LTS
[IgniteKernal] VM total memory: 1.0GB
[IgniteKernal] VM arguments: [-Xms1g, -Xmx1g, ...

Max heap size reduced from java 6 to java 7?

Currently we are encountering an problem, our java application can run smoothly with JRE6, but for JRE7, it will report "Could not create the Java virtual machine" sometimes and failed to start program. we have "-Xmx1024m" option in the launch command line.
after searching in google, people suggest check the max heap size that can be requested in PC. then I tried below ones.
for JRE6,
C:\Users\joey>java -Xmx1214m -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) Client VM (build 20.45-b01, mixed mode)
C:\Users\joey>java -Xmx1215m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
for JRE7,
D:\Program Files\Java\jre7\bin>java.exe -Xmx930m -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) Client VM (build 24.51-b03, mixed mode)
D:\Program Files\Java\jre7\bin>java.exe -Xmx931m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
And when i close most processes in PC and free memory to 2.5 GB, for JRE7, it will succeed with 1094m, like below.
D:\Program Files\Java\jre7\bin>java.exe -Xmx1094m -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) Client VM (build 24.51-b03, mixed mode)
D:\Program Files\Java\jre7\bin>java.exe -Xmx1095m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
It seems that the max heap size that JVM can request is related with system available memory size, does someone know the exact relationship?
Is there some way to enable tracing logs of JRE when creating JVM?
Thank you!
Regards,
Joey
I faced the same issue on various Windows OS flavous (Win7, Win2008R2) while having enough free RAM (3-5G), when switching from jre6_20 to jre7_51 (both 32 bit VMs)
Using vm args -Xmx1300m and -XX:MaxPermSize=256m on java 6 the VM is able to start. However, with java7 I had to lower down startup max memory (xmx with 100m or permgen with 100m) in order to be able to start VM.
The interesting part is this: if I restarted my box, the jvm 7 started with original vm args (-Xmx1300m and -XX:MaxPermSize=256m) so I suspect the root cause might be the OS with JVM combination.
I know that on Windows OSes - 32 bit VMs expect to have a single large chunk of memory, but cannot use the maximum process memory size will range from 1.4G to 1.6G (because of additional constraints)
Any troubleshooting pointers? Oracle does not provide any information in java7 release notes (or I could not spot those)
I have a similar problem Windows 7 64bit, no JAVA_HOME set nor in PATH
This fails
"C:\Program Files (x86)\Java\jdk1.7.0_51\bin\java.exe" -Xms32m -Xmx1024m -XX:MaxPermSize=256m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
This works
"C:\Program Files (x86)\Java\jdk1.7.0_51\bin\java.exe" -Xms32m -Xmx512m -XX:MaxPermSize=256m -version
This also works (I can recompile or not, doesn't make a difference
"C:\Program Files (x86)\Java\jdk1.6.0_45\bin\java.exe" -Xms32m -Xmx1024m -XX:MaxPermSize=256m -version
I tried turning off windows DEP, didnt make a difference
http://windows.microsoft.com/en-us/windows-vista/data-execution-prevention-frequently-asked-questions
I twiddled with the Virtual Memory settings , changed from allowing Windows to Manage
http://windows.microsoft.com/en-us/windows/change-virtual-memory-size#1TC=windows-7
I upped the Virual Memory in windows and it allowed it to work, sometimes...(with xmx1024)
Nothing else is consuming a lot of resources (memory, etc, I did have oracle database running locally, but shut it down)
"C:\Program Files (x86)\Java\jdk1.7.0_51\bin\java.exe" -Xms32m -Xmx1024m -XX:MaxPermSize=256m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
This works
"C:\Program Files (x86)\Java\jdk1.7.0_51\bin\java.exe" -Xms32m -Xmx900m -XX:MaxPermSize=256m -version
This fails
U:>"C:\Program Files (x86)\Java\jdk1.7.0_51\bin\java.exe" -Xms32m -Xmx2024m -XX
:MaxPermSize=256m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
New version , works even with letting Windows Managage the Virtual Memory and DEP turned on
"C:\Program Files (x86)\Java\jdk1.7.0_65\bin\java.exe" -Xms32m -Xmx1024m -XX:MaxPermSize=256m -version

Aptana closes unexpectedly

I just installed Aptana Studio 3.4.2 on a fresh install of Linux Mint 16 with Oracle java. The program closes unexpectedly. So far I've only used it with the Python perspective active. It seems to happen more often when I'm editing an html file. I used it for 3 hours one day before it crashed; other times only 10 minutes.
The log file has a recurring error that may be related but I can't find any info on it:
!MESSAGE Invalid preference page path: XML Syntax.
Does anyone know what is causing this?
Java Version:
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
Aptana Diagnostic:
Host OS: Linux
OS Version: 3.11.0-12-generic
OS Arch: x86_64
JRE Version: 1.7.0_45
JRE Vendor: Oracle Corporation
JRE Home: /opt/java/jdk1.7.0_45/jre
Aptana Studio 3 Version: 3.4.2.201308081805
Install Directory: file:/opt/Aptana_Studio_3/
Workspace Directory: file:/home/jeff/workspace/
VM Arguments: -Xms40m
-Xmx512m
-Declipse.p2.unsignedPolicy=allow
-Declipse.log.size.max=10000
-Declipse.log.backup.max=5
-Djava.awt.headless=true
-XX:MaxPermSize=256m
-jar
/opt/Aptana_Studio_3//plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar
Language: en_US
Node.JS Version: Not installed
NPM Path: Not installed
ENV:
MDMSESSION=default
GNOME_KEYRING_CONTROL=/run/user/1000/keyring-ka4Jsn
SHLVL=1
SSH_AGENT_PID=1662
TEXTDOMAINDIR=/usr/share/locale/
XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
GDM_XSERVER_LOCATION=local
SESSION_MANAGER=local/jeff-main:#/tmp/.ICE-unix/1585,unix/jeff-main:/tmp/.ICE-unix/1585
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
CLUTTER_DISABLE_XINPUT=1
GDMSESSION=default
XDG_SESSION_COOKIE=812a644561ed637b3aa6135652bf871c-1389232561.233987-1413464634
XDG_DATA_DIRS=/usr/share/default:/usr/share/gnome:/usr/local/share/:/usr/share/:/usr/share/mdm/
MANDATORY_PATH=/usr/share/gconf/default.mandatory.path
PWD=/home/jeff
GIO_LAUNCHED_DESKTOP_FILE=/home/jeff/Desktop/AptanaStudio3.desktop
MDM_XSERVER_LOCATION=local
LOGNAME=jeff
GPG_AGENT_INFO=/run/user/1000/keyring-ka4Jsn/gpg:0:1
GIO_LAUNCHED_DESKTOP_FILE_PID=3387
SSH_AUTH_SOCK=/run/user/1000/keyring-ka4Jsn/ssh
NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
TEXTDOMAIN=im-config
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-9GHbJcIE3A,guid=4d478335c530a3bf45ec18fb52ce01b1
SHELL=/bin/bash
GNOME_KEYRING_PID=1509
XDG_CURRENT_DESKTOP=GNOME
XDG_CONFIG_DIRS=/etc/xdg/xdg-default:/etc/xdg
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
DESKTOP_SESSION=default
APTANA_VERSION=3.4.2.1368863613
DISPLAY=:0.0
USER=jeff
HOME=/home/jeff
XAUTHORITY=/home/jeff/.Xauthority
XDG_SEAT=seat0
WINDOWPATH=8
XDG_SESSION_ID=c1
DEFAULTS_PATH=/usr/share/gconf/default.default.path
MDM_LANG=en_US.UTF-8
USERNAME=jeff
XDG_VTNR=8
XDG_RUNTIME_DIR=/run/user/1000
LANG=en_US.UTF-8
I Had the same problem with PHP. I found out that the program terminate when I hover my mouse over an object. So far so good after disabling the hover.
To disable: Window -> Preferences -> Content Assist -> Show information on hover: off
Try add
-Dorg.eclipse.swt.browser.DefaultType=mozilla
in AptanaStudio3.ini
Eclipse continue crashing
open AptanaStudio3.ini and append this line to end of file:
-Dorg.eclipse.swt.browser.DefaultType=mozilla
Try install Xterm and open again.

java.lang.UnsupportedClassVersionError: bad major version at offset=6

I have deployed a WAR file in the webapps folder in my tomcat. And when I am starting my tomcat, like below
C:\apache-tomcat-6.0.35\bin>startup.bat
Using CATALINA_BASE: "C:\apache-tomcat-6.0.35"
Using CATALINA_HOME: "C:\apache-tomcat-6.0.35"
Using CATALINA_TMPDIR: "C:\apache-tomcat-6.0.35\temp"
Using JRE_HOME: "C:\Program Files (x86)\IBM\RationalSDLC\Common\Java5.0\jre"
Using CLASSPATH: "C:\apache-tomcat-6.0.35\bin\bootstrap.jar"
I am getting this below exception-
SEVERE: Error deploying web application archive DirectoryServer.war
java.lang.UnsupportedClassVersionError: (com/services/rest/Listener) bad major version at offset=6 (unable to load class com.services.rest.Listener)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2822)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1159)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1647)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.catalina.startup.WebAnnotationSet.loadClassAnnotation(WebAnnotationSet.java:145)
at org.apache.catalina.startup.WebAnnotationSet.loadApplicationListenerAnnotations(WebAnnotationSet.java:73)
at org.apache.catalina.startup.WebAnnotationSet.loadApplicationAnnotations(WebAnnotationSet.java:56)
at org.apache.catalina.startup.ContextConfig.applicationAnnotationsConfig(ContextConfig.java:297)
at org.apache.catalina.startup.ContextConfig.start(ContextConfig.java:1078)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:261)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4612)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
And this is my JAVA version-
C:\apache-tomcat-6.0.35\bin>java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
Can anyone provide any suggestion why is it happening?
You have essentially compiled your code with JDK 1.6 but running it with Java 5 - Just ensure a Java 6 runtime.
Even I faced same problem. The problem is inappropriate class version, for ex. compiling with different jre.
Solution: IF you are using Eclipse or RAD, set change compiler setting as per your requirement and also change same setting to project configuration by clicking on "Configure Project specific setting". Windows->Preferences->Compiler-> click on link "Configure Project specific setting".
For me the fix was to set the JAVA_Home system variable to the correct path.
It is a compiler issue. Make sure all your modules are configured with java 1.5 as a compiler. Window-->preference--> Java --> Compiler. Make sure 1.5 is selected.

Failed to start Glassfish after increasing heap size

I want to increase the heap size of my Glassfish. For that, I know that I can go up to 4GB:
java -Xmx4000M -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-384, mixed mode)
I tried to set in domain.xml file the -Xmx to 2GB:
<jvm-options>-Xmx2000m</jvm-options>
But I am getting the following error:
asadmin> start-domain
Waiting for ... to start .Error starting domain ...
The server exited prematurely with exit code 1.
Before it died, it produced the following output:
Error occurred during initialization of VM
The size of the object heap + VM data exceeds the maximum representable size
launchctl bsexec failed: Inappropriate ioctl for device
Launching the command with -v option gives this:
12 oct. 2011 11:46:34 com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: JVM invocation command line:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
-XX:+UnlockDiagnosticVMOptions
-XX:MaxPermSize=512m
-XX:NewRatio=2
-XX:+CMSClassUnloadingEnabled
-Xmx2000m
-Xms1000m
...
12 oct. 2011 11:46:35 com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: Successfully launched in 45 msec.
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
Command start-domain failed.
I found the problem. For some reasons which I do not understand, the "-d32" argument was passed in the command, asking to start in 32 bits mode. When adding
<jvm-options>-d64</jvm-options>
to the domain.xml file, glassfish starts. Please note that this option is not present by default in the file.
you must be using a different Java install when doing the "-version" and from within GlassFish. I can reproduce the same error with the default "java" in Mac OS' path but not when using an absolute path :
% /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Xmx2300m -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03-383-11A511)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-383, mixed mode)
% java -Xmx2300m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
Could it be that you don't have enough disk space on your system? Reserving 4GB of RAM may require MAC OS to expand swap space, and you may not have the disk space to do this. Unfortunately, this has burned me before :-/