I recently was modifying some of my server properties in Rational Application Developer to try and increase the memory of my JVM on startup. I forgot to take a backup before doing this, and by adding in an incorrect JVM variable, it seems I have broke my server in an unworking state. Whenever I try and startup my server to do any configuration changes, the JVM refuses to start with invalid params being passed in.
Is there a way to reset any JVM changes for WebSpehere Application Server v7.0 through the filesystem, or a way to do it without needing the server running already? I have been looking around in the wasProfile hoping to stumble onto a file where my settings ultimately live, but have had no luck.
It should be possible to write a wsadmin script to view/adjust the JVM options, but if you're on a non-z/OS platform, the fastest way to get back to working is probably to edit PROFILE_HOME/config/cells/CELL/nodes/NODE/servers/SERVER/server.xml; the JVM settings are typically written at the very end.
Related
I was having intermittent issue running a Mule Batch with huge data in Anypoint Studio. That issue is resolved by enabling 'Always' option under 'Clear Application Data' in 'Run Configurations' (as per the given instruction in Mule ESB - Clear Memory of a batch process). That option is shown in the picture.
How to enable the same 'Always' option in stand alone Mule Runtime during the startup that means when we are not running the batch from Anypoint Studio? Is there any command line argument available that can be used in startup script of the Mule Runtime to achieve the same goal?
By deleting the local data you are deleting batch queues, persistent objects stores and maybe some other information. In a development environment like Anypoint Studio IDE it is usually OK but for a standalone Mule Runtime it means you are deleting production data, for example records that are used by batch to continue processing after a restart. That data will be lost. Having said that, it might be needed if the data is completely corrupted.
It is a best practice and a strong advice to any user to resolve the root cause of the issue rather than delete data. And it should never be done every time you start your production Mule, only when there is absolutely no other alternative.
I don't recommend to delete local files at all. If even after my warnings you absolutely need to do this never ever delete the .mule directory. If you still want to risk losing data delete only the directory with the application name under the .mule directory.
I'm trying to generate a full heapdump from Weblogic 10.3.6 due to an OutOfMemoryError generated by a Web Application deployed on the Server.
I've setted the following start script:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump
When the OutOfMemoryError occurs, Weblogic generates an empty hprof file (0 bytes size) in /path/to/heapdump folder, and nothing happens: the Server remains in RUNNING mode, even if is not reachable anymore.
The java process is still alive, but with 0% of processor.
Even the server.out log seems completely frozen, without any trace of the OutOfMemoryError.
What's wrong with the configuration?
Probably you can use Java Flight Recorder to save events and check which objects are generating OOM.
(any profiler should work as well).
Been there :( . I remember at the time that we've found it was somewhat logical since there was not enough memory for normal operation, the JVM could not automagically find enough memory to create a heapdump either. If memory serves me well, at that time we did 2 things to debug the memory leak. First we were "lucky" enough that the problem was happening fairly regularly so a close manual monitoring was possible (monitoring of the gc.log looking for repeated FullGC and monitoring of the performance tab in the console). Knowing when the onset of the problem was starting we were doing some kill -3 to get the dump manually. We also used jstack {PID} (JDK 1.6 on Linux) with some luck. With those, at the time, the devs were able to identify the memory leak. Hope that helps.
Okay, your configuration looks alright.. you might want to check if the weblogic process user has the rights to edit the heap dump file.
You can take heap dump by Java tools :
JAVA_HOME/bin/jmap -dump:format=b,file=path_of_the_file
OR
%JROCKIT_HOME%\bin\jrcmd hprofdump filename=path_of_the_file
We have a Windows Server 2012 64bit + Weblogic 12c setup. The AdminServer requires a higher PermSize when being used with a 64bit OS, thus we need to modify the "setDomainEnv.cmd" (as described in other questions here on stackoverflow).
When starting the AdminServer through the usual "startWeblogic.cmd" script, it uses the settings in "setDomainEnv.cmd" that sets the PermSize etc. successfully, but when using NodeManager "startServer()" command, it does not.
I read something in the documentation about the fact that one can control the parameters that are loaded on startup of a managed server (with NodeManager), but I did not find the right way to do it.
I would hope that we can achieve a consistent behaviour when starting a managed server (and the AdminServer) through NodeManager or manually.
Any ideas?
UPDATE:
I checked what's going on when starting managed server and(!) in comparison what's going on when starting the AdminServer. Result: the AdminServer process (it starts a 'javaw.exe' instance in contrast to a 'java.exe' instance for a managed server) never get's passed ANY parameters set in the setDomainEnv.cmd script.. it's basically full of Oracle internal parameters.
To me all this looks completely messed up and inconsistent. In addition to this I found an issue reported by Oracle that mystically talks about setting environment variables when running on a 64bit OS (see headline "Developer ZIP Distribution Fails on Windows 64-bit and Linux 64-bit"):
https://docs.oracle.com/cd/E24329_01/doc.1211/e26593/issues.htm#WLSRN238
I have idea if this applies to my version or not, since the version I downloaded does not say "developer" version, it basically was the primary weblogic download for the latest release.
The question that comes to my mind is this: what is the expected way of starting the AdminServer if not using "startServer"? Is there a bug that nobody cares about, since it is usually done differently? I am really disappointed to how confusing this rather simple topic evolves when starting to read Oracle documentation: it simply does not say anything about it at all.
Command line that is triggered when starting the AdminServer through "startServer()" command:
C:\PROGRA~1\Java\JDK17~1.0_6\jre\bin\javaw.exe -classpath "C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\rt.jar;C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\i18n.jar;C:\PROGRA~1\Java\JDK17~1.0_6\lib\tools.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic_sp.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic.jar;D:\Oracle\Middleware\oracle_common\modules\net.sf.antcontrib_1.1.0.0_1-0b3\lib\ant-contrib.jar;D:\Oracle\Middleware\wlserver\modules\features\oracle.wls.common.nodemanager_2.0.0.0.jar;D:\Oracle\Middleware\oracle_common\modules\com.oracle.cie.config-wls-online_8.1.0.0.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derbyclient.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derby.jar;D:\Oracle\Middleware\wlserver\server\lib\xqrl.jar" "-Djava.runtime.name=Java(TM) SE Runtime Environment" -Dpython.cachedir=C:\Users\ADMINI~1\AppData\Local\Temp\2\wlstTempAdministrator -Djava.protocol.handler.pkgs=weblogic.utils|weblogic.utils|weblogic.utils -Djava.vm.version=24.65-b04 "-Djava.vm.vendor=Oracle Corporation" -Djava.vendor.url=http://java.oracle.com/ -Dpath.separator=; "-Djava.vm.name=Java HotSpot(TM) 64-Bit Server VM" -Dweblogic.RootDirectory=D:\Oracle\Middleware\user_projects\domains\test1234\. "-Djava.vm.specification.name=Java Virtual Machine Specification" -Djava.runtime.version=1.7.0_67-b01 -Djavax.rmi.CORBA.UtilClass=weblogic.iiop.UtilDelegateImpl -Djava.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment -Djava.endorsed.dirs=C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\endorsed -Dos.arch=amd64 -Djava.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\2\ -Dline.separator=
"-Djava.vm.specification.vendor=Oracle Corporation" -Djava.naming.factory.url.pkgs=weblogic.jndi.factories:weblogic.corba.j2ee.naming.url "-Dos.name=Windows Server 2012 R2" -Dprod.props.file=D:\Oracle\Middleware\wlserver\.product.properties -Dorg.omg.CORBA.ORBSingletonClass=weblogic.corba.orb.ORB -Djava.library.path=C:\PROGRA~1\Java\JDK17~1.0_6\jre\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;;D:\Oracle\Middleware\wlserver\server\native\win\x64;D:\Oracle\Middleware\wlserver\server\bin;D:\Oracle\Middleware\oracle_common\modules\org.apache.ant_1.9.2\bin;C:\PROGRA~1\Java\JDK17~1.0_6\jre\bin;C:\PROGRA~1\Java\JDK17~1.0_6\bin;D:\Oracle\product\12.1.0\dbhome_1\BIN;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WINDOW~1\v1.0\;C:\PROGRA~2\VISUAL~1\bin;C:\PROGRA~1\doxygen\bin;C:\PROGRA~1\TORTOI~1\bin;C:\PROGRA~2\WINDOW~4\8.0\WINDOW~1\;C:\PROGRA~1\MICROS~1\110\Tools\Binn\;D:\Oracle\Middleware\wlserver\server\native\win\x64\oci920_8;. "-Djava.specification.name=Java Platform API Specification" -Djava.class.version=51.0 -Dorg.omg.CORBA.ORBClass=weblogic.corba.orb.ORB -Dos.version=6.3 -Djavax.rmi.CORBA.PortableRemoteObjectClass=weblogic.iiop.PortableRemoteObjectDelegateImpl -Djava.awt.printerjob=sun.awt.windows.WPrinterJob -Djava.specification.version=1.7 -Djava.class.path=C:\PROGRA~1\Java\JDK17~1.0_6\lib\tools.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic_sp.jar;D:\Oracle\Middleware\wlserver\server\lib\weblogic.jar;D:\Oracle\Middleware\oracle_common\modules\net.sf.antcontrib_1.1.0.0_1-0b3\lib\ant-contrib.jar;D:\Oracle\Middleware\wlserver\modules\features\oracle.wls.common.nodemanager_2.0.0.0.jar;D:\Oracle\Middleware\oracle_common\modules\com.oracle.cie.config-wls-online_8.1.0.0.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derbyclient.jar;D:\Oracle\Middleware\wlserver\common\derby\lib\derby.jar;D:\Oracle\Middleware\wlserver\server\lib\xqrl.jar -Djava.vm.specification.version=1.7 -Dweblogic.management.GenerateDefaultConfig=false -Djava.home=C:\PROGRA~1\Java\JDK17~1.0_6\jre "-Djava.specification.vendor=Oracle Corporation" -Dawt.toolkit=sun.awt.windows.WToolkit "-Djava.vm.info=mixed mode" -Djava.version=1.7.0_67 -Djava.ext.dirs=C:\PROGRA~1\Java\JDK17~1.0_6\jre\lib\ext;C:\Windows\Sun\Java\lib\ext "-Djava.vendor=Oracle Corporation" -Djava.vendor.url.bug=http://bugreport.sun.com/bugreport/ -Dweblogic.store.DisableDiskScheduler=true -Dpython.verbose=warning weblogic.Server
UPDATE 2:
Start the AdminServer through node manager (nmStart('AdminServer')) creates a usual "java.exe" process and starts up the AdminServer with correct memory settings. But this is even more confusing: why is "startServer()" creating a separate process (javaw.exe) with entirely different settings? Why are my settings now totally different for AdminServer? What is the "correct" way of starting the AdminServer (development/production?). Two thumbs down on this environment.
UPDATE 3:
After repeating further tests the solution of getting "startServer()" to work is basically as follows: do not worry about the node manager settings at all, edit the "startWeblogic" script directly by adding additional java options inside of it (as usual by adding -D start parameters). The reason for all this is basically that the global settings (as used by node manager) are ignored completely, see my pasted command line output.
Check the nodemanager.properties file in your Oracle install ( e.g. /opt/ora/mw/wlserver_10.3/common/nodemanager/nodemanager.properties ) and verify that these options are set:
StartScriptName=startManagedWebLogic.sh
StartScriptEnabled=true
so the nodemanager is starting your servers with the appropriate scripts. You also have to option of setting server specific start attributes via the admin console - go to:
Servers -> Server Name -> Server Start tab -> Arguments
You can fill in server specific JVM args, like -XX:MaxPermSize=4096m in this field that will be used by the nodemanager. This may be a better/easier idea than hard coding it in the setDomainEnv script.
UPDATE
Attempt issuing an nmStart() command rather than a startServer() command for the AdminServer.
startServer allows you to start a server WITHOUT the nodemanager. It uses javaw.exe to effectively background the process
nmStart allows you to start the server WITH the nodemanager - which is why you get the correct memory settings. Because the process is started via a service, it is more or less automatically backgrounded, which is why you see the normal java.exe
I have deployed a system integrated with weblogic, but until now I faced a problem is the weblogic increasing the stdout.out size heavily(by GB per week), it caused the system to load slowly and slowly.
Any way to prevent it increase the size heavily or redirect into .log?
Thanks alot
As David Herget says above, using the WebLogic Scripting Tool (WLST) to redirect StdOut and StdErr did not actually work for me either; I had to also do so through the web console (even though they appear to be set on the console) and restart the relevant jvms.
I can't reply to David's comment above due to being a newbie. [Edited since for clarity]
Not totally sure to understand fully your question.
Are you talking about the {server_name}.out file located in the {Domain_Path}/servers/{server_name}/logs ?
If so, I've never found anyway to rotate those logs automatically so I run a script each day to rotate it (basically copying it to another name, zip it and echoing a NULL in the orginal file...erasing the older one after).
If you are talking about redirecting StdOut to the logs though, that can be done within the console for each server in the logging tab by checking "Redirect stdout logging enabled". Configuration to rotate those logs can also be done within that tab.
On that, StdErr can also be redirected, but not from the console (in WL9). You have to put "RedirectStderrToServerLogEnabled" at true in the MBean tree by wlst (it's located at /Servers/{server_name}/Log/{server_name}
I know the question was ask long time ago but hoping it would help nonetheless
Weblogic provides features of log files rotation based on the size and time interval.
You can try rotating the log files based on the size. You would need to configure the log rotation policy from the admin console. Please refer the below link for further details.
http://docs.oracle.com/cd/E12840_01/wls/docs103/ConsoleHelp/taskhelp/logging/RotateLogFiles.html
If you want to rotate the log files on demand, you can use the below WSLT script.
C:\>java weblogic.WLST
#connect WLST to an Administration Server
wls:/offline> connect('username','password')
#navigate to the ServerRuntime MBean hierarchy
wls:/mydomain/serverConfig> serverRuntime()
wls:/mydomain/serverRuntime>ls()
#navigate to the server LogRuntimeMBean
wls:/mydomain/serverRuntime> cd('LogRuntime/myserver')
wls:/mydomain/serverRuntime/LogRuntime/myserver> ls()
-r-- Name myserver
-r-- Type LogRuntime
-r-x forceLogRotation java.lang.Void :
#force the immediate rotation of the server log file
wls:/mydomain/serverRuntime/LogRuntime/myserver> cmo.forceLogRotation()
wls:/mydomain/serverRuntime/LogRuntime/myserver>
http://docs.oracle.com/cd/E12840_01/wls/docs103/logging/config_logs.html#wp1001654
We are running a Domino 8.5.3 and the server log is constantly issuing these errors:
HTTP JVM:
!err.PersistenceServiceResourceProvider.Errorwritingtopersistedcontenttor!
We have not been able to isloate it to a particular page. Eventually, the HTTP task will crash and we need to reboot the server and recompile all the databases on the server. We are using the CKEditor to generate the HTML content. You help would be most appreciated.
We used to get this exact error a lot which appeared to be caused by inline images uploaded via the CKEditor like Paul mentioned.
I don't know why but we fixed it by changing the directory domino uses for uploads via the
xsp property xsp.persistence.dir.xspupload (formally xsp.upload.directory)
changing it to something like c:\temp rather than the windows default made the problem go away. could have been a co-incidence but may have been something in windows interfering
I haven't seen that error before but it reads like it might be something to do with asking the server to make a lot of data available to the xpages between calls to the server- effectively, session and application scope data.