SocketException: Too many open files - socketexception

I am running my application on jboss7.0.2. After some time VM started throwing error "FileNotFoundException : Too many open files". Immediately after this, the exception changed into "SocketException: Too many open files" and all the socket connection started getting rejected.
when i list out the files using ,
lsof -p (PID of java)
java 2981 jboss cwd DIR 202,1 4096 286892 /opt/jboss7/bin
java 2981 jboss mem REG 202,1 3101294 303234 /opt/jboss7/modules/com/sun/xml/bind/main/jaxb-xjc-2.2.jar
java 2981 jboss mem REG 202,1 872839 303232 /opt/jboss7/modules/com/sun/xml/bind/main/jaxb-impl-2.2.jar
java 2981 jboss mem REG 202,1 543044 303191 /opt/jboss7/modules/org/joda/time/main/joda-time-1.6.2.jar
java 2981 jboss mem REG 202,1 7662 337202 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/slf4j-simple-1.6.4.jar-5e712b1d3aa271f5/slf4j-simple-1.6.4.jar
java 2981 jboss mem REG 202,1 20639 337199 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/log4j-over-slf4j-1.6.4.jar-da4eaca58ac78436/log4j-over-slf4j-1.6.4.jar
java 2981 jboss mem REG 202,1 31284 337194 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/jcommander-1.12.jar-c61b19af70c8bf25/jcommander-1.12.jar
java 2981 jboss mem REG 202,1 666665 337191 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/guice-2.0.jar-29aff9bdf30d9ec6/guice-2.0.jar
java 2981 jboss mem REG 202,1 121070 337188 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/junit-3.8.1.jar-e882fc70a94f89bb/junit-3.8.1.jar
java 2981 jboss mem REG 202,1 659413 337185 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/trove-1.1-beta-5.jar-71bab73f2a1e98c9/trove-1.1-beta-5.jar
java 2981 jboss mem REG 202,1 608376 337182 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/c3p0-0.9.1.jar-7d5529e5d213dc5/c3p0-0.9.1.jar
java 2981 jboss mem REG 202,1 539705 337179 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/postgresql-9.0-801.jdbc4.jar-984beb99091b7785/postgresql-9.0-801.jdbc4.jar
java 2981 jboss mem REG 202,1 26361 337176 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/asm-1.5.3.jar-54fb77fde523b0a9/asm-1.5.3.jar
java 2981 jboss mem REG 202,1 2309955 337173 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/hibernate-core-3.3.2.GA.jar-b8a8129c26f4cd51/hibernate-core-3.3.2.GA.jar
java 2981 jboss mem REG 202,1 15347 337170 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/opencsv-2.0.jar-888b8a58480bd822/opencsv-2.0.jar
java 2981 jboss mem REG 202,1 109318 337167 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/xml-apis-1.0.b2.jar-630532004dbb4129/xml-apis-1.0.b2.jar
java 2981 jboss mem REG 202,1 57779 337164 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/commons-fileupload-1.2.1.jar-5294101c9a62497a/commons-fileupload-1.2.1.jar
java 2981 jboss mem REG 202,1 282244 337161 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/cglib-2.1_2.jar-4ec9fd3801bb1495/cglib-2.1_2.jar
java 2981 jboss mem REG 202,1 644148 337158 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/javassist-3.12.1.GA.jar-5495cc51e7d87d08/javassist-3.12.1.GA.jar
java 2981 jboss mem REG 202,1 232121 337155 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/snakeyaml-1.6.jar-71fb94779beb43bc/snakeyaml-1.6.jar
java 2981 jboss mem REG 202,1 25962 337152 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/slf4j-api-1.6.4.jar-87ef6bbe594711ec/slf4j-api-1.6.4.jar
java 2981 jboss mem REG 202,1 4467 337149 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/aopalliance-1.0.jar-9912ab03a6271bc3/aopalliance-1.0.jar
java 2981 jboss mem REG 202,1 568435 337146 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/testng-5.14.9.jar-a15bf7544fe18e09/testng-5.14.9.jar
java 2981 jboss mem REG 202,1 443432 337143 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/antlr-2.7.6.jar-61bc20b99e33636b/antlr-2.7.6.jar
java 2981 jboss mem REG 202,1 87776 337140 /opt/jboss7/standalone/tmp/vfs/deployment89b34d33a7ecad72/commons-io-1.3.2.jar-512e37e2249c5643/commons-io-1.3.2.jar
java 2981 jboss 783u IPv6 54509212 0t0 TCP Cloud.ecommunicate.celebritytalktone.mobi:webcache->Cloud.ecommunicate.celebritytalktone.mobi:36810 (CLOSE_WAIT)
java 2981 jboss 784u IPv6 54509228 0t0 TCP Cloud.ecommunicate.celebritytalktone.mobi:webcache->Cloud.ecommunicate.celebritytalktone.mobi:36812 (CLOSE_WAIT)
java 2981 jboss 785u IPv6 54509501 0t0 TCP Cloud.ecommunicate.celebritytalktone.mobi:webcache->Cloud.ecommunicate.celebritytalktone.mobi:36818 (CLOSE_WAIT)
java 2981 jboss 786u IPv6 54509555 0t0 TCP Cloud.ecommunicate.celebritytalktone.mobi:webcache->Cloud.ecommunicate.celebritytalktone.mobi:36821 (CLOSE_WAIT)
java 2981 jboss 787u IPv6 54509689 0t0 TCP Cloud.ecommunicate.celebritytalktone.mobi:webcache->Cloud.ecommunicate.celebritytalktone.mobi:36825 (CLOSE_WAIT)

You can use linux command ulimit to set the maximum number of opened file handles, to a larger value. For example:
ulimit -n 60000

The ulimit answer is correct.
However, you may first need to edit your /etc/security/limits.conf file to allow the user to increase these limits:
# to give all users a hard limit of 60000 open files
* hard nofile 60000
# to give user "jbossuser" a default limit of 60000 open files
jbossuser - nofile 60000

It Exceeded the number of open file descripted limits.
Solution : Increase the Hard and soft limits on number of open files.
soft nofile 999999
hard nofile 999999
In /etc/security/limits.conf file

Related

How to unload an NVIDIA kernel module 'nvidia' for new driver installation?

I needed to upgrade my nvidia driver so that I have tried running NVIDIA-LInux-x86_64.run file
However, I was seeing following message
ERROR: An NVIDIA kernel module 'nvidia' appears to already be loaded in your kernel. This may be because it is in use (for example, by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading. Please be sure to exit any programs that may be using the GPU(s) before attempting to upgrade your driver. If no GPU-based programs are running, you know that your kernel supports module unloading, and you still receive this message, then an error may have occured that has corrupted an NVIDIA kernel module's usage count, for which the simplest remedy is to reboot your computer.
I have already unloaded nvidia-drm and when I tried to unload nvidia
$ sudo modprobe -r nvidia
modprobe: FATAL: Module nvidia is in use.
Can anyone guide me on installing this new driver without any issue?
Thanks
Use lsof /dev/nvidia* to find the processes that are using the old driver. In my case it was "nvidia-persistenced". Just kill the process by pid and retry the installer NVIDIA-***.run
# lsof /dev/nvidia*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nvidia-pe 1334 nvidia-persistenced 2u CHR 195,255 0t0 420 /dev/nvidiactl
nvidia-pe 1334 nvidia-persistenced 3u CHR 195,0 0t0 421 /dev/nvidia0
nvidia-pe 1334 nvidia-persistenced 5u CHR 195,0 0t0 421 /dev/nvidia0
nvidia-pe 1334 nvidia-persistenced 6u CHR 195,0 0t0 421 /dev/nvidia0
nvidia-pe 1334 nvidia-persistenced 7u CHR 195,0 0t0 421 /dev/nvidia0
I just removed existing driver and reinstalled

How to set KVM VM use shadow page table?

I want to measure shadow page table performance vs EPT. I know in kvm code path, EPT and shadow page table share some code path. There is a switch to check if EPT is enabled.
So, I turn off EPT, I think this is a way to use shadow page table in KVM VM.
I exec some commands on host:
# cat /sys/module/kvm_intel/parameters/ept
Y //check if EPT is enabled, Yes it is
# rmmod kvm_intel
# modprobe kvm_intel ept=0,vpid=0 //Re-modprobe kvm_intel, but turn off EPT and VPID
# cat /sys/module/kvm_intel/parameters/ept
N // EPT is disabled
# cat /sys/module/kvm_intel/parameters/vpid
Y //VPID is still enabled
After these commads, I expect the EPT is disabled, However, I create a VM with 4 vcpu, but in VM, I use htop in VM only find i vcpu. I don't know why.
So, I turn on ept again, I can see 4 vcpu in vm using htop. But, on another server, I turn off ept, in VM, I still can find 4 vcpu.
In host, I test qemu thread num:
pstree -p | grep qemu
|-qemu-kvm(20594)-+-{qemu-kvm}(20612)
| |-{qemu-kvm}(20613)
| |-{qemu-kvm}(20614)
| |-{qemu-kvm}(20615)
| |-{qemu-kvm}(20618)
| `-{qemu-kvm}(23429)
There are still muti threads.
In KVM VM, I use lscpu to check:
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0
Off-line CPU(s) list: 1-3
Thread(s) per core: 1
Core(s) per socket: 1
座: 1
NUMA 节点: 1
厂商 ID: GenuineIntel
CPU 系列: 6
型号: 62
型号名称: Intel(R) Xeon(R) CPU E5-2640 v2 # 2.00GHz
步进: 4
CPU MHz: 1999.999
BogoMIPS: 3999.99
超管理器厂商: KVM
虚拟化类型: 完全
L1d 缓存: 32K
L1i 缓存: 32K
L2 缓存: 4096K
NUMA 节点0 CPU: 0
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase tsc_adjust smep erms xsaveopt
VPID is still enabled cause your command format was incorrect
# modprobe kvm_intel ept=0 vpid=0(Use spaces instead of commas)

Apache MINA - stuck on SSL connection

I am having troubles with Apache MINA core library.
When I deploy my application to a remote server some of the requests are not processed (around 2%). It looks like there might be a problem with SSL.
Log tail: http://pastebin.com/48bwWsjs
When request is not being processed, it is always stuck on the:
org.apache.mina.filter.ssl.SslFilter - Session Server[40](ssl...): Processing the SSL Data
Did something similar happened to any of you?
I tried Apache-mina 2.0.7 and 2.0.16
Env:
bash-4.2$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
bash-4.2$ uname -a
Linux 8d9ad913fa03 4.4.39-34.54.amzn1.x86_64 #1 SMP Fri Dec 30 19:11:28 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Thanks any help!

JBOSS 7.1 server is not starting in linux mint 17.3

I am using Linux Mint 17.3 . i want to configure JBOSS server in my system. whenever i command sh.standalone.sh it is showing following screen but doesnot starting server.
indra#Jarvis ~/jboss-as-7.1.1.Final/bin $ sh standalone.sh
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /home/indra/jboss-as-7.1.1.Final
JAVA: /usr/lib/jvm/java-8-oracle/bin/java
JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml
=========================================================================
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
09:45:02,152 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA
09:45:02,517 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA
09:45:02,629 INFO [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
^C09:47:05,537 INFO [org.jboss.as] JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 1ms
i wait for 15 min but it is not starting .
please someone guide me.
JBoss AS 7.1.1.Final does not work with Java 8, https://developer.jboss.org/message/926279#926279. You consider migrating to WildFly 10 as it requires Java 8.

Jahia/JVM/Tomcat Crash

we are using following configuration:
CATALINA_OPTS="$CATALINA_OPTS -Xms6144m -Xmx6144m -Dcertain.config=/data02/certain/config -Djava.awt.headless=true -XX:MaxPermSize=1024m -server -Dhibernate.jdbc.use_streams_for_binary=true -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:ParallelGCThreads=4 -XX:LargePageSizeInBytes=512m -XX:+UseParallelOldGC -XX:+UseParallelGC -XX:+UseCompressedOops -XX:+AggressiveHeap -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
JDK 1.6 version 22
OS:Red Hat Enterprise Linux Server release 6.

CPU:total 4 (1 cores per cpu, 1 threads per core)

Java HotSpot(TM) 64-Bit Server VM (17.1-b03) for linux-amd64 JRE (1.6.0_22-b04)
we keep getting:

# A fatal error has been detected by the Java Runtime Environment:

# java.lang.OutOfMemoryError: requested 17179869200 bytes for Chunk::new. Out of swap space?

#
#  Internal Error (allocation.cpp:215), pid=5845, tid=139708177794816
  Error: Chunk::new

# JRE version: 6.0_22-b04
# Java VM: Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode linux-amd64 compressed oops)