Connections do not return to the pool (JBoss EAP 6 AND SQL Server 2012) - sql-server-2012

I have an application running under jboss eap 6.3 and connecting to a SQL SERVER 2012 datasource through jTDS driver. The problem is the connections are opened and never return to the pool.
at first, there are no CONNECTION LEAK. I have already configured jboss to investigate this using the configuration below:
<cached-connection-manager debug="true" error="true"/>
Jboss found out one connection leak (in one statement) and i fixed it, but the problem remains.
the database configuration is as follows:
<datasource jta="false" jndi-name="java:jboss/datasources/DatabaseDS" pool-name="DatabaseDS" enabled="true" use-java-context="true" use-ccm="true">
<connection-url>jdbc:jtds:sqlserver://server/database;instance=instance;DatabaseName=database</connection-url>
<driver>jtds</driver>
<pool>
<min-pool-size>200</min-pool-size>
<max-pool-size>1200</max-pool-size>
<prefill>true</prefill>
</pool>
<security>
<user-name>username</user-name>
<password>password</password>
</security>
<timeout>
<blocking-timeout-millis>30000</blocking-timeout-millis>
<idle-timeout-minutes>1</idle-timeout-minutes>
</timeout>
after one day in production, the statistics of jboss-cli show the following data:
"In use" score, is always the same size as the "Count Active" . This number does not make sense to me, because for every request made, we open and close only one connection and at the time the print was made, had 98 sessions connected to jboss.
investigating the SQL Server processes for this application ( master.dbo.sysprocesses ) all "active count" connections are with "AWAITING COMMAND" status and detailed as "sleeping"
I am 100 % sure that all connections that are open are also closed in the application.
in about 2 days the pool is 100 % completed and need to restart the server
what may be causing this behavior ?
the value of "active count" should not be 200 ?
which may be blocking connections to return to the pool ?

I had similar issue and have detailed everything here as how to resolve. Please see this article
http://reddymails.blogspot.com/2013/10/troubleshooting-database-connection.html
In my case I was using Oracle with Hibernate. But the steps are similar to trouble shoot.

Related

weblogic em console's Admin server/managed servers status & metrics are unavailable

Weblogic em console shows "status pending" for the admin servers and managed servers, even though when they are up. It is not showing other metrics as well.
Below are the head error messages.
oracle.sysman.emas.sdk.model.metric.MetricsUnavailableException
at oracle.sysman.emas.sdk.model.metric.WLMetricProvider.getMetricServiceMBean(WLMetricProvider.java:313)
at oracle.sysman.emas.sdk.model.metric.WLMetricProvider.queryMetricTable(WLMetricProvider.java:364)
at oracle.sysman.emas.sdk.model.metric.WLMetricProvider.getMetrics(WLMetricProvider.java:155)
at oracle.sysman.emas.sdk.model.metric.MetricUtil.getMetrics(MetricUtil.java:271)
I would suggest opening a ticket with Oracle. You could try enabling Platform MBean Server which is in Domain home -> Advanced and restarting the domain. Usually this would fix it but without logs, its very hard to tell. Please try to provide us the sysman logs for the em.
I found the solution for this issue. This is a bug and oracle has released a patch for this BUG_13826887/Patch p13826887_111160_Generic.
1.Set ORACLE_HOME, java
2.check which opatch, opatch apply
3.restart complete weblogic

SSRS / Report Viewer - parameters greyed out / reports not rendering

Our production server did an automatic update last week and since then we have issues with our reporting. No code has changed whatsoever.
The setup is we have SSRS2008 reports that are viewed via a reportviewer control (version 10). Server has IIS7 and SQL2008R2. Since the updates we have 2 scenarios:
For reports without any parameters the loading icon appears for a few seconds, then goes away but nothing is rendered. You can see the report viewer control toolbar etc (all disabled) and white space where the report should be. No Errors.
For our reports with dependant parameters you can select the first parameter and thereafter the other parameters are all greyed out and you cant make a selection. Also no errors.
All reports run ok from within report manager.
Not sure if its related but after the update, the server restarted and after that the SQL Server service would not start - gave an error. Turned out we had to disable the VIA protocol in the SQL Server Configuration Manager. Once we did this the service started but also no idea why this occurred as we have never changed protocols etc.
From the reporting services log:
Unhandled exception: Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerHttpRuntimeClientDisconnectionException: A client has disconnected from the Report Server Web service application domain ReportManager_MSSQLSERVER_0-2-130461711338880624. No corrective action is required. An error code has been submitted to ASP.NET to release the connection. The error code is 800708CA. ---> System.Runtime.InteropServices.COMException: This network connection does not exist. (Exception from HRESULT: 0x800708CA)
Any help would be MUCH appreciated!
Have you tried running the report in ReportViewer.aspx ?
This could be a variation of the
'Operation is not valid due to the current state of the object' error
bug. It occurs when you have more than 1'000 parameters in a MULTIVALUE dropdown, and is brought to you by a SQL-Server security update.
You can solve it by adding aspnet:MaxHttpCollectionKeys to the web.config entries (your application & ReportServer):
<?xml version="1.0"?>
<configuration xmlns="http://schemas.microsoft.com/.NetConfiguration/v2.0">
<appSettings>
<add key="aspnet:MaxHttpCollectionKeys" value="50000"/>
Otherwise I'd look at the server version and figure out if service-pack 2 is missing.

SCVMM: A Hardware Management error has occurred trying to contact server %servername%:a:DestinationUnreachable

Today we found our host status is "Needs Attention".
We have upgraded the WMF 3.0.
And to check the health status and it reports the following error:
A Hardware Management error has occurred trying to contact server
iwwbgc8.dir.slb.com :a:DestinationUnreachable :The WS-Management
service cannot process the request. The service cannot find the
resource identified by the resource URI and selectors. .
Check that WinRM is installed and running on server
iwwbgc8.dir.slb.com. For more information use the command "winrm
helpmsg hresult".
ID: 2927 Details: Unknown error (0x8033803b)
Following the post: How to Interpret Job Failures in VMM and How to troubleshoot the “Needs Attention” and “Not Responding” host status in System Center 2012 Virtual Machine Manager
But the error is still there.
And there does some performance issue in events but by following the post How to manually rebuild Performance Counters for Windows Server 2008 64bit or Windows Server 2008 R2 systems, the performance counter can't not be fixed.
Error:
Installing the performance counter strings for service .NET Data Provider for Oracle (_) failed. The first DWORD in the Data section
contains the error code.
Cannot repair performance counters for .NET Data Provider for Oracle service. Reinstall the performance counters manually using the
LODCTR tool.
Event filter with query "SELECT * FROM __InstanceModificationEvent WITHIN 60 WHERE TargetInstance ISA "Win32_Processor" AND
TargetInstance.LoadPercentage > 99" could not be reactivated in
namespace "//./root/CIMV2" because of error 0x80041003. Events cannot
be delivered through this filter until the problem is corrected.
Unable to read Server Queue performance data from the Server service. The first four bytes (DWORD) of the Data section contains the
status code, the second four bytes contains the IOSB.Status and the
next four bytes contains the IOSB.Information.
Any idea bout it?
Later we found the issue is caused by the WMF 3.0 upgrade.
We follow the post Managing Hyper-V hosts using Virtual Machine Manager fails with Error: 0x8033803b after installing WMF 3.0 and apply the hotfix.
The hotfix (Windows6.1-KB2781512-x64) is applied but the issue still exists.
At last I select the solution to uninstall the WMF 3.0.
And the issue is fixed at last.

JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period

I am migrating an application from JBOSS 4 to JBOSSAS 7. I followed the guide for migration, converted all the ejbs to ejb3 fixed the JNDI names. After I deploy the ear I see message that the EJBs are bound. After that it goes in a wait state and then just times out with this message:
JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [1200 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.
I turned on the logging to DEBUG, which gives me a few exceptions about Transaction and NIO etc (which after spending a lot of time in JBOSS message groups and this one, I found were harmless).
I don't know how to debug this. I got the stack dump of the process but all I see is that the deployment scanner was waiting on something. Not sure what :
"DeploymentScanner-threads - 2" prio=5 tid=0000000003d77400 nid=0xb6447000 in Object.wait() [00000000b6446000]
java.lang.Thread.State: WAITING (on object monitor)
Anyone know how to debug this ? I am pretty damm fraustrated at this point. I mean if it doesn't throw an error, how am I supposed to debug this ?
It is an old post but, if some of you have a similar problem, you can try increasing the deployment-timeout in your standalone.xml,
Find This block
<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
<deployment-scanner path="deployments"
relative-to="jboss.server.base.dir"
scan-interval="5000" />
</subsystem>
Then add the attribute deployment-timeout (in seconds) to a value (like 20 minutes)
<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
<deployment-scanner path="deployments" relative-to="jboss.server.base.dir"
scan-interval="5000" deployment-timeout="1200"/>
</subsystem>
look this for reference
JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]
It is a usual problem in overloaded environments.

EhCache - Manual Replication not working

I am trying to set up Manual Replication for Ehcache for to two different servers. Below is my configuraion:
<cache name="codeTaskCache" maxElementsInMemory="1000" eternal="false"
timeToIdleSeconds="0" timeToLiveSeconds="0" overflowToDisk="false" />
<cacheManagerPeerProviderFactory class=
"net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=manual,
rmiUrls=//server1:40001/codeTaskCache | //server2:40001/codeTaskCache"
propertySeparator="," />`
The same configuration is present on both tomcat server which are running on two different unix boxes.
Replication works fine from server1 -> server2 but NOT from server2 -> server1 which is pretty strange.
In document they have a statement 'The rmiUrls is a list of the cache peers of the server being configured. Do not include the server being configured in the list.'
But if it works one way then why not the other way? Could someone please throw some light on this? Thanks in advance.
The RMI urls should not point to itself, just the other ehcache instances - at least that's what worked for me.
Also make sure the hostnames are resolvable from both servers, and in the last desperate attempt you can even try to have "hostname=name_of_server_dot_domain_dot_com" in the listening statement of the listener part XML.