Liferay stopped at database shutdown caused a crash - crash

I was stopping the Liferay portal, but few seconds after, I stopped the database (db2 quiesce, that means, that the connections are closed) and apparently, Liferay did not stopped correctly its execution.
After that, I restarted the database and liferay, but the portal does not work now. It shows this message in the browser:
HTTP Status 500 -
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: Servlet execution threw an exception
com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:72)
...
root cause
java.lang.NoSuchMethodError: com.liferay.portal.util.PortalUtil.getCDNHostHttp()Ljava/lang/String;
com.liferay.portal.events.ServicePreActionExt.servicePre(ServicePreActionExt.java:937)
After looking in the logs, I found the following messages (they are edited):
SEVERE: Error waiting for multi-thread deployment of directories to completehostConfig.deployWar=Deploying web application archive {0}
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1000)
WARN [DefaultConnectionTester:203] SQL State '08001' of Exception which occurred during a Connection test (fallback DatabaseMetaData test) implies that the database is invalid, and the pool should refill itself with fresh Connections.
com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][3.63.75] A communication error occurred during operations on the connection's underlying socket, socket input stream, or socket output stream. Error location: Reply.fill() - insufficient data (-1). Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
at com.ibm.db2.jcc.am.fd.a(fd.java:321)
WARN [DefaultConnectionTester:136] SQL State '08001' of Exception tested by statusOnException() implies that the database is invalid, and the pool should refill itself with fresh Connections.
WARN [C3P0PooledConnectionPool:708] A ConnectionTest has failed, reporting that all previously acquired Connections are likely invalid. The pool will be reset.
WARN [NewPooledConnection:486] [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
WARN [NewPooledConnection:487] [c3p0] Another error has occurred [ com.ibm.db2.jcc.am.SqlNonTransientConnectionException: [jcc][t4][10335][10366][3.63.75] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003 ] which will not be reported to listeners!
com.ibm.db2.jcc.am.SqlNonTransientConnectionException: [jcc][t4][10335][10366][3.63.75] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003
WARN [BasicResourcePool:1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4fad5112 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (3). Last acquisition attempt exception:
com.ibm.db2.jcc.am.SqlNonTransientConnectionException: DB2 SQL Error: SQLCODE=-20157, SQLSTATE=08004, SQLERRMC=FUT5MAN;QUIESCE DATABASE;;, DRIVER=3.63.75
ERROR [PortalJobStore:109] MisfireHandler: Error handling misfires: Unexpected runtime exception: null
org.quartz.JobPersistenceException: Unexpected runtime exception: null [See nested exception: java.lang.reflect.UndeclaredThrowableException]
Caused by: java.lang.reflect.UndeclaredThrowableException
at $Proxy279.prepareStatement(Unknown Source)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.countMisfiredTriggersInState(StdJDBCDelegate.java:413)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
Now, I see that it is almost impossible to start the current Liferay installation. However, I have the database (I made a full backup), and the lucene's data directory. How can I recreate a Liferay installation with these two things? I would like to recover some of this data in a new installation, but I do not how.

This is not the best solution, but I installed Liferay with a new database. Once it was configured, I change the database configuration in order to use the other one.
Probably, it was a problem with the ROOT deployment, but this is very weird.
I could recover all the data from the Lucene and the database.

The database is still quiesced and the Liferay user doesn't have the QUIESCE_CONNECT privilege.
Unquiesce the database and restart Liferay.
Using DB2 instance owner (if you're on Windows, any administrator):
db2 connect to DBNAME
db2 unquiesce database
db2 connect reset
Regards.

Related

Azure SQL database TransparentDataEncryptionProtector InternalServerError: An unexpected error occured while processing the request

I am trying to configure transparent data encryption for my Azure SQL databases using the steps documented in https://learn.microsoft.com/en-us/azure/sql-database/transparent-data-encryption-byok-azure-sql-configure
The command Set-AzSqlServerTransparentDataEncryptionProtector in step 3 is trowing me an error
InternalServerError: An unexpected error occured while processing the
request. Tracking ID: '8decc6a3-b1e5-4860-acc6-37b1114bcc33'.
When I look in the Azure portal at the Activity Log I see the same error "The resource operation completed with terminal provisioning state Failed.
"statusMessage":
"{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The
resource operation completed with terminal provisioning state
'Failed'.\",\"details\":[{\"code\":\"InternalServerError\",\"message\":\"An
unexpected error occured while processing the request. Tracking ID:
'82fbc867-e16a-4681-96af-d410400e97f9'\"}]}}"
Anybody any idea how to trace what goes wrong?

Weblogic 12.1.3 console gives 503 error

I am getting this error whenever I am trying to open console for weblogic. It was working fine a few days ago.
<A problem occurred while dispatching the servlet request.
weblogic.management.DeploymentException: <openjpa-1.1.1-SNAPSHOT-r422266:1445923 nonfatal user error> kodo.jdo.UserException: This operation cannot be performed while a Transaction is active.
at weblogic.application.services.BackgroundDeploymentService$OnDemandDeployer.deploy(BackgroundDeploymentService.java:173)
at weblogic.servlet.internal.OnWebUriDemandDeploymentProvider$OnWebUriDemandListener.OnDemandURIAccessed(OnWebUriDemandDeploymentProvider.java:46)
at weblogic.servlet.internal.OnDemandManager.loadOnDemandURI(OnDemandManager.java:113)
at weblogic.servlet.internal.MuxableSocketHTTP.handleOnDemandContext(MuxableSocketHTTP.java:210)
at weblogic.servlet.internal.HttpConnectionHandler.dispatch(HttpConnectionHandler.java:647)
Truncated. see log file for complete stacktrace
Caused By: <openjpa-1.1.1-SNAPSHOT-r422266:1445923 nonfatal user error> kodo.jdo.UserException: This operation cannot be performed while a Transaction is active.
at org.apache.openjpa.kernel.BrokerImpl.close(BrokerImpl.java:4087)
at org.apache.openjpa.kernel.DelegatingBroker.close(DelegatingBroker.java:1298)
at kodo.jdo.PersistenceManagerImpl.close(PersistenceManagerImpl.java:958)
at com.bea.security.providers.xacml.store.BasePolicyStore.setPolicy(BasePolicyStore.java:757)
at com.bea.security.providers.xacml.store.BasePolicyStore.setPolicy(BasePolicyStore.java:615)
Truncated. see log file for complete stacktrace
Error in IE on opening console
All the other applications deployed on it are working fine.
One of my clients is having the same problem but on WLS12.2.1.4.
So far, I have found 2 hints that could help you:
the embeddedLDAP is corrupted as described by kodo.jdo.UserException: Exception on Running OIM Patch_weblogic script (Doc ID 1108693.1). Solved after regenerating the embeddedLDAP for this particular managed server.
the server is affected by bug 20717082. The patch 20717082 is available for WLS10.3.6, WLS12.1.2, and WLS12.1.3.
Hope some of these options work for you. On my end, I'll keep on investigating.

Windows Azure Storage Exception with Connection String Specifying Account Name and Account Key

I am adding Windows Azure Storage to my ASP.Net application. I am using NetStandard 1.4 and am building on MacOS.
I have the following as my connection string, as recommended by learn.microsoft.com:
DefaultEndpointsProtocol=http;AccountName=XXXXXX;AccountKey=XXXXXX
I am connecting with a new Azure Storage account and have confirmed that my account name and account key are correct. Right before the call to create the table, I added a WriteLine to display the CloudStorageAccount's connection string URI just to make sure it is correct. It has the hostname.
I am getting the following error when performing "dotnet run".
Unhandled Exception: System.Reflection.TargetInvocationException:
Exception has been thrown by the target of an invocation. --->
System.AggregateException: One or more errors occurred. (An error
occurred while sending the request.) --->
Microsoft.WindowsAzure.Storage.StorageException: An error occurred
while sending the request. ---> System.Net.Http.HttpRequestException:
An error occurred while sending the request. --->
System.Net.Http.CurlException: Couldn't resolve host name
Setting explicit table storage endpoint (TableEndpoint=XXX.table.core.windows.net) didn't work either.
Are there any other reasons why Curl will fail when connecting to XXX.table.core.windows.net?
To try capture the network trace for troubleshooting? And try to ping XXX.table.core.windows.net? Is it because the DNS cache not refreshed?

EofException when doing a deployment using the Tooltwist Controller

I'm deploying a ToolTwist application to a production server using FIP, and Im getting this error on Transfer Phase.
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
and in the fipserver console
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:892)
at org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:486)
at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:424)
at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:78)
at org.eclipse.jetty.server.HttpConnection$Output.flush(HttpConnection.java:1094)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:159)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:98)
at tooltwist.fip.jetty.GetFileListServlet.doGet(GetFileListServlet.java:82)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:94)
what should be the solution for this?
This error is occuring in the first stage of the FIP file transfer, where the fipserver creates an index of the existing files on the destination server. This is done in GetFileListServlet.doGet(), which can be seen in the stack trace. It is also indicated on the client side by the message...
Indexing source...
Indexing destination...
ERROR: java.net.SocketTimeoutException Read timed out
Exception: tooltwist.fip.FipException: java.net.SocketTimeoutException: Read timed out
This indexing process involves creating a hash for each file on the destination server, which the fip client then compares with the hashes of files on the source machine. It does this to determine which files are different, and so need to be installed.
A read timeout occurs when the client is waiting too long for the FIP server to index the files on the destination machine. Indexing is normally a fairly quick process, but does involve reading all the files beneath the destination directory (e.g. in ~/server). If monsterously huge files exist within that destination directory then the scanning will take a proportionately long time to complete. If that time is too long, then the client times out and drops the connection, and the server also sees the connection was dropped and stops indexing.
The most common cause of this error is excessively large log files in ~/server/tomcat/logs. If you clean those up, the problem should go away.

Spring XD on YARN

I am getting the below error, while I am trying to install Spring XD on YARN.
Error executing a spring application; nested exception is org.springframework.yarn.YarnSystemException:
Call From c01dfobi43.vcac.dc1.dsghost.net/100.98.226.45 to c01dfobi41.vcac.dc1.dsghost.net:8032 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused;
nested exception is java.net.ConnectException:
Call From c01dfobi43.vcac.dc1.dsghost.net/100.98.226.45 to c01dfobi41.vcac.dc1.dsghost.net:8032 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Not sure where I am committing a mistake. Also do we need to install Spring XD Yarn on all nodes?
It would be great if you can share any documentation mentioned explicitly for YARN.
I am going to assume that c01dfobi41.vcac.dc1.dsghost.net:8032 is a ResourceManager host, I am also going to assume that based on your comment stating that yarn applications do run, you have more then one. In that case what may be happening (and I see this all the time) is that your yarn client attempts to contact the resource manager by looking it up in yarn-site.xml, it picks the first one and gets ConnectionRefused as the standby resource manager does not listen on its RPC port, it moves on to the next one and succeeds. If this is the case this is not a fatal error and can be ignored.