Worklight 6.2 Enterprise Server Deployment error - ibm-mobilefirst

Question

I am trying to deploy an application I have migrated from 6.1 to a 6.2 Server instance running as follows:
Liberty(8.5.5.3)
Derby(10.8)
SLES 11 linux on System z
java version "1.7.0" Java(TM) SE Runtime
Environment (build pxz6470sr6-20131015_01(SR6))
I am attempting deployment using the ant scripts.
I have run configure database script with WLREPORT, WRKLGHT and WLADMIN all successfully deployed. I have then run the configure application ant script with adminstall and install both successfully completing.
Yet when I run my liberty instance, i go to my /worklightconsole endpoint and I get a spinning wheel where I expect my available runtime list to appear. So next stop is the logs folder.
The following is my console.log
Launching worklight62 (WebSphere Application Server 8.5.5.3/wlp-1.0.6.cl50320140731-0257) on IBM J9 VM, version pxz6470sr6-20131015_01 (SR6) (en_GB)
[AUDIT ] CWWKE0001I: The server worklight62 has been launched.
[AUDIT ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT ] CWWKT0016I: Web application available (default_host): http://winlnx0f.hursley.ibm.com:9080/IBMJMXConnectorREST/
[AUDIT ] CWWKT0016I: Web application available (default_host): http://winlnx0f.hursley.ibm.com:9080/appcenterconsole/
[AUDIT ] CWWKZ0001I: Application appcenterconsole started in 0.981 seconds.
[AUDIT ] CWWKT0016I: Web application available (default_host): http://winlnx0f.hursley.ibm.com:9080/worklightconsole/
[AUDIT ] CWWKZ0001I: Application worklightconsole started in 2.065 seconds.
[WARNING ] SRVE9967W: The manifest class path jaxb-api.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path jaxb-impl.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path jsr173_1.0_api.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path activation.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path jaxb-api.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path jaxb-impl.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path jsr173_1.0_api.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path activation.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty8553/usr/servers/worklight62/apps/applicationcenter.war!/WEB-INF/lib/ibm_web20_jaxb-1.0.2.0-20110217.jar or its parent.
[AUDIT ] CWWKT0016I: Web application available (default_host): http://winlnx0f.hursley.ibm.com:9080/applicationcenter/
[WARNING ] CWNEN0070W: The javax.ws.rs.MatrixParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.HeaderParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.QueryParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.core.Context annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.PathParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.CookieParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[err] dataBaseFromDataSource=true
[err] dataBaseUrl=jdbc:derby:/var/ibm/Worklight/derby/APPCNTR
[err] dataBaseDriver=Apache Derby Embedded JDBC Driver
[err] dataBaseDriverVersion=10.8
[err] dataBaseProduct=Apache Derby
[err] dataBaseVersion=10.8.2.3 - (1326460)
[err] dataBaseUser=APPCENTER
[err] 168 appcenterdb-derby INFO [Default Executor-thread-2] openjpa.Runtime - Starting OpenJPA 1.2.2
[err] 296 appcenterdb-derby INFO [Default Executor-thread-2] openjpa.jdbc.JDBC - Using dictionary class "org.apache.openjpa.jdbc.sql.DerbyDictionary" (Apache Derby 10.8.2.3 - (1326460) ,Apache Derby Embedded JDBC Driver 10.8.2.3 - (1326460)).
[AUDIT ] CWWKZ0001I: Application applicationcenter started in 14.314 seconds.
[AUDIT ] CWWKT0016I: Web application available (default_host): http://winlnx0f.hursley.ibm.com:9080/worklight/
[ERROR ] SRVE0293E: [Servlet Error]-[Failed to load listener: com.worklight.server.bundle.project.JeeProjectActivator]: java.lang.NoClassDefFoundError: com.worklight.server.bundle.api.WorklightBundles
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:752)
at java.lang.ClassLoader.loadClass(ClassLoader.java:737)
at com.ibm.ws.classloading.internal.UnifiedClassLoader.loadClass0(UnifiedClassLoader.java:104)
at [internal classes]
[ERROR ] SRVE0279E: Error occured while processing global listeners for the application {0}: {1}
java.lang.NullPointerException
[ERROR ] com.worklight.oauth.AuthorizationServerApplication
com.worklight.oauth.AuthorizationServerApplication
[ERROR ] SRVE0276E: Error while initializing Servlet [AuthorizationServer]: javax.servlet.UnavailableException: com.worklight.oauth.AuthorizationServerApplication
at org.apache.wink.server.internal.servlet.RestServlet.init(RestServlet.java:116)
at javax.servlet.GenericServlet.init(GenericServlet.java:161)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:325)
at [internal classes]
[ERROR ] SRVE0321E: The [authenticationFilter] filter did not load during start up.
Filter [authenticationFilter]: Could not find required filter class - com.worklight.core.auth.impl.AuthenticationFilter.class
[AUDIT ] CWWKZ0001I: Application worklight started in 2.828 seconds.
[AUDIT ] CWWKT0016I: Web application available (default_host): http://winlnx0f.hursley.ibm.com:9080/worklightadmin/
[WARNING ] CWNEN0070W: The javax.ws.rs.MatrixParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.HeaderParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.QueryParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.core.Context annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.PathParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[WARNING ] CWNEN0070W: The javax.ws.rs.CookieParam annotation class will not be recognized because it was loaded from the null location rather than from a product class loader.
[AUDIT ] CWWKF0011I: The server worklight62 is ready to run a smarter planet.
[err] 61 WorklightManagementPU-derby INFO [Default Executor-thread-2] openjpa.Runtime - Starting OpenJPA 1.2.2
[err] 184 WorklightManagementPU-derby INFO [Default Executor-thread-2] openjpa.jdbc.JDBC - Using dictionary class "org.apache.openjpa.jdbc.sql.DerbyDictionary" (Apache Derby 10.8.2.3 - (1326460) ,Apache Derby Embedded JDBC Driver 10.8.2.3 - (1326460)).
[err] 1249 WorklightManagementPU-derby INFO [Default Executor-thread-2] openjpa.Runtime - Though you are using optimistic transactions, OpenJPA is now beginning a datastore transaction because you have requested a lock on some data.
[err] 1333 WorklightManagementPU-derby INFO [Default Executor-thread-2] openjpa.Runtime - Though you are using optimistic transactions, OpenJPA is now beginning a datastore transaction because you have requested a lock on some data.
[AUDIT ] CWWKZ0001I: Application worklightadmin started in 11.130 seconds.
[AUDIT ] CWWKE0055I: Server shutdown requested on Thursday, 28 August 2014 at 02:43. The server worklight62 is shutting down.
[AUDIT ] CWWKT0017I: Web application removed (default_host): http://winlnx0f.hursley.ibm.com:9080/IBMJMXConnectorREST/
[AUDIT ] WTRN0105I: The transaction service has shutdown successfully with no transactions requiring recovery.
[AUDIT ] CWWKZ0009I: The application worklight has stopped successfully.
[AUDIT ] CWWKZ0009I: The application worklightadmin has stopped successfully.
[AUDIT ] CWWKZ0009I: The application applicationcenter has stopped successfully.
[AUDIT ] CWWKZ0009I: The application appcenterconsole has stopped successfully.
[AUDIT ] CWWKZ0009I: The application worklightconsole has stopped successfully.
[AUDIT ] CWWKE0036I: The server worklight62 stopped after 8 minutes, 47.886 seconds.
In here I do not like the look of :
ERROR ] SRVE0293E: [Servlet Error]-[Failed to load listener: com.worklight.server.bundle.project.JeeProjectActivator]: java.lang.NoClassDefFoundError
But having googled and searched I cannot determine what I need to add / fix.
This is the messages.log:
http://pastebin.com/3G99a7gW
I am happy to include ffdc if it will help. As far as I am concerned this has all be done 100% by the book, following step by step.
This is my server.xml:
<server description="new server">
<!-- Enable features -->
<featureManager>
<feature>jsp-2.2</feature>
<!-- Begin of features added by IBM Worklight installer. -->
<feature>ssl-1.0</feature>
<feature>servlet-3.0</feature>
<feature>jdbc-4.0</feature>
<feature>appSecurity-1.0</feature>
<feature>jndi-1.0</feature>
<!-- End of features added by IBM Worklight installer. -->
<!-- Begin of features added by IBM Worklight <configureApplicationServer> ant task for context root '/worklight'. -->
<feature>ssl-1.0</feature>
<feature>servlet-3.0</feature>
<feature>jdbc-4.0</feature>
<feature>jndi-1.0</feature>
<!-- End of features added by IBM Worklight <configureApplicationServer> ant task for context root '/worklight'. -->
<!-- Begin of features added by IBM Worklight <installWorklightAdmin> ant task for context root '/worklightadmin'. -->
<feature>ssl-1.0</feature>
<feature>servlet-3.0</feature>
<feature>jdbc-4.0</feature>
<feature>jndi-1.0</feature>
<feature>restConnector-1.0</feature>
<feature>appSecurity-1.0</feature>
<!-- End of features added by IBM Worklight <installWorklightAdmin> ant task for context root '/worklightadmin'. -->
</featureManager>
<!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
<httpEndpoint id="defaultHttpEndpoint"
httpPort="9080"
httpsPort="9443" host="*" >
<!-- Option soReuseAddr added by IBM Worklight installer. -->
<!-- Option soReuseAddr added by IBM Worklight <configureApplicationServer> ant task for context root '/worklight'. -->
<!-- Option soReuseAddr added by IBM Worklight <installWorklightAdmin> ant task for context root '/worklightadmin'. -->
<tcpOptions soReuseAddr="true"/>
</httpEndpoint>
<!-- Begin of configuration added by IBM Worklight installer. -->
<!-- Declare the IBM Application Center Console application. -->
<application id="appcenterconsole" name="appcenterconsole" location="appcenterconsole.war" type="war">
<application-bnd>
<security-role name="appcenteradmin">
<group name="appcentergroup"/>
</security-role>
</application-bnd>
</application>
<!-- Declare the IBM Application Center Services application. -->
<application id="applicationcenter" name="applicationcenter" location="applicationcenter.war" type="war">
<application-bnd>
<security-role name="appcenteradmin">
<group name="appcentergroup"/>
</security-role>
</application-bnd>
<classloader delegation="parentLast">
<commonLibrary>
<fileset dir="${wlp.install.dir}/lib" includes="com.ibm.ws.crypto.passwordutil_1.0.jar"/>
</commonLibrary>
</classloader>
</application>
<!-- Declare the user registry for the IBM Application Center. -->
<basicRegistry id="applicationcenter-registry" realm="ApplicationCenter">
<!-- Worklight user.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<user name="WorklightRESTUser" password="BCEN3GMmayMY"/>
<!-- Worklight user.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<user name="wlAdmin" password="admin"/>
<!-- The users defined here are members of group "appcentergroup",
thus have role "appcenteradmin", and can therefore perform
administrative tasks through the IBM Application Center Console. -->
<user name="appcenteradmin" password="admin"/>
<user name="demo" password="demo"/>
<group name="appcentergroup">
<member name="appcenteradmin"/>
<member name="demo"/>
</group>
</basicRegistry>
<!-- Declare the JNDI properties for the IBM Application Center. -->
<!-- Define the AppCenter services endpoint in order for the AppCenter console to be able to invoke the REST service.
You need to enable this property if the server is behind a reverse proxy
or if the context root of the Application Center Services application is different from '/applicationcenter'. -->
<!-- <jndiEntry jndiName="ibm.appcenter.services.endpoint" value='"http://proxyhost:proxyport/applicationcenter"'/> -->
<!-- The directory with binaries of the 'aapt' program, from the Android SDK's platform-tools package. -->
<jndiEntry jndiName="android.aapt.dir" value='"/opt/IBM/Worklight62/ApplicationCenter/tools/android-sdk"'/>
<!-- The protocol of the application resources URI. This property is optional. It is only needed if the protocol of the external and internal URI are different. -->
<!-- <jndiEntry jndiName="ibm.appcenter.proxy.protocol" value='"http"'/> -->
<!-- The hostname of the application resources URI. -->
<!-- <jndiEntry jndiName="ibm.appcenter.proxy.host" value='"proxyhost"'/> -->
<!-- The port of the application resources URI. This property is optional. -->
<!-- <jndiEntry jndiName="ibm.appcenter.proxy.port" value="proxyport"/> -->
<!-- Declare the jar file for Derby with the "embedded" deployment option. -->
<library id="DerbyLib">
<fileset dir="${shared.resource.dir}/derby" includes="derby.jar"/>
</library>
<!-- Declare the IBM Application Center database. -->
<dataSource jndiName="jdbc/AppCenterDS" transactional="false">
<jdbcDriver libraryRef="DerbyLib"/>
<properties.derby.embedded databaseName="/var/ibm/Worklight/derby/APPCNTR" user="APPCENTER"/>
</dataSource>
<!-- End of configuration added by IBM Worklight installer. -->
<!-- Begin of configuration added by IBM Worklight <configureApplicationServer> ant task for context root '/worklight'. -->
<!-- Declare the IBM Worklight project runtime application. -->
<application id="worklight" name="worklight" location="GENAPPMobileSample.war" type="war">
<classloader delegation="parentLast">
<privateLibrary>
<fileset dir="${shared.resource.dir}/worklight/lib" includes="worklight-jee-library.jar"/>
</privateLibrary>
</classloader>
</application>
<!-- Declare web container custom properties for the IBM Worklight project runtime application. -->
<webContainer invokeFlushAfterService="false" deferServletLoad="false"/>
<!-- Declare the JNDI properties for the IBM Worklight project runtime. -->
<jndiEntry jndiName="worklight/publicWorkLightProtocol" value='"http"'/>
<jndiEntry jndiName="worklight/publicWorkLightPort" value='"9080"'/>
<!-- Declare the jar file for Derby with the "embedded" deployment option. -->
<library id="worklight/DerbyLib">
<fileset dir="${shared.resource.dir}/worklight/derby" includes="derby.jar"/>
</library>
<!-- Declare the IBM Worklight Server database. -->
<dataSource jndiName="worklight/jdbc/WorklightDS" transactional="false">
<jdbcDriver libraryRef="worklight/DerbyLib"/>
<properties.derby.embedded databaseName="/var/ibm/Worklight/derby/WRKLGHT" user='"WORKLIGHT"'/>
</dataSource>
<!-- Declare the IBM Worklight Server reports database. -->
<dataSource jndiName="worklight/jdbc/WorklightReportsDS" transactional="false">
<jdbcDriver libraryRef="worklight/DerbyLib"/>
<properties.derby.embedded databaseName="/var/ibm/Worklight/derby/WLREPORT" user='"WORKLIGHT"'/>
</dataSource>
<!-- End of configuration added by IBM Worklight <configureApplicationServer> ant task for context root '/worklight'. -->
<!--
IBM Worklight requires SSL and declared the "defaultKeyStore" default keystore.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
This configuration is the minimum one that you need to create an SSL configuration.
With this configuration, the Liberty server creates the keystore and the certificate,
if it does not exist yet, during the SSL initialization.
The created certificate is a self-signed certificate that is valid for 365 days.
Do not use the certificates that the Liberty server created for production use.
For more information see http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.wlp.core.doc/ae/twlp_sec_ssl.html
-->
<keyStore id="defaultKeyStore" password="worklight"/>
<administrator-role>
<!-- Worklight JMX User.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<user>WorklightRESTUser</user>
</administrator-role>
<!-- Worklight JNDI property for JMX connection.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<jndiEntry jndiName="ibm.worklight.admin.jmx.host" value="localhost"/>
<!-- Worklight JNDI property for JMX connection.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<jndiEntry jndiName="ibm.worklight.admin.jmx.port" value="9443"/>
<!-- Worklight JNDI property for JMX connection.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<jndiEntry jndiName="ibm.worklight.admin.jmx.user" value="WorklightRESTUser"/>
<!-- Worklight JNDI property for JMX connection.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<jndiEntry jndiName="ibm.worklight.admin.jmx.pwd" value="BCEN3GMmayMY"/>
<!-- Worklight JNDI property for JMX connection.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<jndiEntry jndiName="ibm.worklight.topology.platform" value="Liberty"/>
<!-- Worklight JNDI property for JMX connection.
[Added by IBM Worklight <installWorklightAdmin> Ant task for context root '/worklightadmin']
-->
<jndiEntry jndiName="ibm.worklight.topology.clustermode" value="Standalone"/>
<!-- Begin of configuration added by IBM Worklight <installWorklightAdmin> ant task for context root '/worklightadmin'. -->
<!-- Declare the Worklight Administration Service application. -->
<application id="worklightadmin" name="worklightadmin" location="worklightadmin.war" type="war">
<application-bnd>
<security-role name="worklightadmin">
<user name="wlAdmin"/>
</security-role>
<security-role name="worklightdeployer">
</security-role>
<security-role name="worklightmonitor">
</security-role>
<security-role name="worklightoperator">
</security-role>
</application-bnd>
<classloader delegation="parentLast">
<commonLibrary>
<fileset dir="${wlp.install.dir}/lib" includes="com.ibm.ws.crypto.passwordutil_1.0.4.jar"/>
</commonLibrary>
</classloader>
</application>
<!-- Declare the JNDI properties for the Worklight Administration Service. -->
<!-- Declare the jar file for Derby with the "embedded" deployment option. -->
<library id="worklightadmin/DerbyLib">
<fileset dir="${shared.resource.dir}/worklightadmin/derby" includes="derby.jar"/>
</library>
<!-- Declare the IBM Worklight Administration database. -->
<dataSource jndiName="worklightadmin/jdbc/WorklightAdminDS" transactional="false">
<jdbcDriver libraryRef="worklightadmin/DerbyLib"/>
<properties.derby.embedded databaseName="/var/ibm/Worklight/derby/WLADMIN" user='"WLADMINISTRATOR"'/>
</dataSource>
<!-- Declare the Worklight Administration Console application. -->
<application id="worklightconsole" name="worklightconsole" location="worklightconsole.war" type="war">
<application-bnd>
<security-role name="worklightadmin">
<user name="wlAdmin"/>
</security-role>
<security-role name="worklightdeployer">
</security-role>
<security-role name="worklightmonitor">
</security-role>
<security-role name="worklightoperator">
</security-role>
</application-bnd>
</application>
<!-- Declare the JNDI properties for the Worklight Administration Console. -->
<jndiEntry jndiName="worklightconsole/ibm.worklight.admin.endpoint" value='"*://*:*/worklightadmin"'/>
<!-- End of configuration added by IBM Worklight <installWorklightAdmin> ant task for context root '/worklightadmin'. -->
</server>
Advice is very much appreciated.

Answer 1

From the comments: I believe WebSphere Liberty profile v8.5.5.2/3 is not a good pick for Worklight 6.2 at this time. Instead, use v8.5.5.1.
See here: No Runtime can be found in Worklight Console