IllegalAccessError for RequestHedgingRMFailoverProxyProvider while launching Apache Twill Application in hadoop cluster after HDP upgrade - hadoop-yarn

I'm trying to launch Apache Twill application from hadoop cluster, the cluster is recently upgraded from HDP 2.2 to HDP 2.5 but I'm getting llegalAccessError for RequestHedgingRMFailoverProxyProvider class . This class is part of org.apache.hadoop.yarn.client package. I'm getting this error in the Application Master. The job status goes directy to 'not running state' right after 'accepted state'.
Exception in thread "Hadoop22YarnAMClient STARTING" Exception in thread "YarnAMClientService STARTING" java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider
at org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75)
at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163)
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93)
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.serviceStart(AMRMClientImpl.java:186)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.twill.internal.yarn.Hadoop21YarnAMClient.startUp(Hadoop21YarnAMClient.java:77)
at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43)
at java.lang.Thread.run(Thread.java:745)
com.google.common.util.concurrent.ExecutionError: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1008)
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001)
at com.google.common.util.concurrent.AbstractService.startAndWait(AbstractService.java:220)
at com.google.common.util.concurrent.AbstractIdleService.startAndWait(AbstractIdleService.java:106)
at org.apache.twill.internal.appmaster.ApplicationMasterMain$YarnAMClientService.startUp(ApplicationMasterMain.java:221)
at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider
at org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75)
at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163)
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93)
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.serviceStart(AMRMClientImpl.java:186)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.twill.internal.yarn.Hadoop21YarnAMClient.startUp(Hadoop21YarnAMClient.java:77)
... 2 more

In general when you see IllegalAccessError this means you have a runtime incompatibility between compiled and runtime code. In this case,
the getProxyInternal() method of ConfiguredRMFailoverProxyProvider is now private. You need to recompile your client code and/or use updated hadoop client libraries to connect to your cluster.

Related

Apache ignite startup failure

Apache ignite .net core server node fails to start with the below error, any idea what could be the reason?
Apache.Ignite.Core.Common.IgniteException
HResult=0x80131500
Message=Java exception occurred [class=java.lang.ExceptionInInitializerError, message=]
Source=Apache.Ignite.Core
StackTrace:
at Apache.Ignite.Core.Ignition.Start(IgniteConfiguration cfg)
at IgniteShared.Cache.CacheService.<Start>b__3_0() in C:\Users\user\IgnitePoc\IgniteShared\Cache.cs
This exception was originally thrown at this call stack:
[External Code]
Inner Exception 1:
JavaException: java.lang.ExceptionInInitializerError
at org.apache.ignite.internal.processors.platform.memory.PlatformAbstractMemory.<clinit>(PlatformAbstractMemory.java:27)
at org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.init(PlatformAbstractBootstrap.java:54)
at org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:65)
Caused by: java.lang.RuntimeException: java.nio.DirectByteBuffer.address field is unavailable.
at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1570)
at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1557)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
at org.apache.ignite.internal.util.GridUnsafe.bufferAddressOffset(GridUnsafe.java:1557)
at org.apache.ignite.internal.util.GridUnsafe.<clinit>(GridUnsafe.java:109)
... 3 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module #69663380
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
at org.apache.ignite.internal.util.GridUnsafe$2.run(GridUnsafe.java:1562)
... 7 more
Apache Ignite requires Java 8 or Java 11.
Java 17 is not yet supported.
https://ignite.apache.org/docs/latest/quick-start/dotnet
(update: Java 17 support is coming soon: IGNITE-16622)
I faced a similar issue when launching Ignite from a java application (the start-up script ignite.sh wasn't used); with Java 17.0.3 and Ignite 2.13.0. At that time, the documentation only described the required JVM options for running Ignite with Java 11 (Running Ignite with Java 11). So, I found the necessary JVM options for running Ignite with Java 17 in the source code (jvmdefaults.sh).

javax.management.ReflectionException: The MBean class could not be loaded by the default loader repository

We are migrating JBOSS from 4.0.3SP1 to Wildfly 10.1.0. Our applications are bundled in separate sars which contain standard as well as Dynamic beans.
We are getting "javax.management.ReflectionException: The MBean class could not be loaded by the default loader repository" exception caused by "java.lang.ClassNotFoundException: com.xxx.ccr.common.adapter.PEAdapterLCM". Stack Trace is added in the end. PEAdapter class is a Dynamic MBean and is present in one of the jars in the common modules under /opt/coreservices/wildfly-10.1.0.Final/modules.
We are using standalone-full.xml to start our Wildfly instance (ccr2)
[root#puiqr710dev08 CCR]# ps -eaf|grep ccr2
xxxiq 28948 7529 21 15:17 ? 00:07:10 /opt/Xxx/CCR/jre/bin/java -verbose:class -Ddss.port=31002 -DCONTAINER_UUID=14132c5860baba870160bac5d84e084a -Dlcm.host= xxx.xxxx.xxx.xxx -server -XX:NewRatio=1 -XX:+UseG1GC -XX:+UseLargePages -XX:MaxGCPauseMillis=1000 -XX:GCTimeRatio=10 -XX:+DisableExplicitGC -Dsun.nio.ch.disableSystemWideOverlappingFileLockCheck=true -Ddss.thread_pool_size=24 -Xss250k -Xms256m -Xmx4096m -DUSE_DELAY1=60000 -DUSE_DELAY2=60000 -Dlog4j.configuration=file:/opt/Xxx/CCR/appserver/jboss-boot.log4j.properties -DLOG_FILE_PREFIX=DataProcessingJBoss_puiqdevdads07 -Ddss.message_lifetime=180 -Djava.awt.headless=true -Dorg.jboss.logging.Log4jService.catchSystemErr=false -classpath /opt/Xxx/CCR/jre/lib/tools.jar -jar /opt/coreservices/wildfly-10.1.0.Final/jboss-modules.jar -mp /opt/coreservices/wildfly-10.1.0.Final/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/opt/coreservices/wildfly-10.1.0.Final -Djboss.server.base.dir=/opt/coreservices/wildfly-10.1.0.Final/ccr2 -Djboss.bind.address=xxx.xxxx.xxx.xxx -Djboss.bind.address.management= xxx.xxxx.xxx.xxx -c standalone-full.xml
· We have four sars that are deployed in this instance. All sars are deployed successfully.
· We have packaged our sars as follows:
--lib (contains all jars)
--META-INF
|--jboss-deployment-structure.xml
|--jboss-service.xml
· Each sar contains some standard MBeans and few Dynamic MBeans.
· We have defined standard MBeans in jboss-services.xml. They are created properly and can be seen in JConsole.
· We can NOT include definition of Dynamic MBeans in jboss-service.xml as their name is constructed at runtime.
· PEAdapterLCM for which we are getting ClassNotFoundException is a dynamic bean and we are getting this error for all Dynamic MBeans.
· We have created modules for all common jars that were part of “server/lib” folder in JBOSS 4.0.3SP1.
We tried following to fix this error but no luck:
1) packaged contents of all sars in one single sar and deployed it.
2) Added a global module entry in standalone-full.xml which contents dynamic MBeans.
3) Added definition of dynamic MBean in jboss-service.xml (just to see if this makes any difference). With this change, we were able to see the MBean in JConsole. But still got the same error.
So what should be done to fix this error. Are there any changes - how to implement and deploy Dynamic MBeans in Wildfly?
Is there any way in Wildfly we can explicitly mention the MBean server class while starting the Wildfly as it was done in JBOSS 4.0.3SP1 using "-Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl -Djboss.platform.mbeanserver -Dcom.sun.management.jmxremote"
Here is complete stack trace:
^[[0m^[[31m15:01:30,324 ERROR [com.xxx.coreservice.lifecycle.jmx.PEController] (MSC service thread 1-6) Exception in PEController:PEControllerJmxJBossService start for peID 14132c58609742440160974fe69307e0: : javax.management.ReflectionException: The MBean class could not be loaded by the default loader repository
at com.sun.jmx.mbeanserver.MBeanInstantiator.findClassWithDefaultLoaderRepository(MBeanInstantiator.java:104)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.createMBean(DefaultMBeanServerInterceptor.java:268)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.createMBean(DefaultMBeanServerInterceptor.java:206)
at com.sun.jmx.mbeanserver.JmxMBeanServer.createMBean(JmxMBeanServer.java:326)
at com.avaya.lifecycle.jmx.agent.PEControllerJmx._start(PEControllerJmx.java:55)
at com.xxx.coreservice.lifecycle.jmx.PEController.start(PEController.java:397)
at com.xxx.coreservice.lifecycle.jmx.PEController.startAll(PEController.java:314)
at com.xxx.coreservice.lifecycle.jmx.PEController.startup(PEController.java:238)
at com.xxx.coreservice.lifecycle.jmx.PEController.postRegister(PEController.java:700)
at com.sun.jmx.mbeanserver.MBeanSupport.postRegister(MBeanSupport.java:182)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postRegister(DefaultMBeanServerInterceptor.java:1024)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:974)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.registerMBean(PluggableMBeanServerImpl.java:1527)
at org.jboss.as.jmx.PluggableMBeanServerImpl.registerMBean(PluggableMBeanServerImpl.java:871)
at org.jboss.as.jmx.MBeanRegistrationService.start(MBeanRegistrationService.java:101)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.xxx.ccr.common.adapter.PEAdapterLCM
at com.sun.jmx.mbeanserver.ClassLoaderRepositorySupport.loadClass(ClassLoaderRepositorySupport.java:232)
Thanks for help in advance.

NoSuchMethodError in Tomcat embedded MULE when executing http:set-cookie

When running Mule ESB 3.2.1 as embedded server inside Tomcat 7.0.27 (executed with webapp-runner), during execution of a flow with the Http endpoint, while sending response back to caller, an Exception is raised:
java.lang.NoSuchMethodError: org.apache.tomcat.util.http.ServerCookie.appendCookieValue(Ljava/lang/StringBuffer;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;IZ)V
Exception Below:
org.mule.api.MuleRuntimeException: Connector that caused exception is: connector.http.mule.default
at org.mule.transport.AbstractConnector.handleWorkException(AbstractConnector.java:2034)
at org.mule.transport.AbstractConnector.workCompleted(AbstractConnector.java:1998)
at org.mule.work.WorkerContext.run(WorkerContext.java:369)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NoSuchMethodError: org.apache.tomcat.util.http.ServerCookie.appendCookieValue(Ljava/lang/StringBuffer;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;IZ)V
at org.mule.transport.http.CookieHelper.formatCookieForASetCookieHeader(CookieHelper.java:310)
at org.mule.transport.http.transformers.MuleMessageToHttpResponse.createResponse(MuleMessageToHttpResponse.java:261)
at org.mule.transport.http.transformers.MuleMessageToHttpResponse.transformMessage(MuleMessageToHttpResponse.java:90)
at org.mule.transformer.AbstractMessageTransformer.transform(AbstractMessageTransformer.java:145)
at org.mule.transformer.AbstractMessageTransformer.transform(AbstractMessageTransformer.java:93)
at org.mule.DefaultMuleMessage.applyAllTransformers(DefaultMuleMessage.java:1387)
at org.mule.DefaultMuleMessage.applyTransformers(DefaultMuleMessage.java:1348)
at org.mule.DefaultMuleMessage.applyTransformers(DefaultMuleMessage.java:1331)
at org.mule.transport.AbstractMessageReceiver.applyResponseTransformers(AbstractMessageReceiver.java:235)
at org.mule.transport.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:214)
at org.mule.transport.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:163)
at org.mule.transport.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:150)
at org.mule.transport.http.HttpMessageReceiver$HttpWorker.doRequest(HttpMessageReceiver.java:299)
at org.mule.transport.http.HttpMessageReceiver$HttpWorker.processRequest(HttpMessageReceiver.java:258)
at org.mule.transport.http.HttpMessageReceiver$HttpWorker.run(HttpMessageReceiver.java:163)
at org.mule.work.WorkerContext.run(WorkerContext.java:310)
If you are using Mule 3.2.1, you can not use the http:response-builder. The feature is just not there. That's why you can't use it.
Check it out:
It's not in the doc: http://www.mulesoft.org/documentation-3.2/display/32X/HTTP+Transport+Reference
It's not in the source: https://github.com/mulesoft/mule/blob/mule-3.2.1/transports/http/src/main/java/org/mule/transport/http/config/HttpNamespaceHandler.java#L56
I had this problem too. Check answer here.
Mule ESB does not work with cookie
In short, make sure you have provided group: 'org.apache.tomcat', name: 'coyote', version: '6.0.44' for mule 3.7.0.
In your case, you should have another library - tomcat-util 5.5.23 because you use different version of mule 3.2.1.
compile group: 'tomcat', name: 'tomcat-util', version: '5.5.23'
When you use SpringBoot, it ovverides a version of some library (depending on version of Spring and Mule), so you will get this error. You have class ServerCookie but the method appendCookieValue will disappear!
Solution - play with libraries. Or another bad workaround, write your own version of one of these classes that fail, and make sure classloader will use your version of the classes. (Again, creating a class with the same name and package to fix a bug - is a risky and bad thing...)

Error while deploying ADF application in glassfish

Hello I am using JDeveloper 11.1.2.3.0
I have configured Glassfish in my computer and I have followed the instructions as Shay explained here: https://blogs.oracle.com/shay/entry/deploying_oracle_adf_applications_to
The problem is that when I try to deploy my ADF application as "Deploy to application server" with glassfish in this case I get an error saying that:
[#|2013-08-21T11:45:47.516+0200|SEVERE|glassfish3.1.2|org.apache.catalina.core.ContainerBase|_ThreadID=62;_ThreadName=Thread-2;|ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: java.lang.IllegalArgumentException: java.lang.ClassNotFoundException: oracle.adf.share.glassfish.listener.ADFGlassFishAppLifeCycleListener
If I deploy the ADF aplication as an EAR file and then I try to deploy this EAR file to glassfish through the admin interface I get this other error:
[#|2013-08-21T15:40:16.452+0200|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=65;_ThreadName=Thread-2;|Exception while invoking class com.sun.enterprise.web.WebApplication start method
java.lang.Exception: java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException: CONFIGURATION FAILED! javax.el.ELContext
Can anyone help on this?
A few things to check-
Did you extract the adf-essentials zip file with the -j option?
Did you mark both your model and view project to have a deployment platform for Glassfish?\
Please make sure that application is deleted from application folder. If not stop-admin server and delete it manually.

Glassfish 3.1.1 start-local-instance fails with JAXBException

I am setting up a GlassFish cluster following the guide at http://javadude.wordpress.com/2011/04/25/glassfish-3-1-clustering-tutorial/. I started from fresh installs of GlassFish 3.1.1. I also have the same architecture as in the guide: two nodes with one instance each. The DAS is on node1.
I've tried starting from scratch several times and am able to create the cluster, nodes and instances without issue. I also have the DAS communicating with node2 via SSH. However, each time when I get to the point where I attempt to start instance2 it fails:
$ ./asadmin start-local-instance --node node1 --sync normal instance2
Previous synchronization failed at Feb 23, 2012 2:41:53 PM
Will perform full synchronization.
Removing all cached state for instance instance2.
CLI802 Synchronization failed for directory config, caused by:
javax.xml.bind.JAXBException
- with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.bind.v2.ContextFactory]
Command start-local-instance failed.
I spent the day Googling and searching GlassFish's Jira, but couldn't find a solution to this issue. I'd very much appreciate any ideas you have on how to solve this problem.
My operating system is CentOS 5.7 and my Java version is 1.6.0_20
Unfortunately my instance directory is empty, I'm assuming because it never started. So there is no log file. I set AS_DEBUG=true but it gives no stack trace. The last debug lines before the error are
Removing all cached state for instance instance2.
Removing: /usr/local/glassfish3_1_1/glassfish/nodes/blade-50/instance2/config
Removing: /usr/local/glassfish3_1_1/glassfish/nodes/blade-50/instance2/applications
Removing: /usr/local/glassfish3_1_1/glassfish/nodes/blade-50/instance2/generated
Removing: /usr/local/glassfish3_1_1/glassfish/nodes/blade-50/instance2/lib
Removing: /usr/local/glassfish3_1_1/glassfish/nodes/blade-50/instance2/docroot
Got exception: javax.xml.bind.JAXBException
Acting on a tip from a user in the Glassfish forum, I learned that Java 1.6.0_20 is an older release of Java that is not supported by Glassfish 3.1.1. I worked with a sysadmin to get Java 1.6.0_31 installed on both nodes of the cluster and that did the trick--both instances start up without errors.