IBM MPF8.0 409 Response from Push Tag POST - ibm-mobilefirst

In a Java adapter when we create a new tag using Push Tag (POST) we always get a "The target resource 'PushTag' already exists" 409 response and an error in the log even though the new tag is successfully created
Is this a known issue? Is there a work-around?
In the logs the sequence is like this;
[INFO ] 2017-10-16-07:51:48.318 [1363] PostRequest.call http://localhost:9080/imfpush/v1/apps/com.ibm.mfpstartercordova/tags/TAG2203137
GET
Authorization: Bearer eyJhbGci...
[ERROR ] FPWSE0010E: Internal server error. FPWSE0001E: Not Found - The target resource 'PushTag' does not exist. Check the 'TAG2203137' parameter.
[ERROR ] FPWSE0010E: Internal server error. FPWSE0001E: Not Found - The target resource 'PushTag' does not exist. Check the 'TAG2203137' parameter.
http://localhost:9080/imfpush/v1/apps/com.ibm.mfpstartercordova/tags/TAG2203137
GET
404
{OkHttp-Sent-Millis=1508158308325, Date=Mon, 16 Oct 2017 12:51:48 GMT, Content-Length=125, OkHttp-Received-Millis=1508158308332, Connection=Close, Content-Type=application/json, X-Powered-By=Servlet/3.1}
{"message":"Not Found - The target resource 'PushTag' does not exist. Check the 'TAG2203137' parameter.","code":"FPWSE0001E"}
[INFO ] 2017-10-16-07:51:48.333 [1363] PostRequest.call http://localhost:9080/imfpush/v1/apps/com.ibm.mfpstartercordova/tags
POST
Authorization: Bearer eyJhbGci...
Content-Type: application/json
{"description":"TAG2203137","name":"TAG2203137"}
[err] com.ibm.mfp.push.server.exceptions.PushWorksEntityExistsException: FPWSE0002E: Conflict - The target resource 'PushTag' already exists. Check the 'TAG2203137' parameter.
[err] at com.ibm.mfp.push.server.rest.resources.TagResource.createTag(TagResource.java:283)
[err] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[err] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[err] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[err] at java.lang.reflect.Method.invoke(Method.java:606)
[err] at org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:63)
[err] at org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)
[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)
etc, etc
[err] at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1287)
[err] at [internal classes]
[err] at com.ibm.mfp.push.server.rest.SecurityFilter.doFilter(SecurityFilter.java:93)
[err] at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:207)
[err] at [internal classes]
[err] at com.ibm.mfp.push.server.rest.filter.RequestDetailLogger.doFilter(RequestDetailLogger.java:94)
[err] at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:207)
[err] at [internal classes]
[err] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[err] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[err] at java.lang.Thread.run(Thread.java:745)
[ERROR ] FPWSE0010E: Internal server error. FPWSE0002E: Conflict - The target resource 'PushTag' already exists. Check the 'TAG2203137' parameter.
http://localhost:9080/imfpush/v1/apps/com.ibm.mfpstartercordova/tags
POST
409
{OkHttp-Sent-Millis=1508158308345, Date=Mon, 16 Oct 2017 12:51:48 GMT, Content-Length=124, OkHttp-Received-Millis=1508158308372, Connection=Close, Content-Type=application/json, X-Powered-By=Servlet/3.1}
{"message":"Conflict - The target resource 'PushTag' already exists. Check the 'TAG2203137' parameter.","code":"FPWSE0002E"}
[err] java.io.IOException: Could not create http://localhost:9080/imfpush/v1/apps/com.ibm.mfpstartercordova/tags
[err] at com.onemain.mfp.push.PushMessage$oauthInstance.pushToTag(PushMessage.java:153)
[err] at com.onemain.mfp.push.pushAdapterApplication.handleMessage(pushAdapterApplication.java:99)
[err] at com.onemain.mfp.push.pushAdapterApplication.waitForMessage(pushAdapterApplication.java:127)
[err] at com.onemain.mfp.push.pushAdapterApplication.pollLocalMQ(pushAdapterApplication.java:158)
[err] at com.onemain.mfp.push.pushAdapterApplication.access$200(pushAdapterApplication.java:48)
[err] at com.onemain.mfp.push.pushAdapterApplication$1.run(pushAdapterApplication.java:255)

Related

ActiveMQ Error Message for Failed to retrieve RMIServer stub

I am using ActiveMQ with version 5.12.1. When I shut down the server today, I got the error message. I could start the ActiveMQ but I got the same error message when I shut it down. The shutdown worked fine before. Could anyone tell what the message mean? And how I may resolve the problem. Thanks,
ACTIVEMQ_HOME: /app/actmq/apache-activemq-5.12.1
ACTIVEMQ_BASE: /app/actmq/apache-activemq-5.12.1
ACTIVEMQ_CONF: /app/actmq/apache-activemq-5.12.1/conf
ACTIVEMQ_DATA: /app/actmq/apache-activemq-5.12.1/data
Connecting to pid: 11546
INFO: failed to resolve jmxUrl for pid:11546, using default JMX url
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
ERROR: java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi
java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi
at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java:116)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxCommand.java:387)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:607)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
ERROR: java.lang.Exception: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi
java.lang.Exception: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi
at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java:117)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxCommand.java:387)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:607)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:227)
at org.apache.activemq.console.command.AbstractJmxCommand.createJmxConnector(AbstractJmxCommand.java:279)
at org.apache.activemq.console.command.AbstractJmxCommand.createJmxConnection(AbstractJmxCommand.java:302)
at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java:81)
... 11 more
Caused by: javax.naming.NameNotFoundException: jmxrmi
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:140)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:204)
at javax.naming.InitialContext.lookup(InitialContext.java:415)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1928)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1895)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 16 more
...............................

Mobilefirst Push Notifications for IOS Native

I have to implement push notification on IOS native mobile application, currently I am doing small POC using Mobilefirst 8.0. I have setup all the configuration as per IBM MFP 8.0 standard. I am trying mobilefirst based sample app which is download from IBM MFP document website. I can able to login and register the users but if I am sending the notifications, I am not able to received.
I have checked logs messages in my development server. its throwing error as below,
[ERROR ] Couldn't send message after 3 retries.Message(Id=2;
Token=D60FFA8C91A465400B203424095CD5530DF2D223EC72BAE386982C0905C1AE1B;
Payload={"payload":"{\"nid\":\"59ef5d3\",\"tag\":\"Push.ALL\"}","aps":{"alert":{"action-loc-key":null,"body":"hi"}}})
Received fatal alert: internal_error [ERROR ] FPWSE1083E: Failure
sending Apple Push Notification Service (APNS) notification with
identifier 2, device token:
D60FFA8C91A465400B203424095CD5530DF2D223EC72BAE386982C0905C1AE1B.
Received fatal alert: internal_error [err] Exception in thread
"pool-9-thread-3" [err] com.notnoop.exceptions.NetworkIOException:
javax.net.ssl.SSLException: Received fatal alert: internal_error [err]
at
com.notnoop.apns.internal.Utilities.wrapAndThrowAsRuntimeException(Utilities.java:277)
[err] at
com.ibm.mfp.push.server.notification.apns.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:319)
[err] at
com.ibm.mfp.push.server.notification.apns.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:292)
[err] at
com.notnoop.apns.internal.ApnsPooledConnection$2.run(ApnsPooledConnection.java:47)
[err] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[err] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[err] at java.lang.Thread.run(Thread.java:748) [err] Caused by:
javax.net.ssl.SSLException: Received fatal alert: internal_error [err]
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) [err] at
sun.security.ssl.Alerts.getSSLException(Alerts.java:154) [err] at
sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
[err] at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
[err] at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
[err] at
sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
[err] at
sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) [err]
at java.io.OutputStream.write(OutputStream.java:75) [err] at
com.ibm.mfp.push.server.notification.apns.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:302)
[err] ... 5 more

Mobile First Platform 8.0 Push notification failed on MacOS High Sierria

I created created Push Notification App(it's cordova app) and MFPPush.registerDevice was succeeded. But "send notification" through MFPConsole(http://localhost:9080/mfpconsole/index.html#/mfp/push/echoview2.MFP8Sample#sendPush) failed.
I think MFP Server may use the IPv6 IP address as its own server IP address.
Anyone can find workaround?
Environment
+ OS: Mac OS High Sierria
+ MFP8: Product version: 8.0.0.00-20180315-134705
[err] com.ibm.mobile.analytics.sdk.events.AnalyticArgumentException: java.lang.IllegalArgumentException: MSAN018E: The supplied value was invalid: fe80:0:0:0:9787:db29:a3d:b9e9%utun0 for serverIpAddress.
[err] at com.ibm.mobile.analytics.sdk.model.PushNotification.setServerIpAddress(PushNotification.java:197)
[err] at com.ibm.mobile.analytics.sdk.events.PushNotification.<init>(PushNotification.java:37)
[err] at com.ibm.mfp.push.server.analytics.plugin.AnalyticsPlugin.sendNotificationDispatchEvent(AnalyticsPlugin.java:172)
[err] at com.ibm.mfp.push.server.notification.Mediator.fireNotificationDispatchEvent(Mediator.java:236)
[err] at com.ibm.mfp.push.server.notification.apns.ApplicationConnection.sendNotification(ApplicationConnection.java:146)
[err] at com.ibm.mfp.push.server.notification.apns.APNSMediator.sendNotification(APNSMediator.java:124)
[err] at com.ibm.mfp.push.server.notification.Mediator$2.run(Mediator.java:105)
[err] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[err] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[err] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[err] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[err] at java.lang.Thread.run(Thread.java:748)
[err] Caused by: java.lang.IllegalArgumentException: MSAN018E: The supplied value was invalid: fe80:0:0:0:9787:db29:a3d:b9e9%utun0 for serverIpAddress.
[err] ... 12 more
[ERROR ] Couldn't send message after 3 retries.Message(Id=2; Token=3BAE6CECD27D2DF48920AF230BE02A7FC3699480529AAAD17D1C27235B6C33C5; Payload={"payload":"{\"nid\":\"46403e8\",\"tag\":\"Push.ALL\"}","aps":{"alert":{"action-loc-key":null,"body":"Foo"}}})
Connection closed by remote host
[ERROR ] FPWSE1083E: Failure sending Apple Push Notification Service (APNS) notification with identifier 2, device token: 3BAE6CECD27D2DF48920AF230BE02A7FC3699480529AAAD17D1C27235B6C33C5.
Connection closed by remote host
[err] Exception in thread "pool-9-thread-3"
[err] com.notnoop.exceptions.NetworkIOException: java.net.SocketException: Connection closed by remote host
[err] at com.notnoop.apns.internal.Utilities.wrapAndThrowAsRuntimeException(Utilities.java:277)
[err] at com.ibm.mfp.push.server.notification.apns.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:319)
[err] at com.ibm.mfp.push.server.notification.apns.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:292)
[err] at com.notnoop.apns.internal.ApnsPooledConnection$2.run(ApnsPooledConnection.java:47)
[err] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[err] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[err] at java.lang.Thread.run(Thread.java:748)
[err] Caused by: java.net.SocketException: Connection closed by remote host
[err] at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1565)
[err] at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:124)
[err] at java.io.OutputStream.write(OutputStream.java:75)
[err] at com.ibm.mfp.push.server.notification.apns.ApnsConnectionImpl.sendMessage(ApnsConnectionImpl.java:302)
[err] ... 5 more
This looks like a configuration error. MFP Push needs to communicate with apple notifications servers (called APNS) in order to successfully send notifications.
It looks like mobilefirst in your case is unable to communicate with the APNS servers. You need to explicitly configure this. Instructions on configuring push notifications proxy for both APNS and GCM (Google cloud messaging for android) can be found here.

hive shell is not working

when i try to access the hive shell, it is showing some error logs. i am using CDH version 5.12.
[cloudera#quickstart ~]$ hive
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked.
WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:810)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1679)
at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1690)
at org.apache.hadoop.hive.ql.processors.CommandUtil.authorizeCommand(CommandUtil.java:55)
at org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:66)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:275)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:318)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:416)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:432)
at org.apache.hadoop.hive.cli.CliDriver.processInitFiles(CliDriver.java:466)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:717)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 45 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:512)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
... 42 more
[cloudera#quickstart ~]$
Try
beeline -u jdbc:hive2://localhost:10000
Please make sure in your hive-site.xml has below configuration,
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/metastore_db?
createDatabaseIfNotExist=true&autoReconnect=true&useSSL=false
</value>
</property

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided

I am getting this error when i try to connect to hive metastore using Spark SQL HiveContext.
i am running this on standalone cluster using spark-submit command from my desktop, not from the hadoop cluster.
Is it something to do with Security related issue? do i have to add something in the hive_site.xml? is there anything we need to update in the below entry?
<property>
<name>hive.metastore.sasl.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.server2.authentication</name>
<value>kerberos</value>
</property>
The spark version is 1.4.0 and the hive-site.xml is placed under conf folder.
below is the error log.
15/08/25 18:27:15 INFO HiveContext: Initializing execution hive, version 0.13.1
15/08/25 18:27:16 INFO metastore: Trying to connect to metastore with URI thrift://metastore.com:9083
15/08/25 18:27:16 ERROR TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Ker
beros tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:336)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:105)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:167)
at com.cap1.ct.SparkSQLHive.main(SparkSQLHive.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 35 more
15/08/25 18:27:16 WARN metastore: Failed to connect to the MetaStore Server...
15/08/25 18:27:16 INFO metastore: Waiting 1 seconds before next connection attempt.
15/08/25 18:27:17 INFO metastore: Trying to connect to metastore with URI thrift://metastore.com:9083
15/08/25 18:27:17 ERROR TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Ker
beros tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:336)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:214)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:105)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:167)
at com.cap1.ct.SparkSQLHive.main(SparkSQLHive.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 35 more
Prerequisite: your hive-site.xml works for hive cli with kerberos enabled.
Spark with hive needs another property:
-Djavax.security.auth.useSubjectCredsOnly=false
Quote from official troubleshooting
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Ticket)
Cause: This may occur if no valid Kerberos credentials are obtained. In particular, this occurs if you want the underlying mechanism to obtain credentials but you forgot to indicate this by setting the javax.security.auth.useSubjectCredsOnly system property value to false (for example via -Djavax.security.auth.useSubjectCredsOnly=false in your execution command).
This issue is inherently tied to krb5.config file which has permissible servers. If it's not found or doesn't have a server domain entry then you may run into this.