DbVisualizer give error while connecting to hive with kerberos Authentication - hive

I am trying to connect to hive using DBVisulizer. And I am following the article mentioned below
https://github.com/timveil/hive-jdbc-uber-jar#kerberized-setup
Following is my config file
[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = dev.abc.com
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  dev.abc.com = {
    admin_server = wplc-dc1.dev.abc.com
    kdc = wplc-dc1.dev.abc.com
  }
I have put the path of this config file in DbVisalizer setting.
And I am using following url
jdbc:hive2://d9lcwphd1m1.dev.abc.com:2181,d9lcwphd1m2.dev.abc.com:2181,d9lcwphd1d1.dev.abc.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST#dev.abc.com
Error:
Java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: GSS initiate failed
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:228)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:166)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.onseven.dbvis.g.B.D.ᅣチ(Z:1548)
at com.onseven.dbvis.g.B.F$A.call(Z:1369)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
I think the problem is that domain of my machine is abc.com and not dev.abc.com
Output of klist
Credentials cache: API:1CFA7C7B-9BF9-41E6-B516-BC10C925E8E7
Principal: gaurang.shah#abc.com
Issued Expires Principal
Jun 8 19:10:28 2018 Jun 9 05:10:28 2018 krbtgt/abc.com#Cabc.com
Update:
if I provide both the realms in the config file I am getting parsing error.
config.
abc.com = {
admin_server = wpcp-dc1.abc.com
kdc = wpcp-dc1.abc.com:88
kdc = wpcp-dc2.abc.com:88
}
dev.abc.com = {
    admin_server = wplc-dc1.dev.abc.com
    kdc = wplc-dc1.dev.abc.com
  }
Error:
Long Message:
Can't get Kerberos realm
Details:
   Type: java.lang.IllegalArgumentException
Stack Trace:
KrbException: Illegal config content:  }
   at java.security.jgss/sun.security.krb5.Config.parseStanzaTable(Unknown Source)
   at java.security.jgss/sun.security.krb5.Config.<init>(Unknown Source)
   at java.security.jgss/sun.security.krb5.Config.getInstance(Unknown Source)

Related

TCP connection refused when hosted in IIS

I have a ASP.NET Core website that listens on 8801 port for normal HTTP request. This program also listens on the 8803 port for MQTT TCP connection.
If I run the progam with Kestrel directly on the server as below:
dotnet MyWebsite.dll # runs the command on the server
I can connect the remote 8803 port with my client successfully. So if I understand it correctly :
The MQTT Server does work fine when using Kestrel.
The 8803 port is not banned by the firewall.
The MQTT TCP Server can accept connection from my client.
However, when hosting within IIS (in-process or out-of-process), the TCP server doesn't work any more:
logs when using MQTT.fx:
2020-08-06 14:14:03,864 INFO --- BrokerConnectorController :
onConnect 2020-08-06 14:14:03,865 INFO --- ScriptsController
: Clear console. 2020-08-06 14:14:03,868 INFO --- MqttFX ClientModel
: MqttClient with ID MQTT_FX_Client assigned. 2020-08-06 14:14:08,100
ERROR --- MqttFX ClientModel : Error when connecting
org.eclipse.paho.client.mqttv3.MqttException: Unable to connect to
server at
org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:94)
~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?] at
org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:701)
~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?] at
java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[?:1.8.0_181] at java.util.concurrent.FutureTask.run(Unknown Source)
~[?:1.8.0_181] at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[?:1.8.0_181] at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[?:1.8.0_181] at
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[?:1.8.0_181] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[?:1.8.0_181] at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: java.net.ConnectException: Connection refused: connect at
java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
~[?:1.8.0_181] at
java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
~[?:1.8.0_181] at java.net.AbstractPlainSocketImpl.doConnect(Unknown
Source) ~[?:1.8.0_181] at
java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
~[?:1.8.0_181] at java.net.AbstractPlainSocketImpl.connect(Unknown
Source) ~[?:1.8.0_181] at java.net.PlainSocketImpl.connect(Unknown
Source) ~[?:1.8.0_181] at java.net.SocksSocketImpl.connect(Unknown
Source) ~[?:1.8.0_181] at java.net.Socket.connect(Unknown Source)
~[?:1.8.0_181] at
org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:84)
~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?] ... 8 more 2020-08-06
14:14:08,102 ERROR --- MqttFX ClientModel : Please verify
your Settings (e.g. Broker Address, Broker Port & Client ID) and the
user credentials! org.eclipse.paho.client.mqttv3.MqttException: Unable
to connect to server at
org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:94)
~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?] at
org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:701)
~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?] at
java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
~[?:1.8.0_181] at java.util.concurrent.FutureTask.run(Unknown Source)
~[?:1.8.0_181] at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[?:1.8.0_181] at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[?:1.8.0_181] at
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[?:1.8.0_181] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[?:1.8.0_181] at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: java.net.ConnectException: Connection refused: connect at
java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
~[?:1.8.0_181] at
java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
~[?:1.8.0_181] at java.net.AbstractPlainSocketImpl.doConnect(Unknown
Source) ~[?:1.8.0_181] at
java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
~[?:1.8.0_181] at java.net.AbstractPlainSocketImpl.connect(Unknown
Source) ~[?:1.8.0_181] at java.net.PlainSocketImpl.connect(Unknown
Source) ~[?:1.8.0_181] at java.net.SocksSocketImpl.connect(Unknown
Source) ~[?:1.8.0_181] at java.net.Socket.connect(Unknown Source)
~[?:1.8.0_181] at
org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:84)
~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?] ... 8 more 2020-08-06
14:14:08,111 INFO --- ScriptsController : Clear console.
2020-08-06 14:14:08,112 ERROR --- BrokerConnectService :
Actually, I cann't connect to server:8803 with normal tcp client.
I'm using MQTTnet(3.0.11) + ASP.NET Core(2.1):
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseKestrel(opts => {
opts.ListenAnyIP(8803, l => l.UseMqtt());
opts.ListenAnyIP(8801);
})
.ConfigureLogging((hostingContext, logging) =>
{
// ...
})
.UseStartup<Startup>();
The HTTP Website works fine (8801).
I can't connect to the MQTT Server (8803) when hosted within IIS.
So my question is : why it happens? How can I host the program with IIS ?
If you need more information, please let me know.
I suggest you could try to use another way to create the MSQT server, and make sure you have open the right port to allow access.
Startup.cs:
var mqttServerOptions = new MqttServerOptionsBuilder() .WithDefaultEndpointPort(1883) .Build(); services .AddHostedMqttServer(mqttServerOptions) .AddMqttConnectionHandler() .AddConnections() .AddMqttTcpServerAdapter();

RedisClusterCommands scriptLoad()

I'm using Lettuce lib to connect to Redis Cluster server and it got error when I try to call RedisClusterCommands.scriptLoad(). It throw exception:
io.lettuce.core.RedisException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7004
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:129) ~[lettuce-core-5.1.6.RELEASE.jar:?]
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123) ~[lettuce-core-5.1.6.RELEASE.jar:?]
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80) ~[lettuce-core-5.1.6.RELEASE.jar:?]
at com.sun.proxy.$Proxy123.scriptLoad(Unknown Source) ~[?:?]
at com.slotty.common.services.redis.RedisManager.loadLuaScriptRaw(RedisManager.kt:106) ~[com.slotty.common-2.0.0.jar:?]
at com.slotty.common.services.redis.RedisManager.loadLuaScript(RedisManager.kt:77) ~[com.slotty.common-2.0.0.jar:?]
at com.slotty.core.server.BaseServer.createServerStore(BaseServer.kt:154) ~[com.slotty.core-2.0.0.jar:?]
at com.slotty.core.server.BaseServer.initStores(BaseServer.kt:150) ~[com.slotty.core-2.0.0.jar:?]
at com.slotty.core.server.BaseZone.initStores(BaseZone.kt:163) ~[com.slotty.core-2.0.0.jar:?]
at com.slotty.core.server.BaseServer.initServices(BaseServer.kt:87) ~[com.slotty.core-2.0.0.jar:?]
at com.slotty.core.server.BaseZone.initServices(BaseZone.kt:49) ~[com.slotty.core-2.0.0.jar:?]
at com.slotty.core.server.BaseServer.initialize(BaseServer.kt:65) ~[com.slotty.core-2.0.0.jar:?]
at com.slotty.core.server.BaseServer.init(BaseServer.kt:55) [com.slotty.core-2.0.0.jar:?]
at com.smartfoxserver.v2.entities.managers.SFSExtensionManager.createExtension(SFSExtensionManager.java:303) [sfs2x.jar:?]
at com.smartfoxserver.v2.entities.managers.SFSZoneManager.createZone(SFSZoneManager.java:426) [sfs2x.jar:?]
at com.smartfoxserver.v2.entities.managers.SFSZoneManager.initializeZones(SFSZoneManager.java:239) [sfs2x.jar:?]
at com.smartfoxserver.v2.SmartFoxServer.start(SmartFoxServer.java:292) [sfs2x.jar:?]
at com.smartfoxserver.v2.Main.main(Main.java:14) [sfs2x.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_102]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_102]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
at com.exe4j.runtime.LauncherEngine.launch(Unknown Source) [i4jruntime.jar:?]
at com.install4j.runtime.launcher.Launcher.main(Unknown Source) [i4jruntime.jar:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7004
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_102]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_102]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:665) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_102]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_102]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:665) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) ~[netty-transport-4.1.33.Final.jar:4.1.33.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
This is my initialize connection code:
I tried to connect to redis server using RedisClusterClient. with IP 10.10.10.1 and port is 7000
val redisUri = with(redisServiceInfo) {
RedisURI.builder()
.withHost(host)
.also { if (password != null) it.withPassword(password) }
.withPort(port)
.withTimeout(Duration.ofMillis(timeout))
.build()
}
val clusterClient = RedisClusterClient.create(redisUri)
val connection = clusterClient.connect()
this.connection = connection
this.clusterClient = clusterClient
Then I do a scriptLoad through RedisClusterCommands
shaResult = redisService.sync().scriptLoad(script)
My Redis Server have 6 nodes: 3 master 3 slave the port from 7000 -> 7002 is master, from 7003 -> 7005 is slave. The error is always happen on slave nodes after testing I found out that the connection is randomly return the URI with different host name, sometime it return redis.domain.local sometime it return correct ip, sometime it return 127.0.0.1
I have tries to replace the IP of the nodes to 10.10.10.1 after connected to server and the function is work well.
Is there any config that I have missed when establish the connection?

How to use MQTT.fx to connect to IOT central?

I am trying to use MQTT directly to connect to IOT central.
I just followed below documentation,
https://learn.microsoft.com/en-us/azure/iot-central/concepts-connectivity#connect-a-single-device
https://www.instructables.com/id/Azure-IoT-Hub-Set-Up-MQTTfx-Sigfox-Callback-and-Dr/
https://github.com/MediaTek-Labs/aws_mbedtls_mqtt/pull/9/files?short_path=04c6e90
https://docs.azure.cn/zh-cn/articles/azure-operations-guide/iot-hub/aog-iot-hub-howto-connect-with-tool-mqtt-fx (Translate to english)
Above documentation I used dps_cstr tool from github to generate connection string with SAS token from this link
https://learn.microsoft.com/en-us/azure/iot-central/tutorial-add-device#prepare-the-client-code
Broker Address = saas-iothub-947867dc-cd5d-446c-90ff-e0f964f020fe.azure-devices.net
Broker port = 8883
Client ID = 92ff3e25-00e5-4249-9074
User Name = saas-iothub-947867dc-cd5d-446c-90ff.azure-devices.net/92ff3e25-00e5-4249-9074
MQTT Version = 3.1.1
Password = OfAlY0BGstmuinZzOcdDDf
I have configured the MQTT.fx with connection profile for iot central by using above connection string and details.
Attached the snap of that
When i try to connect to this, I am getting an error "Not authorized to connect".
Log:
2018-12-20 00:42:49,738 INFO --- BrokerConnectorController : onConnect
2018-12-20 00:42:49,740 INFO --- ScriptsController : Clear console.
2018-12-20 00:42:49,804 INFO --- MqttFX ClientModel : MqttClient with ID 92ff3e25-00e5-4249-9074-854b43b5a949 assigned.
2018-12-20 00:42:53,571 ERROR --- MqttFX ClientModel : Error when connecting
org.eclipse.paho.client.mqttv3.MqttSecurityException: Not authorized to connect
at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:28) ~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?]
at org.eclipse.paho.client.mqttv3.internal.ClientState.notifyReceivedAck(ClientState.java:988) ~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:145) ~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_181]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
2018-12-20 00:42:53,572 ERROR --- MqttFX ClientModel : Please verify your Settings (e.g. Broker Address, Broker Port & Client ID) and the user credentials!
org.eclipse.paho.client.mqttv3.MqttSecurityException: Not authorized to connect
at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:28) ~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?]
at org.eclipse.paho.client.mqttv3.internal.ClientState.notifyReceivedAck(ClientState.java:988) ~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:145) ~[org.eclipse.paho.client.mqttv3-1.2.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_181]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
2018-12-20 00:42:53,574 INFO --- ScriptsController : Clear console.
2018-12-20 00:42:53,574 ERROR --- BrokerConnectService : Not authorized to connect
EDIT:
I created a CA certificate from https://github.com/Azure/azure-iot-sdk-c/blob/master/certs/certs.c as CACert.cer
I selected CS Certificate file option in SSL/TLS option and tried again.
But i have got the same error
I have just finished a test using a MQTT.fx client connected to the Azure IoT Central.
Based on the doc Using the MQTT protocol directly the password must be in the following format, see example:
SharedAccessSignature sr={your hub name}.azure-devices.net%2Fdevices%2FMyDevice01%2Fapi-version%3D2016-11-14&sig=vSgHBMUG.....Ntg%3d&se=1456481802
You have to generate the above password (sas token) string. You can use the following helper function:
string sasToken = SharedAccessSignatureBuilder.GetSASTokenFromConnectionString(connectionString);
public sealed class SharedAccessSignatureBuilder
{
public static string GetHostNameNamespaceFromConnectionString(string connectionString)
{
return GetPartsFromConnectionString(connectionString)["HostName"].Split('.').FirstOrDefault();
}
public static string GetSASTokenFromConnectionString(string connectionString, uint hours = 24)
{
var parts = GetPartsFromConnectionString(connectionString);
return GetSASToken(parts["HostName"], parts["SharedAccessKey"], parts.Keys.Contains("SharedAccessKeyName") ? parts["SharedAccessKeyName"] : null, hours);
}
public static string GetSASToken(string resourceUri, string key, string keyName = null, uint hours = 24)
{
var expiry = GetExpiry(hours);
string stringToSign = System.Web.HttpUtility.UrlEncode(resourceUri) + "\n" + expiry;
HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(key));
var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));
var sasToken = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}", HttpUtility.UrlEncode(resourceUri), HttpUtility.UrlEncode(signature), expiry);
if(!string.IsNullOrEmpty(keyName))
sasToken += String.Format(CultureInfo.InvariantCulture, "&skn={0}", keyName);
return sasToken;
}
#region Helpers
private static Dictionary<string, string> GetPartsFromConnectionString(string connectionString)
{
return connectionString.Split(new[] { ';' }, StringSplitOptions.RemoveEmptyEntries).Select(s => s.Split(new[] { '=' }, 2)).ToDictionary(x => x[0].Trim(), x => x[1].Trim());
}
// default expiring = 24 hours
private static string GetExpiry(uint hours = 24)
{
TimeSpan sinceEpoch = DateTime.UtcNow - new DateTime(1970, 1, 1);
return Convert.ToString((int)sinceEpoch.TotalSeconds + 3600 * hours);
}
#endregion
}
The following screen snippets show the MQTT.fx device (myfirstdevice) connected to the Azure IoT Central and the device Dashboard.

SASL negotiation failure while connecting to Kerberized hive from java

I am have cloudera vm where I enabled Kerberos and I writing a java application on windows machine to get hive connection . but I am getting following exceptions. I followed many examples and documentation to get connection to hive but I am unable to get connection.
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Users/lamadipen/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.4.1/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/Users/lamadipen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Debug is true storeKey false useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is cmf.keytab refreshKrb5Config is false principal is cloudera#CLOUDERA tryFirstPass is false useFirstPass is false storePass is false clearPass is false
principal is cloudera#CLOUDERA
Will use keytab
Commit Succeeded
16:15:35.153 [main] ERROR org.apache.thrift.transport.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_144]
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) [libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [libthrift-0.9.3.jar:0.9.3]
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) [hive-shims-common-2.0.0.jar:2.0.0]
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) [hive-shims-common-2.0.0.jar:2.0.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) [hadoop-common-2.6.0.jar:?]
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) [hive-shims-common-2.0.0.jar:2.0.0]
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:181) [hive-jdbc-2.0.0.jar:2.0.0]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:152) [hive-jdbc-2.0.0.jar:2.0.0]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-2.0.0.jar:2.0.0]
at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_144]
at java.sql.DriverManager.getConnection(DriverManager.java:270) [?:1.8.0_144]
at com.dipen.sch.HiveConnection.run(App.java:146) [classes/:?]
at com.dipen.sch.HiveConnection.run(App.java:1) [classes/:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) [hadoop-common-2.6.0.jar:?]
at com.dipen.sch.App.authentication(App.java:96) [classes/:?]
at com.dipen.sch.App.main(App.java:50) [classes/:?]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770) ~[?:1.8.0_144]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_144]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_144]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_144]
... 21 more
Caused by: sun.security.krb5.KrbException: Clock skew too great (37) - PROCESS_TGS
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.8.0_144]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) ~[?:1.8.0_144]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) ~[?:1.8.0_144]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308) ~[?:1.8.0_144]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126) ~[?:1.8.0_144]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) ~[?:1.8.0_144]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) ~[?:1.8.0_144]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_144]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_144]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_144]
... 21 more
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140) ~[?:1.8.0_144]
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65) ~[?:1.8.0_144]
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60) ~[?:1.8.0_144]
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.8.0_144]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) ~[?:1.8.0_144]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) ~[?:1.8.0_144]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308) ~[?:1.8.0_144]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126) ~[?:1.8.0_144]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) ~[?:1.8.0_144]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) ~[?:1.8.0_144]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_144]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_144]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_144]
... 21 more
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1643)
at com.dipen.sch.App.authentication(App.java:96)
at com.dipen.sch.App.main(App.java:50)
Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://quickstart.cloudera:10000/;principal=hive/cloudera#CLOUDERA: GSS initiate failed
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:207)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:152)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at com.dipen.sch.HiveConnection.run(App.java:146)
at com.dipen.sch.HiveConnection.run(App.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
... 2 more
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:181)
... 11 more
I am getting Commit Succeeded message on my console does it mean I am authenticated.
public static void authentication() throws LoginException, IOException, InterruptedException, PrivilegedActionException
{
System.setProperty("hadoop.home.dir", "C:\\hadoop-common-2.2.0");
System.setProperty("java.security.auth.login.config", "gss-jaas.conf");
System.setProperty("java.security.krb5.realm","CLOUDERA");
System.setProperty("java.security.krb5.kdc","169.254.56.203");
System.setProperty("java.security.krb5.conf", "krb5.conf");
System.setProperty("javax.security.auth.useSubjectCredOnly","false");
LoginContext loginContext = new LoginContext("com.sun.security.jgss.initiate");
loginContext.login();
Subject subject = loginContext.getSubject();
Configuration conf = new Configuration();
conf.set("hadoop.security.authentication", "Kerberos");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation ugi = UserGroupInformation.getUGIFromSubject(subject);
HiveConnection hc = new HiveConnection();
ugi.doAs(hc);
//Subject.doAs(subject, hc);
System.out.println("Before Getting connection");
Connection con = hc.con;
System.out.println("After Getting connection");
}
I am trying to use UserGroupInformation to call PrivilegedExceptionAction and get the connection and I have tired same with Subject also but I am getting same issue which ever way I go.
class HiveConnection implements PrivilegedExceptionAction<Void>{
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
Connection con=null;
public Void run() throws ClassNotFoundException, SQLException, IOException {
Class.forName(driverName);
con = DriverManager.getConnection("jdbc:hive2://quickstart.cloudera:10000/;principal=hive/cloudera#CLOUDERA");
return null;
}
}
krb5.conf file
[libdefaults]
default_realm = CLOUDERA
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
CLOUDERA = {
kdc = 169.254.56.203
admin_server = 169.254.56.203
}
[domain_realm]
gss-jaas.conf file
com.sun.security.jgss.initiate {
com.sun.security.auth.module.Krb5LoginModule required
principal="cloudera#CLOUDERA"
useKeyTab= true
keyTab="cmf.keytab"
storeKey=false
doNotPrompt=false
renewTGT=false
useTicketCache=false
debug=true;
};

401 Unauthorized after NTLM authentication error: Unexpected state: MSG_TYPE1_GENERATED

I would like to know what the error messages MSG_TYPE1_GENERATED and MSG_TYPE3_GENERATED (from httpClient/NTLMScheme.State enum) signify to help debug an issue we are having. Can someone please provide the documentation for these errors?
I have a server application which uses ews-java-api 2.0 for connecting to an Exchange Server 2016, configured with NTLM Authentication.
 
I have configured HttpClient version 4.5.2.
 
The server application uses an Exchange Service Account for retrieving rooms and meetings from the Exchange Server.
 
The rooms and meetings are successfully retrieved, but, after a while of successfully running, I see the following error in logs:
 
ERROR org.apache.http.impl.auth.HttpAuthenticator - NTLM authentication error: Unexpected state: MSG_TYPE3_GENERATED
 
This error is followed by some 401 Unauthorized errors received from Exchange Server and the application never recovers from this 401 error until restart (this usually happens a few times a day).
Caused by: microsoft.exchange.webservices.data.core.exception.http.HttpErrorException: The remote server returned an error: (401)Unauthorized
                at microsoft.exchange.webservices.data.core.request.ServiceRequestBase.getEwsHttpWebResponse(ServiceRequestBase.java:726)
                at microsoft.exchange.webservices.data.core.request.ServiceRequestBase.validateAndEmitRequest(ServiceRequestBase.java:640)
                ... 34 more
 
I am not sure if it is related to this issue, but sometimes after the MSG_TYPE3_GENERATED error log, I see a lot of timeouts:
 
Caused by: java.net.SocketTimeoutException: Read timed out
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
                at java.net.SocketInputStream.read(SocketInputStream.java:170)
                at java.net.SocketInputStream.read(SocketInputStream.java:141)
                at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
                at sun.security.ssl.InputRecord.read(InputRecord.java:503)
                at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
                at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
                at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
                at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
                at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
                at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
                at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
                at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
                at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
                at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
                at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
                at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
                at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
                at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
                at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
                at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
                at microsoft.exchange.webservices.data.core.request.HttpClientWebRequest.executeRequest(HttpClientWebRequest.java:286)
                at microsoft.exchange.webservices.data.core.request.ServiceRequestBase.getEwsHttpWebResponse(ServiceRequestBase.java:721)
                ... 39 more
Sometimes I see MSG_TYPE1_GENERATED in error log instead of MSG_TYPE3_GENERATED.
 
I looked into the HttpClient library NTLMScheme.java source code and found the 
    
enum State {
        UNINITIATED,
        CHALLENGE_RECEIVED,
        MSG_TYPE1_GENERATED,
        MSG_TYPE2_RECEVIED,
        MSG_TYPE3_GENERATED,
        FAILED,
    }
but there does not seem to be any documentation regarding each value.
 
I tried to increase the version of HttpClient to 4.5.3 because I saw an issue in release notes which is related to NTLM
(https://issues.apache.org/jira/browse/HTTPCLIENT-1779). After that my application seems to be more stable. I saw the MSG_TYPE3_GENERATED only once and the application has been running for several days.
Do you think upgrading the HttpClient from 4.5.2 to 4.5.3 did the trick?
 
Do you have any other suggestions?
Thanks,
Alin