How to setup SSL in Zeppelin - ssl

I'm trying to setup SSL in Zeppelin and after following the instructions and all related Google searches, the zeppelin service status says it's ok but the web response is ERR_CONNECTION_REFUSED
These are the parts of the zeppelin-site.xml I modified
<property>
<name>zeppelin.ssl</name>
<value>true</value>
<description>Should SSL be used by the servers?</description>
</property>
<property>
<name>zeppelin.ssl.client.auth</name>
<value>false</value>
<description>Should client authentication be used for SSL connections?</description>
</property>
<property>
<name>zeppelin.ssl.keystore.path</name>
<value>/opt/zeppelin/lib/keystore</value>
<description>Path to keystore relative to Zeppelin configuration directory</description>
</property>
<property>
<name>zeppelin.ssl.keystore.type</name>
<value>JKS</value>
<description>The format of the given keystore (e.g. JKS or PKCS12)</description>
</property>
<property>
<name>zeppelin.ssl.keystore.password</name>
<value><super secret password></value>
<description>Keystore password. Can be obfuscated by the Jetty Password tool</description>
</property>
<property>
<name>zeppelin.ssl.truststore.path</name>
<value>/opt/zeppelin/lib/keystore</value>
<description>Path to truststore relative to Zeppelin configuration directory. Defaults to the keystore path</description>
</property>
<property>
<name>zeppelin.ssl.truststore.type</name>
<value>JKS</value>
<description>The format of the given truststore (e.g. JKS or PKCS12). Defaults to the same type as the keystore type</description>
</property>
Then this is the service status, which says I'm running but in actuality the web page returns an error:
● zeppelin.service - Zeppelin service
Loaded: loaded (/etc/systemd/system/zeppelin.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-05-09 18:26:41 UTC; 28s ago
Process: 347 ExecStop=/opt/zeppelin/bin/zeppelin-daemon.sh stop (code=exited, status=0/SUCCESS)
Process: 413 ExecStart=/opt/zeppelin/bin/zeppelin-daemon.sh start (code=exited, status=0/SUCCESS)
Main PID: 441 (java)
Tasks: 39 (limit: 4662)
CGroup: /system.slice/zeppelin.service
└─441 java -Dfile.encoding=UTF-8 -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -Dlog4j.configuration=file:///opt/zeppelin/conf/log4j.properties -Dzeppelin.log.file=/opt/zeppelin/logs/zeppelin-zeppelin-myserver.log -cp ::/op

You should set zeppelin.ssl property to true :
<property>
<name>zeppelin.ssl</name>
<value>true</value>
<description>Should SSL be used by the servers?</description>
</property>

Related

ApacheIgnite TcpDiscoveryKubernetesIpFinder fails in Azure Kubernetes cluster in a vNet

vNet IP Address space: 10.106.8.0/22
Apache Ignite version: 2.9.1
Kubernetes version: 1.19.7
Service CIDR: 10.0.0.0/16
DNS Service IP: 10.0.0.10
Docker bridge CIDS: 172.17.0.1/16
We deployed a AKS cluster in a vNet after that deployed Apache Ignite cluster 2.9.1. The sqlline.sh and thin client (dotnet) was able to connect using port 10800 and internal load balancer IP Address. But, the server node (clientmode=true, dotnet thick client) wasn't able to connect (xml config file attached) below is the error.
Any help to resolve the issue is much appreciated.
[Error] [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] Failed to get registered addresses from IP finder on start
After the above, below error repeats forever...
[06:04:20] [Error] [org.apache.ignite.internal.util.typedef.G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=tcp-client-disco-msg-worker, threadName=tcp-client-disco-msg-worker-#4-#35, blockedFor=13s]
[06:04:20] [Warn] [] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=Unmodifia[72901-default-config.xml][1]bleSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-client-disco-msg-worker, igniteInstanceName=null, finished=false, heartbeatTs=1614578647003]]]
[06:04:20] [Warn] [org.apache.ignite.internal.processors.cache.CacheDiagnosticManager] Page locks dump:
Apache Ignite server nodes were deployed in cohort-store k8s namespace...
kubectl get pods -n cohort-store
NAME READY STATUS RESTARTS AGE
cohortstore-0 1/1 Running 0 3d6h
cohortstore-1 1/1 Running 0 3d6h
cohortstore-2 1/1 Running 0 3d6h
kubectl -n cohort-store get svc
kubectl -n cohort-store get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cohortstore-load-balancer-internal LoadBalancer 10.0.113.146 10.106.8.255 8080:31417/TCP,10800:32719/TCP,10900:31208/TCP 29h
Apache Ignite client node (dotnet think client) was deployed in cohort-frontdoor k8s namespace...
kubectl get pods -n cohort-frontdoor
NAME READY STATUS RESTARTS AGE
cohortfrontdoor-665f99bb6b-tdl5z 1/1 Running 0 72m
Client XML SpringConfig file
<?xml version="1.0" encoding="UTF-8"?>
<!--
Configuration example with Kubernetes IP finder and Ignite persistence enabled.
WAL files and database files are stored in separate disk drives.
-->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="clientmode" value="true"/>
<property name="failureDetectionTimeout" value="5000"/>
<property name="clientFailureDetectionTimeout" value="10000"/>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="ephi"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="backups" value="0"/>
</bean>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="networkTimeout" value="10000" />
<property name="localPort" value="47500" />
<property name="ipFinder">
<!--
Enables Kubernetes IP finder and setting custom namespace and service names.
-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
<property name="namespace" value="cohort-store"/>
<property name="serviceName" value="cohortstore-load-balancer-internal"/>
</bean>
</property>
<property name="socketTimeout" value="300" />
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="slowClientQueueLimit" value="1000"/>
</bean>
</property>
</bean>
</beans>
The error went away after I did clusterRoleBinding (see below for more info). I thought, had to do this, since my Ignite server nodes and the think client nodes are in two different K8S namespaces & had two different service accounts.
But now, below is what I'm see in a big loop on the server nodes...
INFO: TCP discovery accepted incoming connection [rmtAddr=/10.106.8.32, rmtPort=43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: TCP discovery spawning a new thread for connection [rmtAddr=/10.106.8.32, rmtPort=43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started serving remote node connection [rmtAddr=/10.106.8.32:43883, rmtPort=43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Initialized connection with remote client node [nodeId=59a5ce6f-2d0d-4abb-aaf5-b2b9f51f7e44, rmtAddr=/10.106.8.32:43883]
Mar 02, 2021 12:02:44 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Finished serving remote node connection [rmtAddr=/10.106.8.32:43883, rmtPort=43883
Mar 02, 2021 12:02:46 AM org.apache.ignite.logger.java.JavaLogger info
clusterrolebinding info
k describe clusterrolebinding cohortstore-RoleBinding
Name: cohortstore-RoleBinding
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: cohortstore
meta.helm.sh/release-namespace: cohort-store
Role:
Kind: ClusterRole
Name: cohortstore-Role
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount cohortstore.com cohort-store
ServiceAccount cohort-frontdoor.com cohort-frontdoor

HiveServer2 'Failed to connect to localhost:10000 Unknown HS2 problem when communicating with Thrift server. '

beeline -u "jdbc:hive2://namenode:10000/default"
Connecting to jdbc:hive2://namenode:10000/default
19/05/11 17:21:52 [main]: WARN jdbc.HiveConnection: Failed to connect to namenode:10000
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://namenode:10000/default: java.net.SocketException: Connection reset (state=08S01,code=0)
Beeline version 3.1.0 by Apache Hive
hive-site.xml
javax.jdo.option.ConnectionURL
jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true
<property>
<name>metastore.warehouse.dir</name>
<value>hdfs://namenode:9820/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value>
</property>
<property>
<name>metastore.schema.verification</name>
<value>true</value>
</property>
<property>
<name>metastore.hmshandler.retry.attemp</name>
<value>10</value>
<description>The number of times to retry a HMSHandler call if there were a connection error.</description>
</property>
<property>
<name>metastore.thrift.uris</name>
<value>thrift://namenode:9083</value>
</property>
<property>
<name>metastore.thrift.port</name>
<value>9083</value>
</property>
<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
<description>The server transport mode. The value can be binary or http. Set to http to enable HTTP transport mode.
</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
<description>TCP port number to listen on</description>
</property>
<property>
<name>hiver.server2.thrift.bind.host</name>
<value>namenode</value>
<description>TCP interface to bind to</description>
</property>
<property>
<name>hive.server2.thrift.http.port</name>
<value>100002</value>
<description>HTTP port number to listen on</description>
</property>
<property>
<name>hive.server2.thrift.http.min.worker.threads</name>
<value>5</value>
<description>Maximum worker threads in the server pool</description>
</property>
<property>
<name>hive.server2.thrift.http.max.worker.threads</name>
<value>500</value>
<description>Maximum worker threads in the server pool</description>
</property>
<property>
<name>hive.server2.thrift.min.worker.threads</name>
<value>5</value>
<description>Minimum number of worker threads</description>
</property>
<property>
<name>hive.server2.thrift.max.worker.threads</name>
<value>500</value>
<description>Maximum number of worker threads</description>
</property>
<property>
<name>hive.server2.authentication</name>
<value>NOSASL</value>
<description>
Expects one of [nosasl, none, ldap, kerberos, pam, custom].
Client authentication types.
NONE: no authentication check
LDAP: LDAP/AD based authentication
KERBEROS: Kerberos/GSSAPI authentication
CUSTOM: Custom authentication provider
(Use with property hive.server2.custom.authentication.class)
PAM: Pluggable authentication module
NOSASL: Raw transport
</description>
</property>
<property>
<name>hive.server2.thrift.http.path</name>
<value>cliservice</value>
<description>The path component of the URL endpoint when in HTTP mode</description>
</property>
<property>
<name>hive.metastore.event.db.notification.api.auth</name>
<value>false</value>
<description>
Should metastore do authorization against database notification related APIs such as get_next_notification.
If set to true, then only the superusers in proxy settings have the permission
</description>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://namenode:9820/tmp/hive</value>
<description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission.</description>
</property>
<property>
<name>hive.exec.local.scratchdir</name>
<value>/tmp/${user.name}</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/tmp/${user.name}_resources</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>hive.scratch.dir.permission</name>
<value>733</value>
<description>The permission for the user specific scratch directories that get created.</description>
</property>
Hello all,
hive server is listening on port 10000(tcp) and 10002(http).
If I hit, beeline -u jdbc:hive2://
It works, but when I try to access from host-ip or hostname,
It shows above errors. Anyone have idea?

Not able to connect Hive from Hue

I have a cloudera setup and in it I have enabled the SSL for hive and also enabled hue TLS/SSL Communication with HiveServer2 in hue.ini conf file.
After the changes has been done, trying to connect to Hive from hue then, the following error is being displayed
"Certificate error with remote host: hostname 'quickstart.cloudera' doesn't match u'cm01.example.com'".
Hive Configuration:
<property>
<name>hive.server2.use.SSL</name>
<value>true</value>
<description>enable/disable SSL </description>
</property>
<property>
<name>hive.server2.keystore.path</name>
<value>/opt/cloudera/security/jks/cmhost-keystore.jks</value>
<description>keystore-file-path</description>
</property>
<property>
<name>hive.server2.keystore.password</name>
<value>xyz</value>
<description>keystore password</description>
</property>
Hue Configuration:
# SSL communication enabled for this server.
enabled=true
# Path to Certificate Authority certificates.
cacerts=/etc/hue/cmhost.pem
# Path to the private key file.
## key=/etc/hue/key.pem
# Path to the public certificate file.
## cert=/etc/hue/cert.pem
# Choose whether Hue should validate certificates received from the server.
validate=true

'hiveserver2 not listening on port 10000 and 10001'

When I run:
hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.root.logger=INFO,console
It shows
Starting HiveServer2
and nothing listens on port 10000 and 10001
The HiveServer2 service does not output error information, causing it hard to diagnostic the problem. You can try to start the metastore service provided by Hive, which listens on port 9083 and might give some information when your configuration is not properly set:
hive --service metastore # not detach from terminal to see logs
In my case, this service cannot be started, with error message:
MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema
version 1.2.0 Metastoed or corrupt)
One of the direct solution to resolve this error is to ignore the version difference by setting the hive-site.xml if there is only one hive version in your machine (another solution is to modify the metastore_db version):
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
After this problem is resolved, the HiveServer2 service can be running and listening on port 10000.
hive --service hiveserver2 > /dev/null 2>&1 &
If your HiveServer2 access metastore via Derby or MySQL JDBC driver, then the aforementioned metastore service is not needed for HiveServer2. However, if HiveServer2 access metastore via thrift protocol, as configed in conf/hive-site.xml like
<property>
<name>hive.metastore.uris</name>
<value>thrift://hadoop-master:9083</value>
<description>
Thrift URI for the remote metastore.
Used by metastore client to connect to remote metastore.
</description>
</property>
Then, the metastore service must be started at first.
I had a hard time to set up hive-3.1.2. I write this maybe it helps someone out. in order to diagnose the problem first try to launch metastore and hiveserver2 like this:
metastore:
hive --service metastore --hiveconf hive.root.logger=INFO,console
hiveserver2:
hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.root.logger=INFO,console
then carefully read the the exceptions were thrown.
my problem was user hive is not allowed to perform this api call
and to solve that I added the following property to hive-site.xml:
<property>
<name>hive.metastore.event.db.notification.api.auth</name>
<value>false</value>
<description>
Should metastore do authorization against database notification related APIs such as get_next_notification.
If set to true, then only the superusers in proxy settings have the permission
</description>
</property>
also I add my full hive-site.xml as a sample:
<configuration>
<property>
<name>datanucleus.schema.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://server-2:3306/metastore?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>mysql_username</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>mysql_password</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://server-2:9083</value>
</property>
<property>
<name>atanucleus.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>server-2</value>
</property>
<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.event.db.notification.api.auth</name>
<value>false</value>
</property>
</configuration>
Thanks. There is typo. It should hive.metastore not as shown below.
**metastore**.metastore.event.db.notification.api.auth
false

HRegionServer shows "error telling master we are up". Showing socket exception: Invalid argument

Iam trying to create a hbase cluster in 3 centos machines. Hadoop(v - 2.8.0) is up and running on top I configured HBase(v - 1.2.5).Hbase start up is fine it started HMaster and Region servers but still it shows the follwing error in region servers and in HMaster log it shows no region servers are checked in.
2017-04-20 19:30:33,950 WARN [regionserver/localhost/127.0.0.1:16020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.net.SocketException: Invalid argument
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:240)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2316)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:907)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
JPS of my master node
[hadoop#localhost bin]$ jps
20624 SecondaryNameNode
20800 ResourceManager
20401 NameNode
18061 Jps
17839 HMaster
JPS of myregion nodes are
[hadoop#localhost bin]$ jps
11168 Jps
482 DataNode
10840 HQuorumPeer
10974 HRegionServer
hbase-site.xml of all nodes
<configuration>
<property>
<name>hbase.master.hostname</name>
<value>NameNode</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://NameNode:8020/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>hdfs://NameNode:8020/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>DataNode1,DataNode2</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
</configuration>
regionservers file contain
DataNode1
DataNode2
etc/hosts file in all nodes contain actual ips rather than loopback ips
192.168.00.00 NameNode
192.168.00.00 DataNode1
192.168.00.00 DataNode2
Note configuration is same in all nodes. Any help will be appreciated.
I put the following property in all region servers hbase-site.xml solved my problem.<property> <name>hbase.regionserver.hostname</name> <value>DataNode1</value> </property> <property> <name>hbase.regionserver.port</name> <value>16020</value> </property>
i was facing the same problem but...
changing hostname resolved my problem
sudo hostnamectl set-hostname new_hostname
i had a master and a node called node1
link to wiki that have the configs