on EMR 4.2 - Hive 1.0 version, I want to connect to a remote mysql metastore.
<property>
<name>hive.metastore.uris</name>
<value>thrift://hive-metastore-remotemysql.aws.com:9083</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
This remorte metastore is on Hive .12 and I still want to connect to same metastore from a new cluster. Because of the new hive-site.xml format I cannot give proper value to metastore-uris. As 9083 deoes not exist on remote. Id I give local then hive does not know about all databases.
Anyone has faced this and solved?
Thanks!
Related
If I enter the remote machine dlw2nia-bd01, and I execute beeline and I execute this connecting string
!connect jdbc:hive2://dlw2nia-bd02.walgreens.com:2181,dlw2nia-bd03.walgreens.com:2181,dlw2nia-bd10.walgreens.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
, I can connect to Hive, in fact I see:
Connected to: Apache Hive (version 1.2.1000.2.6.4.0-91)
Driver: Hive JDBC (version 1.2.1000.2.6.4.0-91)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://dlw2nia-bd02.walgreens.com:21>
I would like to use, however, a client to better show the Hive tables.
I installed JDBC Driver Clouder and I tried to configure Squirrel to connect to Hive but I'm having an issue connection. Another SQL Client such as DBEaver or SQlDeveloper would also be fine.
I was thinking of downloading the drivers JDBC used for Beeline and using them with an SQL client, but I don't know where to find drivers in the remote machine.
Can you help me configure a client to connect to Hive?
I have configured an ODBC connection in Windows ODBC Data Source Administrator and test successfully.
I want to connect to HIVE database (the ODBC connection) using Spark. As my understanding, the connection should be configured in hive-site.xml and put to folder /config. But how to configure hive-site.xml using ODBC connection? Can anyone provide an example for hive-site.xml?
I am using cloudera ODBC driver.
Environment： Windows, spark 2.3.3
I am new to hive, and some question confusing me very much.
first, after installation of hive, I just run hive, then I can create, select tables. where is the hive server, what is the use of it.
second, what is the use of metastore server, I know we need the metastore to access the metadata about hive tables, does that mean if I start a metastore server I can request it in other app and get the information?
Metastore server talks to the backend such as Derby/MySql to store and retrieve table metadata. If any Hive component wants to get/set metadata, it calls the MetaStore APIs. APIs are such getTable(tableName), createDatabase(dbName) etc. Basically metastore abstracts and provides backend (derby/mysql/postgres) independent API layer. Similar to HiveServer this can also run as a server. If there is no metastore server running, then the Driver will load the metastore in its process. If metastore is running as a separate server then the Driver object communicates with the metastore over network.
I have installed Hive in my local windows system, I have configured it to use MySQL as metastore and I need to configure it to use Azure Data lake as warehouse.
How to configure hive to use Azure Datalake as warehouse?.
This is possible, but you have to configure your local hadoop system accordingly.
Ensure that you have the latest ADLS libs and modify the cores-site.xml to access the Azure Data Lake Store:
<configuration>
<property>
<name>dfs.adls.oauth2.access.token.provider.type</name>
<value>ClientCredential</value>
</property>
<property>
<name>dfs.adls.oauth2.refresh.url</name>
<value>YOUR TOKEN ENDPOINT</value>
</property>
<property>
<name>dfs.adls.oauth2.client.id</name>
<value>YOUR CLIENT ID</value>
</property>
<property>
<name>dfs.adls.oauth2.credential</name>
<value>YOUR CLIENT SECRET</value>
</property>
<property>
<name>fs.adl.impl</name>
<value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
</property>
<property>
<name>fs.AbstractFileSystem.adl.impl</name>
<value>org.apache.hadoop.fs.adl.Adl</value>
</property>
</configuration>
A step by step guide can be found here.
Finally ensure that in the hive-site.xml your "hive.metastore.warehouse.dir" points to the ADL.
This is not a supported use case of Azure Data Lake. Azure Data Lake is a cloud based data lake and currently supports HDInsight, Azure Data Lake Analytics, or Azure Databricks as its compute engines. It cannot connect with a locally run instance of Hive.
How to connect Hive database in Ignite framework?
Can we use already existing hive database and hive tables over Ignite?
There are two options I think:
Load all data from Hive database and recreate the same schema in Ignite and stream all data to it with DataStreamer
Connect to Hive Database by using CacheJdbcPojoStore and Hive JDBC Driver.