I'm not getting any error in any step but after creation of the tables in Hive ,when i'm trying to querying in mysql (show tables), its showing empty result(Empty set (0.00 sec)).
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/metastore_db?createDatabaseIfNotExist=true</value>
<description>metadata is stored in a MySQL server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>MySQL JDBC driver class</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hiveuser</value>
<description>user name for connecting to mysql server </description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hivepassword</value>
<description>password for connecting to mysql server </description>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9083</value>
</property>
First run
mysql -u hiveuser
show databases
use metastore_db
show tables
Are you able to create tables in hive?
Do you find logs like this?
[root#sandbox ~]# cd /var/log/hive
[root#sandbox hive]# ls -ltr *.log
-rw-r--r-- 1 hive hadoop 1255 2015-11-02 23:54 hive.log
-rw-r--r-- 1 hive hadoop 1255 2015-11-02 23:54 hive-server2.log
-rw-r--r-- 1 hive hadoop 179943 2015-12-08 10:28 hivemetastore.log
-rw-r--r-- 1 hive hadoop 1738852 2015-12-08 10:34 hiveserver2.log
[root#sandbox hive]#
Related
Im getting error while connecting presto server to hive metastore.
Here is my hive.properties:
connector.name=hive-hadoop2
hive.metastore.uri=thrift://localhost:9083
hive.config.resources=/home/eweb/Downloads/hadoop/etc/hadoop/core-site.xml,/home/eweb/Downloads/hadoop/etc/hadoop/hdfs-site.xml
And here is my hive-site.xml :
<property>
<name>hive.metastore.uri</name>
<value>thrift://localhost:9083</value>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/Demo?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>
I have tried this in terminal :
bin/presto --server localhost:8080 --catalog hive --schema bookvip
presto:bookvip> SHOW SCHEMAS;
Is there any kind of setup error or any other error ??
I just use the following commands step by step:
sudo netstat -tulpn | grep 9083
Then kill the process using this command
sudo kill -s 9 processid
And run once again
hive --service metastore
In my case, I missed to run the hive metastore service and so the error "Failed to connect to the MetaStore Server..."
To resolve, go to /usr/local/Cellar/hive/3.1.2_3/bin and run:
hive --service metastore
When I run:
hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.root.logger=INFO,console
It shows
Starting HiveServer2
and nothing listens on port 10000 and 10001
The HiveServer2 service does not output error information, causing it hard to diagnostic the problem. You can try to start the metastore service provided by Hive, which listens on port 9083 and might give some information when your configuration is not properly set:
hive --service metastore # not detach from terminal to see logs
In my case, this service cannot be started, with error message:
MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema
version 1.2.0 Metastoed or corrupt)
One of the direct solution to resolve this error is to ignore the version difference by setting the hive-site.xml if there is only one hive version in your machine (another solution is to modify the metastore_db version):
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
After this problem is resolved, the HiveServer2 service can be running and listening on port 10000.
hive --service hiveserver2 > /dev/null 2>&1 &
If your HiveServer2 access metastore via Derby or MySQL JDBC driver, then the aforementioned metastore service is not needed for HiveServer2. However, if HiveServer2 access metastore via thrift protocol, as configed in conf/hive-site.xml like
<property>
<name>hive.metastore.uris</name>
<value>thrift://hadoop-master:9083</value>
<description>
Thrift URI for the remote metastore.
Used by metastore client to connect to remote metastore.
</description>
</property>
Then, the metastore service must be started at first.
I had a hard time to set up hive-3.1.2. I write this maybe it helps someone out. in order to diagnose the problem first try to launch metastore and hiveserver2 like this:
metastore:
hive --service metastore --hiveconf hive.root.logger=INFO,console
hiveserver2:
hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.root.logger=INFO,console
then carefully read the the exceptions were thrown.
my problem was user hive is not allowed to perform this api call
and to solve that I added the following property to hive-site.xml:
<property>
<name>hive.metastore.event.db.notification.api.auth</name>
<value>false</value>
<description>
Should metastore do authorization against database notification related APIs such as get_next_notification.
If set to true, then only the superusers in proxy settings have the permission
</description>
</property>
also I add my full hive-site.xml as a sample:
<configuration>
<property>
<name>datanucleus.schema.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://server-2:3306/metastore?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>mysql_username</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>mysql_password</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://server-2:9083</value>
</property>
<property>
<name>atanucleus.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>server-2</value>
</property>
<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.event.db.notification.api.auth</name>
<value>false</value>
</property>
</configuration>
Thanks. There is typo. It should hive.metastore not as shown below.
**metastore**.metastore.event.db.notification.api.auth
false
My task is to create an oozie workflow to Load Data to Hive tables every hour.
I am using CDH 5.7 in virtualbox
When i run the hive script which contains LOAD DATA INPATH '/sqoop_import_increment' INTO TABLE customer; it works perfectly, data gets loaded to the hive table.
But When i run the same script on oozie workflow the job get killed at 66% and the error message is Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [10001]
Note: but hive script for create table works perfectly with oozie workflow.
plz help.
hive script:
use test;
create external table if not exists customer(customer_id int,name string,address string)row format delimited fields terminated by ',';
load data inpath /sqoop_import_increment into table customer;
workflow.xml:
<workflow-app name="hive_script" xmlns="uri:oozie:workflow:0.5">
<start to="hive-4327"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="hive-4327" cred="hcat">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>lib/hive-config.xml</job-xml>
<script>lib/impala-script.hql</script>
</hive>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
job.properties:
oozie.use.system.libpath=True
security_enabled=False
dryrun=False
jobTracker=localhost:8032
nameNode=hdfs://quickstart.cloudera:8020
hive-config.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://127.0.0.1/metastore?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>cloudera</value>
</property>
<property>
<name>hive.hwi.war.file</name>
<value>/usr/lib/hive/lib/hive-hwi-0.8.1-cdh4.0.0.jar</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://127.0.0.1:9083</value>
<description>IP address (or fully-qualified domain name) and port of the metastore host</description>
</property>
</configuration>
The last time I ran into this problem, it turned out that the hive client was not installed on all data nodes.
When you run the hive query manually, you presumably do it from a node that has the hive client installed.But when oozie is asked to run the query, it will do so from a random data node. As such you will need to setup the hive client on all data nodes.
This assumes that you are not able to let oozie run hive queries in general (and don't have any specific issues with this particular command).
I'm launching a Spark hive-server cluster that use S3 as a warehouse. I've redundantly set up my AWS credential using 3 methods, namely:
hdfs-site.xml under $SPARK_HOME/conf:
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>****</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>****</value>
</property>
setting executor's system property by using spark.executor.extraJavaOptions in start-hivethrift parameter:
--conf "spark.executor.extraJavaOptions=-Dfs.s3.awsAccessKeyId=**** -Dfs.s3.awsSecretAccessKey=****" \
setting environment variables before start-hivethrift.
The launching script looks like this:
AWS_ACCESS_KEY_ID=**** \
AWS_SECRET_ACCESS_KEY=**** \
$SPARK_HOME/sbin/start-thriftserver.sh \
--conf "spark.executor.extraJavaOptions=-Dfs.s3.awsAccessKeyId=**** -Dfs.s3.awsSecretAccessKey=****" \
--hiveconf hive.metastore.warehouse.dir=s3n://testdata \
but when I run any create table query I still get:
Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).) (state=,code=0)
What is going on here? Why none of them works as in documentation?
Oops, there is a problem in my hdfs-site.xml. I should add all possible schema names supported by S3:
<configuration>
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>****</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>****</value>
</property>
<property>
<name>fs.s3n.awsAccessKeyId</name>
<value>****</value>
</property>
<property>
<name>fs.s3n.awsSecretAccessKey</name>
<value>****</value>
</property>
<property>
<name>fs.s3a.awsAccessKeyId</name>
<value>****</value>
</property>
<property>
<name>fs.s3a.awsSecretAccessKey</name>
<value>****</value>
</property>
</configuration>
Seems no more problem now. Its kind of inconvenient but I'm glad it works now.
I have a windows machine and in it I have configured a VMwire with ubuntu.
I installed hadoop and HIVE in the ubuntu.
I have mysql database in my windows
Now I want my metastore to be created in my windows mysql.
I am using the below hive-site.xml. I am getting hive prompt also, but the metastore is not being created in my windows mysql database?? Any body has any idea about this. Please help. Thanks !!!
hive-site.xml
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
<description>Driver class name for a JDBC metastore</description>
</property>
</configuration>