i just installed hive and mysql..
and copied the mysqlconnector to the hive_home/lib folder
but when i try show databases and create table commands in the hive> prompt giving me the error as below:
create database saty;
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
and my hive_site.xml is
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hadoop?CreateDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
and i dont have a directory called /user/hive/warehouse in my file system.
i created these path with mkdir command.. and tried after reboot..
bout still getting the error..
regards,
satya
Try Specifying these two properties
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>username</value>
<description>Username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
<description>Password to use against metastore database</description>
</property>
And Similarly the username of your mysql login should have the permission to access the db sepcified in the JDBC connection string. This Can be acheived using the following command
GRANT ALL ON Databasename.* TO username#'%' IDENTIFIED BY 'password';
The answer is located at http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/5.0/CDH5-Installation-Guide/cdh5ig_hive_schema_tool.html
To suppress the schema check and allow the metastore to implicitly modify the schema, you need to set the hive.metastore.schema.verification configuration property to false in hive-site.xml.
reconfigure your hive using -hiveconf hive.root.logger=warn,console than find the detail reason that why you could not instantiate your hive mate store client.
The problem i met was wrong mysql configuration, the error message is "Binary logging not possible. Message: Transaction level 'READ-COMMITTED' in InnoDB is not safe for binlog mode 'STATEMENT'". When i change binlog_format from "statement" to "binlog_format=mixed", then hive meta store client instantiate successfully.
Hope it work for you.
I had similar issue and the cause in my case was selinux where it prevented Postgres from proper running.
I inserted following line to the first line of /etc/rc3.d/S64postgresql -
echo 0 > /selinux/enforce # Disable selinux temporarily
and restarted the node with the hive metastore.
So generally you can check two things:
Check whether the DB for the metastore is running properly
Whether the user/passwd from the property is correct
Related
Error
Receiving this error with HBase Storage Handler in Hive when I run a query in a Kerberized environment.
on HBase 1.5
Caused by: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException:
No registered coprocessor service found for name AuthenticationService in region hbase:meta,,1
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8499)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2282)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2264)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36808)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2399)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291)
The important part being:
No registered coprocessor service found for name AuthenticationService
in region hbase:meta,,1
I did some reading and learned that AuthenticationService is provided by TokenProvider coprocessor.
In hbase-site.xml ensure these options are configured
hadoop.security.authentication
hbase.coprocessor.master.classes
hbase.coprocessor.region.classes
Ensure values are configured as follows:
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.hadoop.hbase.security.access.AccessController</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
</property>
Note:
In older versions of HBase the settinghbase.coprocessor.regionserver.classes was used, make sure you are using the correct; hbase.coprocessor.region.classes
I have tried to configured Impala to run on top of Alluxio, but failed.
Here is the Impala configurations:
/etc/impala/conf/core-site.xml(http://www.alluxio.org/docs/1.6/en/Running-Hadoop-MapReduce-on-Alluxio.html)
<configuration>
<property>
<name>fs.alluxio.impl</name>
<value>alluxio.hadoop.FileSystem</value>
<description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
<name>fs.AbstractFileSystem.alluxio.impl</name>
<value>alluxio.hadoop.AlluxioFileSystem</value>
<description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>
</configuration>
/etc/impala/conf/hive-site.xml(http://www.alluxio.org/docs/1.6/en/Running-Hive-with-Alluxio.html)
<property>
<name>fs.defaultFS</name>
<value>alluxio://master_hostname:port</value>
</property>
Then I started Impala(impala-server, impala-catalogd, impala-state-store), but in the log I found this:
...impala-server.cc:282] Currently configured default file system: FileSystem. fs.defaultFS (alluxio://192.168.1.10:19998/) is not supported.
...impala-server.cc:285] Aborting Impala Server startup due to improper configuration. Impalad exiting.
I have searched a lot on Bing but got no luck. Even there is few result on search key words 'impala on alluxio'. So can impala run on top of alluxio? Any suggestions will be appreciated.
My Impala version: 2.10.0-cdh5.13.0 RELEASE, Alluxio version: alluxio-1.8.0-hadoop-2.7
Have you tried running Hive with external tables on Alluxio? Instead of setting Alluxio as defaultFS, remove
<property>
<name>fs.defaultFS</name>
<value>alluxio://master_hostname:port</value>
</property>
and use something like the following to create a table on Alluxio:
hive> CREATE TABLE u_user (
userid INT,
age INT,
gender CHAR(1),
occupation STRING,
zipcode STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
LOCATION 'alluxio://master_hostname:port/table_path';
That might help workaround Impala's filesystem implementation check. Also there is a bug in CDH 5.13 and below which prevents Impala from reading data in Alluxio. You might want to upgrade to CDH 5.14 which fixed that issue.
Failed to open new session: java.lang.RuntimeException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User: hadoop is not allowed to impersonate cheng
User:hadoop is my hadoop install use,and cheng is ubuntu user.
I have already the following configuration in my core-site.xml:
<name>hadoop.proxyuser.hive.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hive.groups</name>
<value>*</value>
</property>
the hive user is not exist before,so I change the hadoop.proxyuser.hive.groups to
hadoop.proxyuser.hadoop.group and so on.in hue config hue.ini,set the hue user.
so the problem is solution.
I am running a hive query and I got the following error when setting the hive.execution.engine=tez, while the query is working under engine=MR.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
My query is an inner join and the data is quite big.
Another thing is that I have met this problem before. But tez works later so I thought it was about something unstable about hive.
While running your HQL via hive include following parameter. This will give you detailed logs and you can determine the root cause easily.
-hiveconf hive.root.logger=DEBUG,console
I faced similar problem and above property help me big time.
e.g.: I got following message
16/04/14 10:29:26 ERROR exec.Task: Failed to execute tez graph.
org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=20480, maxMemory=11288
When I changed my setting to 11288, my query went through fine.
Once check your yarn-site.xml with following properties.
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4</value>
<description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
</configuration>
Found this post, which made it work for me. Needed to add the username
hadoop
In hive-site.xml(hive-0.9.0),I have 2 properties
1) javax.jdo.option.ConnectionUserName
2) javax.jdo.option.ConnectionPassword
what does these do ?, which username & password should I give ?
are they related to root/mysql ?
Alongside javax.jdo.option.ConnectionURL and javax.jdo.ConnectionDriver they are intended to connect to Hive's metastore. Are you planning on using MySQL or another database as metastore? You only need to set these if you're not using Hive's standard metastore (Derby).
An example of how to set up hive-site.xml when using MySQL:
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/<databasename></value>
</property>
<property>
<name>avax.jdo.ConnectionDriver</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value><your database user></value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value><your database user password></value>
</property>
If you're indeed planning on running another database as metastore don't forget to create a database and run the scripts in $HIVE_HOME/scripts/metastore/upgrade/.
The easiest to do this latter in my opinion is to first create your database in the MySQL shell then exit the shell and use this command from your shell: mysql -u <username> -p -h <host> <databasename> < <sql script>
By the way, I think it's better to create a Hadoop specific database user instead of using your root user.