I begin learning BigData with Hadoop Hive
I can't upload local data to Hive table
Hive command is:
load data local inpath '/usr/local/nhanvien/testHive.txt' into table nhanvien;
I get error :
Loading data to table hivetest.nhanvien Failed with exception Unable
to move source file:/usr/local/nhanvien/testHive.txt to destination
hdfs://localhost:9000/user/hive/warehouse/hivetest.db/nhanvien/testHive_copy_3.txt
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
was try:
hadoop fs -chmod g+w /user/hive/warehouse
sudo chmod -R 777 /home/abc/employeedetails
it still get this error
can someone give me solution ?
You can try with:
export HADOOP_USER_NAME=hdfs
hive -e "load data local inpath '/usr/local/nhanvien/testHive.txt' into table nhanvien;"
Its a permission issue. Try giving permission to local file and directory where your file exists.
sudo chmod -R 777 /usr/local/nhanvien/testHive.txt
Then
Login as $HDFS_USER and run the following command:
hdfs dfs -chown -R $HIVE_USER:$HDFS_USER /user/hive
hdfs dfs -chmod -R 775 /user/hive
hdfs dfs -chmod -R 775 /user/hive/warehouse
You can also configure for hdfs-site.xml such as:
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
This configure will disable permissions on HDFS. So, a regular user can do the operations on HDFS.
Hope this help.
Related
I am using Hortonworks Sandbox (not sure if this matters) and was trying to copy a csv file from '/user/root/' directory into '/user/nyc_taxi/' directory but it gave me a 'No such file or directory' error. I am very sure the csv, called 'yellow_tripdata_2016_01.csv', is in the '/user/root/' directory.
I use Windows 10 and below are the commands I wrote in terminal:
>sudo su - hdfs
>hdfs dfs -mkdir /user/root
>hdfs dfs -mkdir /user/nyc_taxi
>hdfs dfs -chown root:hdfs /user/root
>hdfs dfs -chown root:hdfs /user/nyc_taxi
>exit
>hadoop fs -put yellow_tripdata_2016_01.csv /user/nyc_taxi/
The first 6 commands had no problem. But when I ran the last one the terminal returned the 'No such file or directory' error. I had also tried specifying the actual path of the csv by running this command:
hadoop fs -put /user/root/yellow_tripdata_2016_01.csv /user/nyc_taxi/
But the same error message still showed up.
As can be seen from the screenshot below, the csv file is clearly in the '/user/root/' directory.
Any feedback/recommendation is greatly appreciated!
-put command looks for yellow_tripdata_2016_01.csv file in local file system not on hdfs.
But your file is in HDFS that's the reason why you are getting that error.
Refer to this link for more details regards to Hadoop -put/-cp/-mv commands.
If you want to copy/move the HDFS file into another HDFS directory use below commands
Copy the file into another hdfs directory:
hadoop fs -cp /user/root/yellow_tripdata_2016_01.csv /user/nyc_taxi/
Move the file into another directory:
hadoop fs -mv /user/root/yellow_tripdata_2016_01.csv /user/nyc_taxi/
I'm seeing an issue with running the Hive CLI. When I run the CLI on an edge node I receive the following error regarding HDFS permissions:
c784gnj:~ # sudo hive
/usr/lib/hive/conf/hive-env.sh: line 5: /usr/lib/hive/lib/hive-hbase-handler-1.1.0-cdh5.5.2.jar,/usr/lib/hbase/hbase-common.jar,/usr/lib/hbase/lib/htrace-core4-4.0.1-incubating.jar,/usr/lib/hbase/lib/htrace-core-3.2.0-incubating.jar,/usr/lib/hbase/lib/htrace-core.jar,/usr/lib/hbase/hbase-hadoop2-compat.jar,/usr/lib/hbase/hbase-client.jar,/usr/lib/hbase/hbase-server.jar,/usr/lib/hbase/hbase-hadoop-compat.jar,/usr/lib/hbase/hbase-protocol.jar: No such file or directory
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
16/10/11 10:35:49 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist
Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.5.2.jar!/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=app1_K, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:216)
What is hive trying to write to in the /user directory in HDFS?
I can already see that /user/hive is created:
drwxrwxr-t - hive hive 0 2015-03-16 22:17 /user/hive
As you can see I am behind kerberos auth on Hadoop.
Thanks in advance!
Log says you need to set permission on HDFS /user directory to user app1_K
Command
hadoop fs -setfacl -m -R user:app1_K:rwx /user
Execute this command as privileged user from Hadoop bin
If you get similar permission error on any other hdfs directory, then you have to grant permission on that directory.
Refer the below link for more information.
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#ACLs_Access_Control_Lists
Instead of disabling HDFS access privileges altogether, as suggested by #Kumar, you might simply create a HDFS home dir for every new user on the system, so that Hive/Spark/Pig/Sqoop jobs have a valid location to create temp files...
On a Kerberized cluster:
kinit hdfs#MY.REALM
hdfs dfs -mkdir /user/app1_k
hdfs dfs -chown app1_k:app1_k /user/app1_k
Otherwise:
export HADOOP_USER_NAME=hdfs
hdfs dfs -mkdir /user/app1_k
hdfs dfs -chown app1_k:app1_k /user/app1_k
How can I set the umask for an Hive HQL script? Either via statements within the script or via a client side configuration set before running the script? I want to make the change on the client side without changing the server side configuration.
I've found that this works from a shell prompt, but I'd like to do it from inside a hive script.
$ hdfs dfs -Dfs.permissions.umask-mode=000 -mkdir /user/jeff/foo
$ hdfs dfs -Dfs.permissions.umask-mode=000 -put bar /user/jeff/foo
These tries don't work:
hive> dfs -mkdir -Dfs.permissions.umask-mode=000 /user/jeff/foo;
-mkdir: Illegal option -Dfs.permissions.umask-mode=000
hive> dfs -Dfs.permissions.umask-mode=000 -mkdir /user/jeff/foo;
-Dfs.permissions.umask-mode=000: Unknown command
Setting hive.files.umask.value in .hiverc doesn't have the desired effect (The g+w and o+w bits aren't set which was what I was trying to do with this umask.):
hive> set hive.files.umask.value;
hive.files.umask.value=000
hive> dfs -mkdir /user/jeff/foo;
hive> dfs -ls -d /user/jeff/foo;
drwxr-xr-x - jeff hadoop 0 2016-02-23 15:19 /user/jeff/foo
It looks like I'll need to sprinkle a bunch of "dfs -chmod 777 ..." statements in my HQL script.
Ideas??
while creating the table (with user admin) with Hive I have this error :
Your query has the following error(s):
Error while compiling statement: FAILED: SemanticException No valid privileges Required privileges for this query: Server=server1->Db=*->Table=+->action=insert;Server=server1->Db=*->Table=+->action=select;
I do not understand why admin has not enough privileges, what should I do to solve the problem ?
Thanks
Login to terminal and run this command sudo -u hdfs hadoop fs -chmod 777 -R /user/hive/warehouse and try creating databases/tables.
The user admin in hue need not have permissions under hdfs location /user/hive/warehouse where all the Hive tables will be pointing to.
I'm trying to setup hadoop on mesos using the document below:
https://docs.mesosphere.com/tutorials/run-hadoop-on-mesos/
I'm facing a problem on step-9
sudo -u mapred ./hadoop-2.0.0-mr1-cdh4.2.1/bin/hadoop dfs -rm -f /hadoop-2.0.0-mr1-cdh4.2.1.tgz
sudo -u mapred /usr/bin/hadoop dfs -copyFromLocal ./hadoop-2.0.0-mr1-cdh4.2.1.tgz /
I am still new to this concept. I have to configure a mesos cluster using this tutorial:
https://www.digitalocean.com/community/tutorials/how-to-configure-a-production-ready-mesosphere-cluster-on-ubuntu-14-04
Now I'm getting errors while performing dfs commands:
root#station1:~# sudo -u mapred ./hadoop-2.0.0-mr1-cdh4.2.1/bin/hadoop dfs -rm -f /hadoop-2.0.0-mr1-cdh4.2.1.tgz
-rm: Expected authority at index 7: hdfs://
Usage: hadoop fs [generic options] -rm [-f] [-r|-R] [-skipTrash] <src> ...
This tutorial assumes you have HDFS already installed on your cluster. You can do this by manually installing HDFS on each node, or you can try out the new HDFS framework: https://github.com/mesosphere/hdfs
Does hadoop fs -ls hdfs:// work on its own? If not, you'll need to install and configure HDFS appropriately.