When I connect to a beeline database, I don't need to enter the user and password, I just press enter and I get access to the database.
but when I want to write onto the database I get a permission denied error:
Error: java.io.IOException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hive, access=READ, inode="/apps/hive/warehouse
I would know which default user should I use to connect to my database in order to add policies to it.
I thought that it's hive, but it seems it's not.
beeline takes a user account when you connect
beeline -u 'url' -n username
Use hdfs dfs -ls to find tables permissions (assuming you're not using Sentry or Ranger to manage permissions)
In any case, you don't use beeline to add permissions. By default, it's simply HDFS user/group permissions using chmod / chown, assuming you have ACLs enabled.
Try one of the below two.
Login with hadoop user
Give full permissions to /apps/hive/warehouse folder
Related
I am very new to hive and the hadoop ecosystem.
I am trying to create a new table in hive but I am encountering this error:
According to some suggestions, I have to set the Ranger policies but upon checking, policies already had permissions to "All"
Same permissions were also given to other policies.
Did I miss something? Thank you.
You might need HDFS user directory which can be created by the administrator using sudo -u hdfs hdfs dfs -mkdir /user/<user_id>
In case you want to check if there is one exists:
hdfs dfs -ls /user | grep <user_id>
I had a similar issue. You may want to check Ranger > Audit > Plugin Status to see if the policy is being applied. If it's not it may be that you have a jersey-client classpath conflict. More details here:
https://jonmorisissqlblog.blogspot.com/2021/06/ranger-hive-policy-activation-time.html
I have installed the Cloud SDK for Google Cloud. I've logged in using auth which redirected me to the gmail-login. Created the SSH key and even logged in by SFTP using Filezilla.
The problem is, when I log in using the gmail auth, SDK shell (or putty?) logs me into an account that is not admin. It has created another SSH user account (named 'Acer', after my pc) and logs me into it. Due to this, FTP starts at the /home/Acer folder. I want access to the /home/admin/web folder, but I don't have it now.
How can I create a SSH key for the admin account so that I can gain access to the folder mentioned above? Otherwise, is it possible to grant 'Acer' the permissions to access all the folders?
I have a few suggestions.
First a bit of background. If you run this command on your home workstation:
sudo find / -iname gcloud
You'll discover a gcloud configuration folder for each user on your home workstation. You'll probably see something like this:
/root/.config/gcloud
/home/Acer/.config/gcloud
If you change directory into /home/Acer/.config/gcloud/configurations you'll see a file named 'config_default'. This file will contain the default account to use for that user ('Acer').
Because you have performed gcloud auth login as that user, and during that process selected your gmail account, it will contain that gmail ID/account within the config file for that user. If you would like a user named 'admin' to log into your project, you could try adding a user named 'admin' to your home workstation, and then before attempting to use gcloud auth login, ensure you switch user on your home workstation to user 'admin'. This will generate a gcloud configuration on your home workstation for user admin, and propagate SSH keys etc.
If you want to create ssh keys manually there's some useful info here.
(For what it's worth, if you decide to use gcloud compute ssh to log into your instance home workstation, you can specify the user in the command you would like to log in as. For example gcloud compute ssh admin#INSTANCE_NAME).
I want access to the /home/admin/web folder, but I don't have it now.
Even if you are logged into the machine as a different user (in this case 'Acer'), the folder /home/admin/web should still exist on the instance if it existed previously. If you land in folder /home/Acer have you tried changing directory to the folder above and then listing the folders to see if /home/admin/ exists?
For example, from /home/Acer run:
$ cd ..
then
$ ls
You should be able to see /home/admin/.
Otherwise, is it possible to grant 'Acer' the permissions to access
all the folders?
Yes this is also possible. If you access the instance as the project owner (the easiest way would be to log into the Console as the owner of the project and use the SSH functionality in the console to access the instance). Now you can run this command:
$ sudo chown Acer.Acer -R /home/admin/web
This will make user 'Acer' owner of directory /home/admin/web and all files/directories below it (thanks to the -R switch).
Now when you next access the instance as user 'Acer' you'll be able to access /home/admin/web by running the following and you'll also have read/write capabilities:
$ cd /home/admin/web
I have a Compute VM that has storage permission of read-write. When I try to run the following command:
gsutil rsync -R local-dir gs://folder1/prod/www.domain.tld
I get insufficient permission error:
Building synchronization state...
Skipping cloud sub-directory placeholder object (gs://folder1/prod/www.domain.tld/) because such objects aren't needed in (and would interfere with) directories in the local file system
Starting synchronization
Copying file://local-dir/.gitignore [Content-Type=application/octet-stream]...
Uploading gs://folder1/prod/www.domain.tld/.gitignore: 174 B/174 B
AccessDeniedException: 403 Insufficient Permission
I am not sure what account needs to have specific permissions or how I would even assign the permissions. Can someone provide some direction as to what I need to look into? The only permissions I can think of is a service account but I am not sure how to create one or if that is what I even need to do. Also, once I grant an account access/permission, how would I use the account to authenticate?
Use gcloud auth list command to lookup the active account. The service account of Comoute Engine is similar to the following:
123845678986-compute#developer.gserviceaccount.com (active)
By default, this service account is a member of your project with Edit permission. Check ACLs of your GCS bucket and its folders and make sure the GCE service account or the group which it is belong to, has ownership or editing rights to them.
I have a cloudera cluster with HDFS and Hue services and I'm trying to unify the authentication using LDAP.
I have my LDAP server running thanks to 389-ds (not sure if is the best way) and I can log into Hue with users from the LDAP server. When I login for first time, Hue creates the home directory in the HDFS.
But is not using the UID I set when I added the user to the LDAP server.
It wouldn't be a problem if I just access the HDFS via Hue but I also have a machine with the HDFS mounted via NFS.
I'm also having problems to add LDAP authentication in the machine with the NFS mount. I can do su username (username being a user in the LDAP server) and the system adds a home directory, but I cannot authenticate via SSH using LDAP users. I need this to avoid adding local users too.
My main question is: How to force HDFS or Hue to use the same UID I set when I create LDAP users.
More details:
I have configured LDAP in cloudera for both Hue and Hadoop (not sure if the latter is using it properly)
I know I could, maybe, change the UID a posteriori to the one set by Hue at the first login, but is more a workaround than a clean solution.
Pictures:
In this example, potato user has an uid 10104, but if I do ls -la /users/potato in the NFS mount, it says that the folder belongs to a user with uid 3312528423.
I am facing some problems with hive partition creation where the permissions user has in hdfs are acl based.
1. I created a normal user in linux.
2. I gave him permissions recursively on a directory which is referred to by an external table.
(e.g. hdfs dfs -setfacl -R -m default:user:newUserName:rwx /apps/dbname/tblname)
I checked to see the permissions are recursively applied and I can read and write to the
directory even though I do not have any POSIX related permissions on the same. i.e. I only have the ACLS.
I logged in as newUserName and started hive and ran an "alter table add partition" command, where the location for the partition is /apps/dbname/tablename/somefolder
hive responds with the error: Authorization failed: java.security.AccessControlException: action WRITE not permitted on path hdfs://sandbox.hortonworks.com:8020/apps/dbname/tblname for user newUserName. Use SHOW GRANT to get more details.
What am I missing here. dfs.namenode.acls.enabled is true. I thought that was all that was required for ACLs to work. I am using hortonworks hdp2.1
Thanks
In current version of hive (0.13.x) Support for HDFS ACL (introduced in Apache Hadoop 2.4) is yet to be added.
Please find below jira which addresses the above issue:
https://issues.apache.org/jira/browse/HIVE-7714