Using sqoop version 1.4.5. while using sqoop import on remote hdfs, facing kerberos authentication issue. https://photos.app.goo.gl/KDbf1LKe4LCXahAw5
And this
https://photos.app.goo.gl/Vxzv1HpwrUYjJTzy6
Please advice configuration changes required to remove this error?
Related
I am trying to run a bigquery query on Airflow with MWAA.
This query uses a table that is based on a Google Sheet. When I run it, I have the following error:
google.api_core.exceptions.Forbidden: 403 Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.
I already have a working Google cloud connection on Airflow with an admin service account.
Also:
This service account has access to the google sheet
I added https://www.googleapis.com/auth/drive in the scopes of the Airflow connection
I re-generated a JSON file
Am I doing something wrong? Any idea what I can do to fix this problem?
Thanks a lot
I fixed my issue by creating a NEW Airflow connection. It's a new google cloud connection, with the exact same values as the default google_cloud_default values. Now it works perfectly.
Hope it can help !
I need to connect to a hadoop (apache hive) database that requires authentication with user/password.
I normally access it with DBeaver (JDBC drivers installed) but now I need to retrieve data using a simple query with a script in Python 3.7 (not Python 2.x).
I tried several methods without success.
Does someone have a solution that works?
I tried pyhive, presto, pyhs2, impala, ibis
You can follow below steps for connecting hive from python by ODBC driver on windows machine,
Install required driver for Hive. If you are using Cloudera/Hortonworks you can find respective drivers on their website.
Once installation done, go to 64-bit ODBC Administrator from startup menu or control panel.
Fill required server details and connection user id and password. Below is screen shot for same.
Once that is done check test connection by clicking on Test button, and check if it is successful or not.
You can use same DSN while connecting hive from python, below is code snippet for same.
import pyodbc
import pandas as pd
with pyodbc.connect("DSN=<replace DSN name>", autocommit=True) as conn:
df = pd.read_sql("<Hive Query>", conn)
#ASK many thanks!
Following your suggestion, I created the DSN and was able to connect and read the table using the following code:
import pyodbc
import pandas as pd
table = "table_name"
query = 'SELECT * FROM {0}'.format(table)
with pyodbc.connect("DSN=Hive_connection",autocommit=True) as conn:
df=pd.read_sql_query(query,conn)
Ambari shows me an alert message:
"Ranger Admin password check
User:admin credentials on Ambari UI are not in sync with Ranger"
How I can resolve this ? What should I need to do.
I have tried to set the same password for (amb_ranger_admin, admin, etc) but it didn't help.
My version of Ambari is 2.7.1
HDP 3.0.1
This topic seems to have been discussed here: https://community.hortonworks.com/answers/221995/view.html
To summarize, the key steps are:
Set the password first in ranger and then in ambari
Check if the passwords are the same
If they are not the same? : Analyze which one (likely Ambari) is not updating the config
If they are the same? : Analyze why you still get the warning
I am working on a Laravel project that uses dingo package to manage some APIs. I changed CACHE_DRIVER=array variable in my .env file to CACHE_DRIVER=redis because dingo no longer support array for CACHE_DRIVER. I therefore installed redis on my system and included the package in my Laravel project by adding "predis/predis": "~1.0" in my composer.json and updating with the command composer update. Up until now everything works just fine. However, to create database table and seed them using php artisan migrate --seed, I get the error:
[Predis\Connection\ConnectionException]
SELECT failed: NOAUTH Authentication required. [tcp://127.0.0.1:6379]
Note: when I was installing redis, I added a password. I also authenticated using the two commands redis-cli to switch to redis and then AUTH mypassword. Yet when I try to seed, it still throws the same error. Please what am I doing wrong?
Thanks for any help.
I would start with setting redis password in REDIS_PASSWORD environment variable (e.g. in .env file). See https://laravel.com/docs/5.3/redis#configuration for more details about redis configuration in Laravel.
Make sure you are running your redis server when you are seeding.
Run: redis-server
I need your help in resolving an issue related to Hive Permission. I have enabled Sentry in CDH 5.5 environment but Kerberos is not sent. I could not able to create roles or grant permission to hive objects. For every access need to login through root and sudo to hive and access the object.
When i try su to hive from hdfs user and try giving below command
set role admin;
FAILED: SemanticException The current builtin authorization in Hive is incomplete and disabled.
Please suggest me how to resolve this.
Thanks & Regards