I got error "Credential cache is empty" during ODBC Hive tests. See full error detail
ODBC Hive - Test Results
[Cloudera][Hardy] (34) Error from server: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credential cache is empty).
Do you have any experience?
I tested different setting of MIT Kerberos in Windows e.g.:
generate kerberos ticket, kinit.exe -k -t app_store.keytab app_store#HW.PROD.BDP'
checked kerberos tickets in cache, klist.exe
setup KRB5CCNAME=C:\cache\krb5cache and KRB5_CONFIG=c:\ProgramData\MIT\Kerberos5\krb5.ini
I see a few possible issues:
You have to check that krb5cache is file (not directory), it is important point
The path to the cache have to be different for different person use this setting for variable KRB5CCNAME=%USERPROFILE%\krb5cache
You have to generate kerberos ticket before run ODBC Test, see
kinit.exe -k -t "c:\Apps\MIT\Kerberos\store.keytab" store#HW.PROD.BDP
I'm trying to connect impala to superset, and when I test the connection prints: "Seems OK!", and when I try to see databases on impala with the SQL Editor in the left side it shows all databases without problems.
Preview of Databases/Tables
But when i write a query and click on "Run Query", it gives the error: "Could not start SASL: b'Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket expired)'"
Error running query
I'm running superset with SSL and in production mode (with Gunicorn) and Impala with SSL in a Kerberized Hadoop Cluster, and my impala database config is:
Impala Config
And in the extras I put:
{
"metadata_params": {},
"engine_params": {
"connect_args": {
"port": 21050,
"use_ssl": "True",
"ca_cert": "path/to/my/ca_cert.pem",
"auth_mechanism": "GSSAPI"
}
},
"metadata_cache_timeout": {},
"schemas_allowed_for_csv_upload": []
}
How can I solve this error? In my superset log it only shows:
Triggering query_id: 65
INFO:superset.views.core:Triggering query_id: 65
Query 65: Running query on a Celery worker
INFO:superset.views.core:Query 65: Running query on a Celery worker
Versions: Superset 0.36.0, Impyla 0.16.2
I was able to fix this error doing this steps:
1 - Created service user for celery-worker, created a kerberos ticket for him and created a crontab to renew the ticket.
2 - Runned celery worker from this service user, instead running from root.
3 - Killed an celery-worker that was running in another machine of my cluster
4 - Restarted Impala and Superset
I think this error ocurred because in some queries instead of use the celery worker in my superset machine, it was using the celery worker that was in another machine without a valid kerberos ticket. I could fix this error because when I was reading celery-worker log , it showed that a connection with the celery worker in other machine failed in a query running.
I am running puppet 3.7. The certs are expiring for me so I have updated the certs (after creating a backup so I am able to get back to the original state and that's fine). After updating the certs on puppetmaster using this, updating certs on the agent using this and updating certs on puppetdb using this, I am unable to run puppet agent successfully on a client box. It gives me the following error:
root#ip-10-181-36:/var/lib/puppet# sudo puppet agent -t
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in 'issue_deprecation_warning')
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: newer-generic-host(127.0.0.1) access to /node/ip-10-181-36 [find] authenticated at :39
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: newer-generic-host(127.0.0.1) access to /catalog/ip-10-181-36 [find] authenticated at :1
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: newer-generic-host(127.0.0.1) access to /report/ip-10-181-36 [save] authenticated at :91
I am stuck at this point and no googling or reading docs or seeing the logs is helping. Does anyone have any ideas?
My Cassandra instances are not listening on 127.0.0.1. When I start datastax-agent I find this in logs:
# tail -n 100 /var/log/datastax-agent/agent.log
...
ERROR [Initialization] 2015-05-19 22:35:04,064 Can't connect to Cassandra, retrying soon.
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:220)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1231)
at com.datastax.driver.core.Cluster.init(Cluster.java:158)
at com.datastax.driver.core.Cluster.connect(Cluster.java:246)
at clojurewerkz.cassaforte.client$connect_or_close.doInvoke(client.clj:149)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojurewerkz.cassaforte.client$connect.invoke(client.clj:165)
at opsagent.cassandra$setup_cassandra$fn__8157.invoke(cassandra.clj:344)
at again.core$with_retries_STAR_$fn__8013.invoke(core.clj:98)
at again.core$with_retries_STAR_.invoke(core.clj:97)
at opsagent.cassandra$setup_cassandra.invoke(cassandra.clj:339)
at opsagent.opsagent$setup_cassandra.invoke(opsagent.clj:153)
at opsagent.jmx$determine_ip.invoke(jmx.clj:276)
at opsagent.jmx$setup_jmx$fn__8438.invoke(jmx.clj:293)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
How can I change which address the Datastax Agent connects to? I have tried setting local_interface in the agent's address.yaml (and restarting agent), but that doesn't seem to work.
The secret was to set rpc_address to 0.0.0.0. Cred to LHWizard for pointing this out.
After I login to rabbitmq, I get the following error :
Got response code 500 with body
Internal Server Error
The server encountered an error while processing this request:
{error,{error,{badmatch,{error,nxdomain}},
[{rabbit_nodes,cluster_name_default,0},
{rabbit_nodes,cluster_name,0},
{rabbit_mgmt_wm_overview,to_json,2},
{webmachine_resource,resource_call,3},
{webmachine_resource,do,3},
{webmachine_decision_core,resource_call,1},
{webmachine_decision_core,decision,1},
{webmachine_decision_core,handle_request,2}]}}
I see the following error in the log file in /var/log/rabbitmq :
=ERROR REPORT==== 31-Oct-2014::06:20:40 ===
webmachine error: path="/api/overview"
{error,{error,{badmatch,{error,nxdomain}},
[{rabbit_nodes,cluster_name_default,0},
{rabbit_nodes,cluster_name,0},
{rabbit_mgmt_wm_overview,to_json,2},
{webmachine_resource,resource_call,3},
{webmachine_resource,do,3},
{webmachine_decision_core,resource_call,1},
{webmachine_decision_core,decision,1},
{webmachine_decision_core,handle_request,2}]}}
The workers are able to connect to the broker and are receiving the messages, also the new relic plugin for rabbitmq seems to be working fine. However I am unable to login thru the management plugin. Any pointers in this regard will be helpful.
I had updated the hostname of the system and that was causing the issue. See the link below
https://groups.google.com/forum/#!msg/rabbitmq-users/9P-BAwGVHJU/fwOpZPJywwYJ
I added 127.0.0.1 'hostname' in /etc/hosts. That solved the management plugin problem. However rabbitmqctl still showed the following error. Restarted rabbitmq and it solved the rabbitmqctl problem as well
Listing queues ...
Error: unable to connect to node 'rabbit#<hostname>': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit#<hostname>']
rabbit#<hostname>:
* connected to epmd (port 4369) on <hostname>
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
current node details:
- node name: <nodename>
- home dir: <homedir>
- cookie hash: <cookiehash>