The create table stage -
CREATE TABLE IF NOT EXISTS `aa_db_aaa_prod.BASE` (`UpdatedByName` STRING, `UpdatedOn` BIGINT, `UpdatedOnTimeZoneOffset` INTEGER);
ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-
packages/cloudera/cdp/airflow/operators/cdw_operator.py", line 108, in execute
self.hook.run_cli(hql=self.hql, schema=self.schema, hive_conf=self.hiveconfs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-
packages/cloudera/cdp/airflow/hooks/cdw_hook.py", line 204, in run_cli
sub_process = subprocess.Popen(
File "/opt/bitnami/python/lib/python3.8/subprocess.py", line 858, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/opt/bitnami/python/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'beeline'
However, my confusion is, this is not new installation and nothing has changed. Also, other tables in the dag are successful. Beeline connection through the cli on airflow server with same user is successful. I cant understand the cause for this error ? Any leads on this please ?
The cause was, /repos/cloudera/parcels/CDH/bin/beeline got removed on one of the airflow node, which was causing this issue.
Related
I have one issue that I would like fixed but I am not able to. I have a small script that requires SSH key to work properly. When the SSH key is not properly loaded, the following Python Error appears:
SSH: Permission denied (publickey). Traceback (most recent call last):
File "/path/to/python3file.py", line 117, in
func.func_check() File "/path/to/python3file.py", line 18, in func_check
ssh = subprocess.check_output(["ssh", "-p22", "{}#{}".format("user", self.host), command]) File
"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py",
line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File
"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py",
line 512, in run
raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ssh', '-p22', 'user#chost',
' script func user']' returned non-zero exit status 255.
Basically, I want to replace the entire error code from the above with something more user-friendly like:
Please import SSH key
Is that even possible?
Thank you.
Use Try Except, and specify the type of error you are handling. In this case it is IOError.
try:
print('my code here')
except IOError, e:
print('Please import SSH key')
I started a JHBuild with the wrong arguments (forgot 'build') and hit control-C at what appears to have been the wrong moment.
Now when I try any JHBuild command, e.g. jhbuild bootstrap, I get:
Traceback (most recent call last):
File "/Users/gnucashdev/Source/jhbuild/jhbuild/config.py", line 197, in load
execfile(filename, config)
File "/Users/gnucashdev/.jhbuildrc", line 408, in <module>
execfile(_userrc)
File "/Users/gnucashdev/.jhbuildrc-custom", line 22, in <module>
setup_sdk()
File "/Users/gnucashdev/.jhbuildrc", line 260, in setup_sdk
gcc = _popen("xcrun -f gcc")
File "/Users/gnucashdev/.jhbuildrc", line 41, in _popen
raise RuntimeError, "Failed to close %s stream" % cmd_arg
RuntimeError: Failed to close xcrun -f gcc stream
jhbuild: could not load config file
I've tried re-installing jhbuild with
./gtk-osx-build-setup.sh
but the next step - i.e.
jhbuild bootstrap
yields the above error. Some file appears to have been compromised, perhaps truncated. But I'm having a hard time figuring out which.
I had the same error. xcrun is returning an error, probably due to an incorrect environment variable. In my case, I was running jhbuild while in a jhbuild shell, which caused the SDKDIR environment variable to contain 2 copies of the path to the SDK directory. Exiting the jhbuild shell fixed the problem.
This is my first time using mrjob, however I encounter the following problems when executing the relevant python script using mrjob:
No configs found; falling back on auto-configuration
Looking for hadoop binary in /home/work/alex/tools/hadoop-client-1.5.5/hadoop/bin...
Found hadoop binary: /home/work/alex/tools/hadoop-client-1.5.5/hadoop/bin/hadoop
Creating temp directory /tmp/simrank_mr.work.20161204.050846.350418
Using Hadoop version 2
STDERR: 16/12/04 13:08:48 INFO common.UpdateService: ZkstatusUpdater to hn01-lp-hdfs.dmop.ac.com:54310 started
STDERR: mkdir: cannot create directory -p: File exists
STDERR: java.io.IOException: cannot create directory -p: File exists
STDERR: at org.apache.hadoop.fs.FsShell.mkdir(FsShell.java:1020)
STDERR: at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1934)
STDERR: at org.apache.hadoop.fs.FsShell.run(FsShell.java:2259)
STDERR: at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
STDERR: at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
STDERR: at org.apache.hadoop.fs.FsShell.main(FsShell.java:2331)
Traceback (most recent call last):
File "simrank_mr.py", line 121, in <module>
MRSimRank.run()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/job.py", line 429, in run
mr_job.execute()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/job.py", line 447, in execute
super(MRJob, self).execute()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/launch.py", line 158, in execute
self.run_job()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/launch.py", line 228, in run_job
runner.run()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/runner.py", line 481, in run
self._run()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/hadoop.py", line 335, in _run
self._upload_local_files_to_hdfs()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/hadoop.py", line 362, in _upload_local_files_to_hdfs
self.fs.mkdir(self._upload_mgr.prefix)
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/fs/composite.py", line 76, in mkdir
return self._do_action('mkdir', path)
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/fs/composite.py", line 63, in _do_action
raise first_exception
IOError: Could not mkdir hdfs:///user/work/alex/tmp/cluster/mrjob/tmp/tmp/simrank_mr.work.20161204.050846.350418/files/
Anyone knows how to solve this problem? Many thanks!
I am using HDP 2.4.2 and I had previously installed the zeppelin server. It was working fine but today when i restarted the cluster ( AWS nodes were restarted), Ambari shows that Zeppelin server is not running and fails to start the server with the following error:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 235, in <module>
Master().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 169, in start
+ params.zeppelin_log_file, user=params.zeppelin_user)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/zeppelin-server/lib/bin/zeppelin-daemon.sh start >> /var/log/zeppelin/zeppelin-setup.log' returned 1. /usr/hdp/current/zeppelin-server/lib/bin/zeppelin-daemon.sh: line 187: /var/run/zeppelin-notebook/zeppelin-zeppelin-ip-10-0-0-11.eu-west-1.compute.internal.pid: Permission denied
cat: /var/run/zeppelin-notebook/zeppelin-zeppelin-ip-10-0-0-11.eu-west-1.compute.internal.pid: No such file or directory
In the zeppelin logs:
ERROR [2016-06-06 03:20:36,714] ({main} VFSNotebookRepo.java[list]:140) - Can't read note file:///usr/hdp/current/zeppelin-server/lib/notebook/screenshots java.io.IOException: file:///usr/hdp/current/zeppelin-server/lib/notebook/screenshots/note.json not found
ERROR [2016-06-06 03:34:12,795] ({main} Notebook.java[loadNoteFromRepo]:330) - Failed to load 2BHU1G67J java.io.IOException: file:///usr/hdp/current/zeppelin-server/lib/notebook/2BHU1G67J is not a directory
But for some reason, the zeppelin port is listening and despite these errors, the zeppelin server is running fine and executing all the queries. Please advice on how to correct the issue in Ambari and start the service without error from ambari.
The problem is with the PID file for the zeppelin service. It's either owned by the wrong user or has the wrong permissions. Manually stop the zeppelin service then delete the pid file locate at: /var/run/zeppelin-notebook/zeppelin-zeppelin-ip-10-0-0-11.eu-west-1.compute.internal.pid. Double check the owner/permissions on the /var/run/zeppelin-notebook folder as well. You should then be able to restart the service in the Ambari UI.
Every time I run a Kivy app I see OSError (see it in last line of my given example). Even though my app runs successfully. What could be the cause of this error?
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/dist-packages/kivy/input/providers/mtdev.py", line 197, in _thread_run
_device = Device(_fn)
File "/usr/lib/python2.7/dist-packages/kivy/lib/mtdev.py", line 131, in __init__
self._fd = os.open(filename, os.O_NONBLOCK | os.O_RDONLY)
OSError: [Errno 13] Permission denied: '/dev/input/event5'
This error is not important, it just means that kivy checked the possible input providers in your OS and found that this one is forbidden.