This is my first time using mrjob, however I encounter the following problems when executing the relevant python script using mrjob:
No configs found; falling back on auto-configuration
Looking for hadoop binary in /home/work/alex/tools/hadoop-client-1.5.5/hadoop/bin...
Found hadoop binary: /home/work/alex/tools/hadoop-client-1.5.5/hadoop/bin/hadoop
Creating temp directory /tmp/simrank_mr.work.20161204.050846.350418
Using Hadoop version 2
STDERR: 16/12/04 13:08:48 INFO common.UpdateService: ZkstatusUpdater to hn01-lp-hdfs.dmop.ac.com:54310 started
STDERR: mkdir: cannot create directory -p: File exists
STDERR: java.io.IOException: cannot create directory -p: File exists
STDERR: at org.apache.hadoop.fs.FsShell.mkdir(FsShell.java:1020)
STDERR: at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1934)
STDERR: at org.apache.hadoop.fs.FsShell.run(FsShell.java:2259)
STDERR: at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
STDERR: at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
STDERR: at org.apache.hadoop.fs.FsShell.main(FsShell.java:2331)
Traceback (most recent call last):
File "simrank_mr.py", line 121, in <module>
MRSimRank.run()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/job.py", line 429, in run
mr_job.execute()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/job.py", line 447, in execute
super(MRJob, self).execute()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/launch.py", line 158, in execute
self.run_job()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/launch.py", line 228, in run_job
runner.run()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/runner.py", line 481, in run
self._run()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/hadoop.py", line 335, in _run
self._upload_local_files_to_hdfs()
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/hadoop.py", line 362, in _upload_local_files_to_hdfs
self.fs.mkdir(self._upload_mgr.prefix)
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/fs/composite.py", line 76, in mkdir
return self._do_action('mkdir', path)
File "/home/work/.jumbo/lib/python2.7/site-packages/mrjob-0.5.6-py2.7.egg/mrjob/fs/composite.py", line 63, in _do_action
raise first_exception
IOError: Could not mkdir hdfs:///user/work/alex/tmp/cluster/mrjob/tmp/tmp/simrank_mr.work.20161204.050846.350418/files/
Anyone knows how to solve this problem? Many thanks!
Related
I am trying to cross build chromium on mac host. I have followed the instructions from https://chromium.googlesource.com/chromium/src.git/+/HEAD/docs/win_cross.md and everything seems fine. But when I run autoninja -C out/win chrome I face with the following error. Any help would be appreciated.
ninja: Entering directory `out/win'
[0/1] Regenerating ninja files
ERROR at //build/config/win/visual_studio_version.gni:27:7: Script returned non-zero exit code.
exec_script("../../vs_toolchain.py", [ "get_toolchain_dir" ], "scope")
^----------
Current dir: /chromium/src/out/win/
Command: python3 /chromium/src/build/vs_toolchain.py get_toolchain_dir
Returned 1.
stderr:
Traceback (most recent call last):
File "/chromium/src/build/vs_toolchain.py", line 587, in <module>
sys.exit(main())
File "/chromium/src/build/vs_toolchain.py", line 583, in main
return commands[sys.argv[1]](*sys.argv[2:])
File "/chromium/src/build/vs_toolchain.py", line 561, in GetToolchainDir
win_sdk_dir = SetEnvironmentAndGetSDKDir()
File "/chromium/src/build/vs_toolchain.py", line 554, in SetEnvironmentAndGetSDKDir
return NormalizePath(os.environ['WINDOWSSDKDIR'])
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 679, in __getitem__
raise KeyError(key) from None
KeyError: 'WINDOWSSDKDIR'
See //third_party/angle/gni/angle.gni:35:5: whence it was imported.
import("//build/config/win/visual_studio_version.gni")
^----------------------------------------------------
See //chrome/elevation_service/BUILD.gn:7:1: whence it was imported.
import("//build/toolchain/win/midl.gni")
^--------------------------------------
See //BUILD.gn:660:7: which caused the file to be included.
"//chrome/elevation_service:elevation_service_unittests",
^-------------------------------------------------------
FAILED: build.ninja
../../buildtools/mac/gn --root=../.. -q --ide=vs --regeneration gen .
ninja: error: rebuilding 'build.ninja': subcommand failed
I've built Tensorflow with custom SIMD extensions and created a wheel for it. If I simply do pip install /tmp/tensorflow_pkg/tensorflow-1.0.0-cp27-cp27mu-linux_x86_64.wh on the box that I built it on, that works. However if I upload the whl file to cloud storage, and do pip install https://storage.cloud.google.com/path/to/tensorflow-1.0.0-cp27-cp27mu-linux_x86_64.whl I get this error:
Collecting tensorflow==1.0.0 from https://storage.cloud.google.com/path/to/tensorflow-1.0.0-cp27-cp27mu-linux_x86_64.whl
Downloading https://storage.cloud.google.com/path/to/tensorflow-1.0.0-cp27-cp27mu-linux_x86_64.whl
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 620, in _prepare_file
session=self.session, hashes=hashes)
File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 821, in unpack_url
hashes=hashes
File "/usr/local/lib/python2.7/site-packages/pip/download.py", line 663, in unpack_http_url
unpack_file(from_path, location, content_type, link)
File "/usr/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 599, in unpack_file
flatten=not filename.endswith('.whl')
File "/usr/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 484, in unzip_file
zip = zipfile.ZipFile(zipfp, allowZip64=True)
File "/usr/local/lib/python2.7/zipfile.py", line 770, in __init__
self._RealGetContents()
File "/usr/local/lib/python2.7/zipfile.py", line 811, in _RealGetContents
raise BadZipfile, "File is not a zip file"
BadZipfile: File is not a zip file
Do I need to configure my build differently somehow?
(capturing the solution as an answer)
The URL used for the download is not correct. The base url needed to be storage.googleapis.com
I have installed all the pre-requests as in the tutorials.
When I tried to run odoo by using the following command,
./odoo.py --addons-path=addons,../mymodules --db-filter=mydb$
I am getting the following error message.
-bash: ./odoo.py: No such file or directory
Jinn-MacBook-Air:odoo juasoft$ ./odoo-bin
Traceback (most recent call last):
File "./odoo-bin", line 9, in <module>
odoo.cli.main()
File "/Users/juasoft/git/odoo/odoo/cli/command.py", line 64, in main
o.run(args)
File "/Users/juasoft/git/odoo/odoo/cli/server.py", line 164, in run
main(args)
File "/Users/juasoft/git/odoo/odoo/cli/server.py", line 122, in main
odoo.tools.config.parse_config(args)
File "/Users/juasoft/git/odoo/odoo/tools/config.py", line 315, in parse_config
odoo.netsvc.init_logger()
File "/Users/juasoft/git/odoo/odoo/netsvc.py", line 120, in init_logger
resetlocale()
File "/Users/juasoft/git/odoo/odoo/tools/translate.py", line 1185, in resetlocale
for ln in get_locales():
File "/Users/juasoft/git/odoo/odoo/tools/translate.py", line 1154, in get_locales
lang = locale.getdefaultlocale()[0]
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 545, in getdefaultlocale
return _parse_localename(localename)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 477, in _parse_localename
raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: UTF-8
Jinn-MacBook-Air:odoo juasoft$
change to your home directory
cd ~
Edit .bash_profile (with vim or any text editor)
vim .bash_profile
Add the following lines to it
export LC_ALL=es_US.UTF-8
export LANG=en_US.UTF-8
And then finally source the file to make the changes take effect immediately (without this you'll have to logout and login again as .bash_profile is executed only on login)
source .bash_profile
I am using HDP 2.4.2 and I had previously installed the zeppelin server. It was working fine but today when i restarted the cluster ( AWS nodes were restarted), Ambari shows that Zeppelin server is not running and fails to start the server with the following error:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 235, in <module>
Master().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 169, in start
+ params.zeppelin_log_file, user=params.zeppelin_user)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/zeppelin-server/lib/bin/zeppelin-daemon.sh start >> /var/log/zeppelin/zeppelin-setup.log' returned 1. /usr/hdp/current/zeppelin-server/lib/bin/zeppelin-daemon.sh: line 187: /var/run/zeppelin-notebook/zeppelin-zeppelin-ip-10-0-0-11.eu-west-1.compute.internal.pid: Permission denied
cat: /var/run/zeppelin-notebook/zeppelin-zeppelin-ip-10-0-0-11.eu-west-1.compute.internal.pid: No such file or directory
In the zeppelin logs:
ERROR [2016-06-06 03:20:36,714] ({main} VFSNotebookRepo.java[list]:140) - Can't read note file:///usr/hdp/current/zeppelin-server/lib/notebook/screenshots java.io.IOException: file:///usr/hdp/current/zeppelin-server/lib/notebook/screenshots/note.json not found
ERROR [2016-06-06 03:34:12,795] ({main} Notebook.java[loadNoteFromRepo]:330) - Failed to load 2BHU1G67J java.io.IOException: file:///usr/hdp/current/zeppelin-server/lib/notebook/2BHU1G67J is not a directory
But for some reason, the zeppelin port is listening and despite these errors, the zeppelin server is running fine and executing all the queries. Please advice on how to correct the issue in Ambari and start the service without error from ambari.
The problem is with the PID file for the zeppelin service. It's either owned by the wrong user or has the wrong permissions. Manually stop the zeppelin service then delete the pid file locate at: /var/run/zeppelin-notebook/zeppelin-zeppelin-ip-10-0-0-11.eu-west-1.compute.internal.pid. Double check the owner/permissions on the /var/run/zeppelin-notebook folder as well. You should then be able to restart the service in the Ambari UI.
Every time I run a Kivy app I see OSError (see it in last line of my given example). Even though my app runs successfully. What could be the cause of this error?
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/dist-packages/kivy/input/providers/mtdev.py", line 197, in _thread_run
_device = Device(_fn)
File "/usr/lib/python2.7/dist-packages/kivy/lib/mtdev.py", line 131, in __init__
self._fd = os.open(filename, os.O_NONBLOCK | os.O_RDONLY)
OSError: [Errno 13] Permission denied: '/dev/input/event5'
This error is not important, it just means that kivy checked the possible input providers in your OS and found that this one is forbidden.