im trying to get netbox to get configs using NAPALM, but even the basic connection using napalm isnt working, always errors around the key file, which we need to use (cannot use password).
napalm==3.4.1
python=3.8
referred to the documentation at https://napalm.readthedocs.io/en/latest/support/ for use of arguments
I can directly SSH to the router using the key in ~/.ssh/ for the user netbox
netbox#netbox:~/.ssh$ ssh -c aes128-cbc netbox#172.1.1.1
The authenticity of host '172.1.1.1 (172.1.1.1)' can't be established.
RSA key fingerprint is SHA256:<fingerprinthere>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.1.1.1' (RSA) to the list of known hosts.
PHC-SW01>
Prompts for password and fails
napalm --user netbox --vendor ios --optional_args 'use_keys=true, secret=passwordhere' --debug 172.1.1.1 call get_facts
/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.18ubuntu0.18.04.1 is an invalid version and will not be supported in a future release
warnings.warn(
/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.18ubuntu0.18.04.1 is an invalid version and will not be supported in a future release
warnings.warn(
Enter password:
2022-08-24 13:03:01,926 - napalm - DEBUG - Starting napalm's debugging tool
2022-08-24 13:03:01,926 - napalm - DEBUG - Gathering napalm packages
2022-08-24 13:03:01,926 - napalm - DEBUG - napalm==3.4.1
2022-08-24 13:03:01,926 - napalm - DEBUG - get_network_driver - Calling with args: ('ios',), {}
2022-08-24 13:03:01,927 - napalm - DEBUG - get_network_driver - Successful
Traceback (most recent call last):
File "/usr/local/bin/napalm", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 308, in main
run_tests(args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 269, in run_tests
optional_args = helpers.parse_optional_args(args.optional_args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 131, in parse_optional_args
return {
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 132, in <dictcomp>
x.split("=")[0]: ast.literal_eval(x.split("=")[1])
File "/usr/lib/python3.8/ast.py", line 99, in literal_eval
return _convert(node_or_string)
File "/usr/lib/python3.8/ast.py", line 98, in _convert
return _convert_signed_num(node)
File "/usr/lib/python3.8/ast.py", line 75, in _convert_signed_num
return _convert_num(node)
File "/usr/lib/python3.8/ast.py", line 66, in _convert_num
_raise_malformed_node(node)
File "/usr/lib/python3.8/ast.py", line 63, in _raise_malformed_node
raise ValueError(f'malformed node or string: {node!r}')
ValueError: malformed node or string: <_ast.Name object at 0x7fa719a35af0>
napalm --user netbox --password '' --vendor ios --optional_args 'use_keys=true, secret=passwordhere' --debug 172.1.1.1 call get_facts
warnings.warn(
/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.18ubuntu0.18.04.1 is an invalid version and will not be supported in a future release
warnings.warn(
2022-08-24 13:05:13,330 - napalm - DEBUG - Starting napalm's debugging tool
2022-08-24 13:05:13,330 - napalm - DEBUG - Gathering napalm packages
2022-08-24 13:05:13,331 - napalm - DEBUG - napalm==3.4.1
2022-08-24 13:05:13,331 - napalm - DEBUG - get_network_driver - Calling with args: ('ios',), {}
2022-08-24 13:05:13,331 - napalm - DEBUG - get_network_driver - Successful
Traceback (most recent call last):
File "/usr/local/bin/napalm", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 308, in main
run_tests(args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 269, in run_tests
optional_args = helpers.parse_optional_args(args.optional_args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 131, in parse_optional_args
return {
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 132, in <dictcomp>
x.split("=")[0]: ast.literal_eval(x.split("=")[1])
File "/usr/lib/python3.8/ast.py", line 99, in literal_eval
return _convert(node_or_string)
File "/usr/lib/python3.8/ast.py", line 98, in _convert
return _convert_signed_num(node)
File "/usr/lib/python3.8/ast.py", line 75, in _convert_signed_num
return _convert_num(node)
File "/usr/lib/python3.8/ast.py", line 66, in _convert_num
_raise_malformed_node(node)
File "/usr/lib/python3.8/ast.py", line 63, in _raise_malformed_node
raise ValueError(f'malformed node or string: {node!r}')
ValueError: malformed node or string: <_ast.Name object at 0x7f86f4974b80>
napalm --user netbox --vendor ios --optional_args 'alt_host_keys=True, alt_key_file=/home/netbox/.ssh/id_rsa' --debug 172.1.1.1 call get_facts
Prompts for password, when entered gives below.. shouldnt need password when keyfile is used.
/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.18ubuntu0.18.04.1 is an invalid version and will not be supported in a future release
warnings.warn(
/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.18ubuntu0.18.04.1 is an invalid version and will not be supported in a future release
warnings.warn(
Enter password:
2022-08-24 13:14:59,741 - napalm - DEBUG - Starting napalm's debugging tool
2022-08-24 13:14:59,741 - napalm - DEBUG - Gathering napalm packages
2022-08-24 13:14:59,741 - napalm - DEBUG - napalm==3.4.1
2022-08-24 13:14:59,741 - napalm - DEBUG - get_network_driver - Calling with args: ('ios',), {}
2022-08-24 13:14:59,742 - napalm - DEBUG - get_network_driver - Successful
Traceback (most recent call last):
File "/usr/local/bin/napalm", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 308, in main
run_tests(args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 269, in run_tests
optional_args = helpers.parse_optional_args(args.optional_args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 131, in parse_optional_args
return {
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 132, in <dictcomp>
x.split("=")[0]: ast.literal_eval(x.split("=")[1])
File "/usr/lib/python3.8/ast.py", line 59, in literal_eval
node_or_string = parse(node_or_string, mode='eval')
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
File "<unknown>", line 1
/home/netbox/.ssh/id_rsa
^
SyntaxError: invalid syntax
napalm --user netbox --vendor ios --optional_args "hostkey_verify=False, look_for_keys=true, key_file=~/.ssh/id_rsa" --debug 172.1.1.1 call get_facts
/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.18ubuntu0.18.04.1 is an invalid version and will not be supported in a future release
warnings.warn(
/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.18ubuntu0.18.04.1 is an invalid version and will not be supported in a future release
warnings.warn(
Enter password:
2022-08-24 13:18:37,500 - napalm - DEBUG - Starting napalm's debugging tool
2022-08-24 13:18:37,501 - napalm - DEBUG - Gathering napalm packages
2022-08-24 13:18:37,501 - napalm - DEBUG - napalm==3.4.1
2022-08-24 13:18:37,501 - napalm - DEBUG - get_network_driver - Calling with args: ('ios',), {}
2022-08-24 13:18:37,501 - napalm - DEBUG - get_network_driver - Successful
Traceback (most recent call last):
File "/usr/local/bin/napalm", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 308, in main
run_tests(args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/cl_napalm.py", line 269, in run_tests
optional_args = helpers.parse_optional_args(args.optional_args)
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 131, in parse_optional_args
return {
File "/usr/local/lib/python3.8/dist-packages/napalm/base/clitools/helpers.py", line 132, in <dictcomp>
x.split("=")[0]: ast.literal_eval(x.split("=")[1])
File "/usr/lib/python3.8/ast.py", line 99, in literal_eval
return _convert(node_or_string)
File "/usr/lib/python3.8/ast.py", line 98, in _convert
return _convert_signed_num(node)
File "/usr/lib/python3.8/ast.py", line 75, in _convert_signed_num
return _convert_num(node)
File "/usr/lib/python3.8/ast.py", line 66, in _convert_num
_raise_malformed_node(node)
File "/usr/lib/python3.8/ast.py", line 63, in _raise_malformed_node
raise ValueError(f'malformed node or string: {node!r}')
ValueError: malformed node or string: <_ast.Name object at 0x7ff3a5c5e6d0>
If i specify --password '' it doesnt prompt for the password.
If i use "alt_key_file" or "key_file" always has the same error:
SyntaxError: invalid syntax
running napalm as user 'netbox' accessing the keyfile should work.
Update:
tried putting quotes within quotes re: https://github.com/napalm-automation/napalm-base/issues/309#issuecomment-461185980
but didnt work
--optional_args "hostkey_verify=False, look_for_keys=true, key_file='~/.ssh/id_rsa'"
by putting quotes within each variable i was able to get the command to work
napalm --user netbox --password '' --vendor ios --optional_args "alt_host_keys='True',alt_key_file='/home/netbox/.ssh/id_rsa'" --debug 172.16.1.1 call get_facts
its failing on the enable after connection, but its connecting using the SSH Key now which is what this was about.
Related
I'm trying to install numpy on a Virtual Environment using pip, there is a failure when building wheel for it, however.
The problem is only present when trying to install it in the virtualenv, I can install and update it on my system just fine.
It is running on Windows 10, Python 3.10.5, pip 22.3.1.
Running from numpy source directory.
setup.py:67: DeprecationWarning:
`numpy.distutils` is deprecated since NumPy 1.23.0, as a result
of the deprecation of `distutils` itself. It will be removed for
Python >= 3.12. For older Python versions it will remain present.
It is recommended to use `setuptools < 60.0` for those Python versions.
For more details, see:
https://numpy.org/devdocs/reference/distutils_status_migration.html
import numpy.distutils.command.sdist
Processing numpy/random\_bounded_integers.pxd.in
Processing numpy/random\bit_generator.pyx
Processing numpy/random\mtrand.pyx
Processing numpy/random\_bounded_integers.pyx.in
Processing numpy/random\_common.pyx
Processing numpy/random\_generator.pyx
Processing numpy/random\_mt19937.pyx
Processing numpy/random\_pcg64.pyx
Processing numpy/random\_philox.pyx
Processing numpy/random\_sfc64.pyx
Cythonizing sources
INFO: blas_opt_info:
INFO: blas_armpl_info:
Looking for python310.dll
Traceback (most recent call last):
File "C:\stable_diffusion\diffusers_venv\lib\python3.10\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 363, in <module>
main()
File "C:\stable_diffusion\diffusers_venv\lib\python3.10\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\stable_diffusion\diffusers_venv\lib\python3.10\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 261, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "C:\Users\Vitor\AppData\Local\Temp\pip-build-env-amr3d935\overlay\lib\python3.10\site-packages\setuptools\build_meta.py", line 230, in build_wheel
return self._build_with_temp_dir(['bdist_wheel'], '.whl',
File "C:\Users\Vitor\AppData\Local\Temp\pip-build-env-amr3d935\overlay\lib\python3.10\site-packages\setuptools\build_meta.py", line 215, in _build_with_temp_dir
self.run_setup()
File "C:\Users\Vitor\AppData\Local\Temp\pip-build-env-amr3d935\overlay\lib\python3.10\site-packages\setuptools\build_meta.py", line 267, in run_setup
super(_BuildMetaLegacyBackend,
File "C:\Users\Vitor\AppData\Local\Temp\pip-build-env-amr3d935\overlay\lib\python3.10\site-packages\setuptools\build_meta.py", line 158, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 479, in <module>
setup_package()
File "setup.py", line 471, in setup_package
setup(**metadata)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\core.py", line 135, in setup
config = configuration()
File "setup.py", line 118, in configuration
config.add_subpackage('numpy')
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\misc_util.py", line 1050, in add_subpackage
config_list = self.get_subpackage(subpackage_name, subpackage_path,
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\misc_util.py", line 1016, in get_subpackage
config = self._get_configuration_from_setup_py(
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\misc_util.py", line 958, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\setup.py", line 9, in configuration
config.add_subpackage('core')
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\misc_util.py", line 1050, in add_subpackage
config_list = self.get_subpackage(subpackage_name, subpackage_path,
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\misc_util.py", line 1016, in get_subpackage
config = self._get_configuration_from_setup_py(
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\misc_util.py", line 958, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\core\setup.py", line 853, in configuration
blas_info = get_info('blas_opt', 0)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 585, in get_info
return cl().get_info(notfound_action)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 845, in get_info
self.calc_info()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 2077, in calc_info
if self._calc_info(blas):
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 2063, in _calc_info
return getattr(self, '_calc_info_{}'.format(name))()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 1979, in _calc_info_armpl
info = get_info('blas_armpl')
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 585, in get_info
return cl().get_info(notfound_action)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 845, in get_info
self.calc_info()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 1337, in calc_info
info = self.check_libs2(lib_dirs, armpl_libs)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 1001, in check_libs2
exts = self.library_extensions()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 960, in library_extensions
c = customized_ccompiler()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\system_info.py", line 216, in customized_ccompiler
global_compiler = _customized_ccompiler()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\__init__.py", line 62, in customized_ccompiler
c = ccompiler.new_compiler(plat=plat, compiler=compiler, verbose=verbose)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\ccompiler.py", line 780, in new_compiler
compiler = klass(None, dry_run, force)
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\mingw32ccompiler.py", line 64, in __init__
build_import_library()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\mingw32ccompiler.py", line 348, in build_import_library
return _build_import_library_amd64()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\mingw32ccompiler.py", line 397, in _build_import_library_amd64
dll_file = find_python_dll()
File "C:\Users\Vitor\AppData\Local\Temp\pip-install-1sr3obr4\numpy_fa19a5e39e9448e7b6553e0f5a275159\numpy\distutils\mingw32ccompiler.py", line 219, in find_python_dll
raise ValueError("%s not found in %s" % (dllname, lib_dirs))
ValueError: python310.dll not found in ['C:\\stable_diffusion\\diffusers_venv\\', 'C:\\stable_diffusion\\diffusers_venv\\lib', 'C:\\stable_diffusion\\diffusers_venv\\bin', 'C:\\Program Files\\Inkscape\\', 'C:\\Program Files\\Inkscape\\lib', 'C:\\Program Files\\Inkscape\\bin', 'C:\\WINDOWS\\System32']
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects
As I faced a similar problem before, my instinct was to update pip, change the running python version or try an older, specific, version of numpy, but nothing seems to work and the problem persist.
My tests pass locally and in fact on Github Actions it also says "ran 8 tests" and then "OK" (and I have 8). However, the test stage fails due to a strange error in the traceback.
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 421, in execute
return Database.Cursor.execute(self, query)
sqlite3.OperationalError: near "SCHEMA": syntax error
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/runner/work/store/store/manage.py", line 22, in <module>
main()
File "/home/runner/work/store/store/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
super().run_from_argv(argv)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/core/management/commands/test.py", line 55, in handle
failures = test_runner.run_tests(test_labels)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/test/runner.py", line 736, in run_tests
self.teardown_databases(old_config)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django_heroku/core.py", line 41, in teardown_databases
self._wipe_tables(connection)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django_heroku/core.py", line 26, in _wipe_tables
cursor.execute(
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 421, in execute
return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: near "SCHEMA": syntax error
Error: Process completed with exit code 1.
These are all just default Django files and I haven't messed with any of them. I don't really know what to do about it and internet searches yield nothing helpful.
Just had the same issue. For me I had django_heroku in my project settings.py. It looks like, based on the comments that your app is using this.
I had something like this in settings.py:
import django_heroku
django_heroku.settings(locals())
change to the following so it doesn't use django_heroku on github actions:
if os.environ.get('ENVIRONMENT') != 'github':
import django_heroku
django_heroku.settings(locals())
and then declare an environment variable in the workflow file, I have this:
jobs:
build:
runs-on: ubuntu-latest
strategy:
max-parallel: 4
matrix:
python-version: [3.7, 3.8, 3.9, '3.10']
env:
DJANGO_SECRET_KEY: "someKey"
ENVIRONMENT: github
Note: this is assuming you have environment variables setup. I know there are different ways of doing it, so depending on how they are setup you might have to change how they are accessed
more info here
edit:
As per the comments, you don't have to declare an additional environment variable because github has a default variable called GITHUB_ACTIONS
so you can do
if os.environ.get('GITHUB_ACTIONS') != 'true':
import django_heroku
django_heroku.settings(locals())
I tried:
- configuring s3 connection from the value.yaml:
connections:
id: aws_default
type: aws
login: xxxaws_access_key_idxxx
password: xxxxxxxxxxxxxaws_secret_access_keyxxxxxxxxxxxxxxxxxx
id: my_s3
type: s3
login: xxxaws_access_key_idxxx
password: xxxxxxxxxxxxxaws_secret_access_keyxxxxxxxxxxxxxxxxxx
write DAG that uses the s3Hook and write string to s3.
run test from scheduler pod:
/entrypoint airflow test dag_id task_id date_before_the_start_data_of_DAG
the file is created and content is ok
# Airflow UI activating the
DAG and RUN it is queued and fail.
any suggestions?
BTW DAG ARGS:
args = {
'owner': 'airflow',
'start_date': airflow.utils.dates.days_ago(0),
'trigger_rule': 'dummy',
#'pool': 'my_workers_pool',
'catchup': False, # dont fill back
}
in addition I added 5 min sleep to the task that cause the DAG to fail and watched the pod creation at the kubectl but the task started for few seconds and disappears. any Ideas how to debug this issue?
logs from task pod:
kubectl logs postgressexamplesababaegozimpostgress-7322d44cb2684a09bef95ad3080b9505
-n airflow-research-p-8482f --tail=200 [2020-08-24 12:04:44,262] {{settings.py:252}} INFO - settings.configure_orm(): Using pool
settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1
/usr/local/lib/python3.7/site-packages/psycopg2/init.py:144:
UserWarning: The psycopg2 wheel package will be renamed from release
2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see:
http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.
""") [2020-08-24 12:04:44,833] {{init.py:51}} INFO - Using
executor LocalExecutor [2020-08-24 12:04:44,834] {{dagbag.py:92}} INFO
Filling up the DagBag from /usr/local/airflow/dags/postgress_example.py Traceback (most recent
call last): File "/usr/local/bin/airflow", line 37, in
[2020-08-24 12:04:44,841] {{dagbag.py:207}} ERROR - Failed to import:
/usr/local/airflow/dags/postgress_example.py Traceback (most recent
call last): File
"/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py",
line 204, in process_file
m = imp.load_source(mod_name, filepath) File "/usr/local/lib/python3.7/imp.py", line 171, in load_source
module = _load(spec) File "", line 696, in _load File "", line 677, in
_load_unlocked File "", line 728, in exec_module File "", line 219,
in _call_with_frames_removed File
"/usr/local/airflow/dags/postgress_example.py", line 33, in
import boto3 ModuleNotFoundError: No module named 'boto3'
args.func(args) File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line
74, in wrapper
return f(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 529,
in run
dag = get_dag(args) File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 148,
in get_dag
'parse.'.format(args.dag_id)) airflow.exceptions.AirflowException: dag_id could not be found: postgress_example. Either the dag did not
exist or it failed to parse.
Getting the follow running the example from the docs. Anyone have ideas why?
Looks like gprc. Maybe this is related? https://github.com/grpc/grpc/issues/15933
docker run --network host -e FLYTE_PLATFORM_URL='127.0.0.1:30081' lyft/flytesnacks:v0.1.0 pyflyte -p flytesnacks -d development -c sandbox.config register workflows
Using configuration file at /app/sandbox.config
Flyte Admin URL 127.0.0.1:30081
Running task, workflow, and launch plan registration for flytesnacks, development, ['workflows'] with version 46045e6383611da1cb763d64d846508806fce1a4
Registering Task: workflows.edges.edge_detection_canny
Traceback (most recent call last):
File "/app/venv/bin/pyflyte", line 11, in <module>
sys.exit(main())
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/flytekit/clis/sdk_in_container/register.py", line 86, in workflows
register_all(project, domain, pkgs, test, version)
File "/app/venv/lib/python3.6/site-packages/flytekit/clis/sdk_in_container/register.py", line 24, in register_all
o.register(project, domain, name, version)
File "/app/venv/lib/python3.6/site-packages/flytekit/common/exceptions/scopes.py", line 158, in system_entry_point
return wrapped(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/flytekit/common/tasks/task.py", line 141, in register
_engine_loader.get_engine().get_task(self).register(id_to_register)
File "/app/venv/lib/python3.6/site-packages/flytekit/engines/flyte/engine.py", line 234, in register
self.sdk_task
File "/app/venv/lib/python3.6/site-packages/flytekit/clients/friendly.py", line 50, in create_task
spec=task_spec.to_flyte_idl()
File "/app/venv/lib/python3.6/site-packages/flytekit/clients/raw.py", line 12, in handler
return fn(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/flytekit/clients/raw.py", line 77, in create_task
return self._stub.CreateTask(task_create_request)
File "/app/venv/lib/python3.6/site-packages/grpc/_channel.py", line 604, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/app/venv/lib/python3.6/site-packages/grpc/_channel.py", line 506, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"#1578482148.715995870","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3876,"referenced_errors":[{"created":"#1578482148.715993301","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":395,"grpc_status":14}]}"
I think the problem is in FLYTE_PLATFORM_URL='127.0.0.1:30081'.Minikube runs in VM on OSX.
When you run docker run --network host, the docker daemon shares the networking namespace with this host - https://docs.docker.com/network/host/
Thus if you have the docker daemon and Kubernetes running on the same host (not a VM), you should be able to refer to services (nodeport) or ingress on Kubernetes using 127.0.0.1.
But, since minikube runs in a VM on osx, you will need to use minikube ip to find the ip address of the minikube machine and then set
FLYTE_PLATFORM_URL=':30081'
I am installing a complete new cluster using HDP 2.3
When Ambari tries to install the services (HDFS Client or Namenode) it errors out. below is the stderr
stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py", line 35, in <module> BeforeAnyHook().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 220, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py", line 26, in hook import params File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py", line 187, in <module> hadoop_conf_dir = conf_select.get_hadoop_conf_dir(force_latest_on_upgrade=True) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/conf_select.py", line 374, in get_hadoop_conf_dir select(stack_name, "hadoop", version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/conf_select.py", line 252, in select shell.checked_call(get_cmd("set-conf-dir", package, version), logoutput=False, quiet=False, sudo=True) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages**/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0' returned 1. /usr/hdp/2.3.4.0-3485/hadoop/conf does not exist Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-302.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-302.json', 'INFO', '/var/lib/ambari-agent/tmp']**
stdout: 2016-02-26 23:04:51,884 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485 2016-02-26 23:04:51,885 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0 2016-02-26 23:04:51,885 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-02-26 23:04:51,928 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '') 2016-02-26 23:04:51,929 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False} Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-302.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-302.json', 'INFO', '/var/lib/ambari-agent/tmp']
The answer is in the error log, highlighting it for you:
Execution of 'conf-select set-conf-dir --package hadoop
--stack-version 2.3.4.0-3485 --conf-version 0' returned 1
/usr/hdp/2.3.4.0-3485/hadoop/conf does not exist
Fix your configuration and it will start to work.