[ERROR]: training mobnetv2 with object_detection api and I get the error while training - tensorflow2.0

Question

I want to train my mobnetv2 to detect sign language. I get this error when i run the command to start the training process. I need help to settle this work. i am using tensorflowv2 2.9.1, python 3.11, CUDA 10.1, and cuDNN 7.6.5.
Use fn_output_signature instead
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
2022-11-23 00:44:54.177192: W tensorflow/core/framework/op_kernel.cc:1733] UNKNOWN: JIT compilation failed.
Traceback (most recent call last):
File "C:\SignLanguage\RealTimeObjectDetection\Tensorflow\models\research\object_detection\model_main_tf2.py", line 114, in <module>
tf.compat.v1.app.run()
File "C:\Users\messi\AppData\Roaming\Python\Python39\site-packages\tensorflow\python\platform\app.py", line 36, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\messi\anaconda3\lib\site-packages\absl\app.py", line 308, in run
_run_main(main, args)
File "C:\Users\messi\anaconda3\lib\site-packages\absl\app.py", line 254, in _run_main
sys.exit(main(argv))
File "C:\SignLanguage\RealTimeObjectDetection\Tensorflow\models\research\object_detection\model_main_tf2.py", line 105, in main
model_lib_v2.train_loop(
File "C:\Users\messi\anaconda3\lib\site-packages\object_detection\model_lib_v2.py", line 685, in train_loop
losses_dict = _dist_train_step(train_input_iter)
File "C:\Users\messi\AppData\Roaming\Python\Python39\site-packages\tensorflow\python\util\traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\messi\AppData\Roaming\Python\Python39\site-packages\tensorflow\python\eager\execute.py", line 54, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:
Detected at node 'train_input_images/write_summary/mod' defined at (most recent call last):
File "C:\Users\messi\anaconda3\lib\threading.py", line 937, in _bootstrap
self._bootstrap_inner()
File "C:\Users\messi\anaconda3\lib\threading.py", line 980, in _bootstrap_inner
self.run()
File "C:\Users\messi\anaconda3\lib\site-packages\object_detection\model_lib_v2.py", line 629, in train_step_fn
if record_summaries:
File "C:\Users\messi\anaconda3\lib\site-packages\object_detection\model_lib_v2.py", line 630, in train_step_fn
tf.compat.v2.summary.image(
File "C:\Users\messi\AppData\Roaming\Python\Python39\site-packages\tensorboard\plugins\image\summary_v2.py", line 141, in image
tag=tag, tensor=lazy_tensor, step=step, metadata=summary_metadata
File "C:\Users\messi\anaconda3\lib\site-packages\object_detection\model_lib_v2.py", line 599, in <lambda>
lambda: global_step % num_steps_per_iteration == 0):
Node: 'train_input_images/write_summary/mod'
Detected at node 'train_input_images/write_summary/mod' defined at (most recent call last):
File "C:\Users\messi\anaconda3\lib\threading.py", line 937, in _bootstrap
self._bootstrap_inner()
File "C:\Users\messi\anaconda3\lib\threading.py", line 980, in _bootstrap_inner
self.run()
File "C:\Users\messi\anaconda3\lib\site-packages\object_detection\model_lib_v2.py", line 629, in train_step_fn
if record_summaries:
File "C:\Users\messi\anaconda3\lib\site-packages\object_detection\model_lib_v2.py", line 630, in train_step_fn
tf.compat.v2.summary.image(
File "C:\Users\messi\AppData\Roaming\Python\Python39\site-packages\tensorboard\plugins\image\summary_v2.py", line 141, in image
tag=tag, tensor=lazy_tensor, step=step, metadata=summary_metadata
File "C:\Users\messi\anaconda3\lib\site-packages\object_detection\model_lib_v2.py", line 599, in <lambda>
lambda: global_step % num_steps_per_iteration == 0):
Node: 'train_input_images/write_summary/mod'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node train_input_images/write_summary/mod}}]]
[[while/body/_1/train_input_images/write_summary/Equal_1/_12]]
(1) UNKNOWN: JIT compilation failed.
[[{{node train_input_images/write_summary/mod}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference__dist_train_step_52302]