Celery erroring out on runtime due to rabbitmq password rotation using Hashicorp Vault - rabbitmq

Question

Im trying to use rabbitmq as a broker for celery and using Hashicorp Vault for root credential rotation. How do I use vault's rabbitmq secret engine with celery?
def get_broker_url():
url = "http://localhost:8200/v1/rabbitmq/creds/dev-role"
payload = {}
headers = {
'X-Vault-Token': 'VAULT_TOKEN'
}
response = requests.get(url, headers=headers, data=payload).json()
return "amqp://{username}:{password}#127.0.0.1:5672/".format(username=response["data"]["username"], password=response["data"]["password"])
And while initialising celery
app = Celery(app_name, broker=get_broker_url(), backend="rpc://")
Im getting this error when lease expires
[2022-11-24 13:05:54,489: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 628, in start
c.loop(*c.loop_args())
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
cb(*cbargs)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/transport/base.py", line 235, in on_readable
reader(loop)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/transport/base.py", line 217, in _read
drain_events(timeout=0)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 525, in drain_events
while not self.blocking_read(timeout):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 531, in blocking_read
return self.on_inbound_frame(frame)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/method_framing.py", line 53, in on_frame
callback(channel, method_sig, buf, None)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 538, in on_inbound_method
method_sig, payload, content,
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
listener(*args)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 668, in _on_close
(class_id, method_id), ConnectionError)
amqp.exceptions.ConnectionForced: (0, 0): (320) CONNECTION_FORCED - user 'root-3a978f79-e2fb-2889-08d4-0c69d2bcbffd' is deleted
[2022-11-24 13:05:54,498: WARNING/MainProcess] /Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning:
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered back to the queue.
You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss setting.
In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.
warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
[2022-11-24 13:05:54,505: CRITICAL/MainProcess] Unrecoverable error: AccessRefused(403, 'ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.', (0, 0), '')
Traceback (most recent call last):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/connection.py", line 21, in start
c.connection = c.connect()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 428, in connect
conn = self.connection_for_read(heartbeat=self.amqheartbeat)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 435, in connection_for_read
self.app.connection_for_read(heartbeat=heartbeat))
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 462, in ensure_connected
callback=maybe_shutdown,
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 381, in ensure_connection
self._ensure_connection(*args, **kwargs)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 437, in _ensure_connection
callback, timeout=timeout
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
return fun(*args, **kwargs)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 877, in _connection_factory
self._connection = self._establish_connection()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 812, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
conn.connect()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 329, in connect
self.drain_events(timeout=self.connect_timeout)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 525, in drain_events
while not self.blocking_read(timeout):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 531, in blocking_read
return self.on_inbound_frame(frame)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/method_framing.py", line 53, in on_frame
callback(channel, method_sig, buf, None)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 538, in on_inbound_method
method_sig, payload, content,
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
listener(*args)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 668, in _on_close
(class_id, method_id), ConnectionError)
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
Since broker root user's lease expires based on the ttl, How to add new broker credential for celery to use?
celery version - 5.2.7