OpenStack Cinder replication with Ceph - replication

Question

I set up two clusters ceph (version 12.2.9 luminous). The first cluster has the name of the "primary", the second "secondary". Two-way replication is configured between the two clusters using an RBD mirror. Images are created and replicated successfully. I want to configure Volume Replication in Cinder. I'm having trouble configuring Сinder. Help me please.
My settings:
file /etc/cinder/cinder.conf
.....
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
replication_device = backend_id:secondary,conf:/etc /ceph/secondary.conf,user:cinder,pool:volumes
rbd_cluster_name = primary
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/primary.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
volume_backend_name = ceph
rbd_user = cinder
rbd_secret_uuid = 5e54739d-37cf-49e8-93f5-5489f51d8ef0
List /etc/ceph/
$ ls /etc/ceph
primary.client.cinder.keyring primary.client.glance.keyring primary.conf secondary.client.cinder.keyring secondary.client.glance.keyring secondary.conf
Cinder Type Configuration
$ cinder type-create replicated
+--------------------------------------+------------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+------------+-------------+-----------+
| f839f331-4af0-4626-b377-e9baa9789c9d | replicated | - | True |
+--------------------------------------+------------+-------------+-----------+
$ cinder type-key replicated set volume_backend_name=ceph
$ cinder type-key replicated set replication_enabled='<is> True'
$ cinder extra-specs-list
+--------------------------------------+------------+---------------------------------------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+------------+---------------------------------------------------------------------+
| 1fef1cf4-3cbc-4f8a-9f06-ef07444c2570 | type2 | {} |
| 709a93bc-b8fb-4b45-8af4-d4422d10fc73 | type1 | {} |
| f839f331-4af0-4626-b377-e9baa9789c9d | replicated | {'replication_enabled': '<is> True', 'volume_backend_name': 'ceph'} |
+--------------------------------------+------------+---------------------------------------------------------------------+
Replicated Volume Creation
$ cinder create --volume-type replicated --name fingers-crossed 1
$ cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+
| e2ed712d-fd47-4c5b-ae97-75e5834120e4 | error | fingers-crossed | 1 | replicated | false | |
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+
volume log
$ cat /var/log/cinder/volume.log
.......
2018-11-08 13:34:47.369 2348 WARNING cinder.volume.manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62ad1df9-9f38-4e4a-af1d-688f8fd793b1) transitioned into state 'FAILURE' from state 'RUNNING'
5 predecessors (most recent first):
Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-11-08T10:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='fingers-crossed',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='server.loc#ceph#ceph',id=e2ed712d-fd47-4c5b-ae97-75e5834120e4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e92f59855c986',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-11-08T10:34:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-11-08T10:34:46Z,user_id='e92f563fa9',volume_attachment=<?>,volume_type=VolumeType(f839f331-4af0-4626-b377-e9baa9789c9d),volume_type_id=f839f331-4af0-4626-b377-e9baa9789c9d), 'context': <cinder.context.RequestContext object at 0x7fc0fe180390>}, 'provides': None}
|__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-11-08T10:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='fingers-crossed',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='server.loc#ceph#ceph',id=e2ed712d-fd47-4c5b-ae97-75e5834120e4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e92f59855c986',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-11-08T10:34:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-11-08T10:34:46Z,user_id='e92f563fa9',volume_attachment=<?>,volume_type=VolumeType(f839f331-4af0-4626-b377-e9baa9789c9d),volume_type_id=f839f331-4af0-4626-b377-e9baa9789c9d), 'request_spec': RequestSpec(CG_backend=<?>,backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(e2ed712d-fd47-4c5b-ae97-75e5834120e4),volume_id=e2ed712d-fd47-4c5b-ae97-75e5834120e4,volume_properties=VolumeProperties,volume_type=VolumeType(f839f331-4af0-4626-b377-e9baa9789c9d)), 'context': <cinder.context.RequestContext object at 0x7fc0fe180390>}, 'provides': {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-e2ed712d-fd47-4c5b-ae97-75e5834120e4', 'type': 'raw', 'volume_id': u'e2ed712d-fd47-4c5b-ae97-75e5834120e4'}}
|__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-11-08T10:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='fingers-crossed',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='server.loc#ceph#ceph',id=e2ed712d-fd47-4c5b-ae97-75e5834120e4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e92f59855c986',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-11-08T10:34:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-11-08T10:34:46Z,user_id='e92f563fa9',volume_attachment=<?>,volume_type=VolumeType(f839f331-4af0-4626-b377-e9baa9789c9d),volume_type_id=f839f331-4af0-4626-b377-e9baa9789c9d), 'filter_properties': {u'config_options': {}, u'request_spec': {u'backup_id': None, u'volume_properties': {u'status': u'creating', u'volume_type_id': u'f839f331-4af0-4626-b377-e9baa9789c9d', u'project_id': u'e92f59855c986', u'user_id': u'e92f563fa9', u'availability_zone': u'nova', u'reservations': [u'c2f86481-cc7f-47e2-91f5-6081cf4fc75d', u'e58874c0-7e52-4af2-9b92-1f14b27ceb1e', u'f30eee62-9f37-4d01-9459-5e61255dfef1', u'f2e6d81d-4728-42e3-87cc-df2687947cab'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'display_name': u'fingers-crossed', u'group_id': None, u'consistencygroup_id': None, u'size': 1}, u'source_volid': None, u'cgsnapshot_id': None, u'volume': {u'migration_status': None, u'provider_id': None, u'availability_zone': u'nova', u'terminated_at': None, u'updated_at': None, u'provider_geometry': None, u'replication_extended_status': None, u'replication_status': None, u'snapshot_id': None, u'ec2_id': None, u'deleted_at': None, u'id': u'e2ed712d-fd47-4c5b-ae97-75e5834120e4', u'size': 1, u'display_name': u'fingers-crossed', u'display_description': None, u'cluster_name': None, u'name_id': u'e2ed712d-fd47-4c5b-ae97-75e5834120e4', u'volume_admin_metadata': [], u'project_id': u'e92f59855c986', u'launched_at': None, u'scheduled_at': None, u'status': u'creating', u'volume_type_id': u'f839f331-4af0-4626-b377-e9baa9789c9d', u'multiattach': False, u'deleted': False, u'service_uuid': None, u'provider_location': None, u'volume_glance_metadata': [], u'admin_metadata': {}, u'host': None, u'glance_metadata': {}, u'consistencygroup_id': None, u'source_volid': None, u'provider_auth': None, u'previous_status': None, u'group_id': None, u'name': u'volume-e2ed712d-fd47-4c5b-ae97-75e5834120e4', u'user_id': u'e92f563fa9', u'bootable': False, u'shared_targets': True, u'attach_status': u'detached', u'volume_metadata': [], u'_name_id': None, u'encryption_key_id': None, u'replication_driver_data': None, u'metadata': {}, u'created_at': u'2018-11-08T10:34:44.000000'}, u'image_id': None, u'snapshot_id': None, u'consistencygroup_id': None, u'volume_type': {u'name': u'replicated', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2018-11-08T10:33:34.000000', u'updated_at': None, u'extra_specs': {u'replication_enabled': u'<is> True', u'volume_backend_name': u'ceph'}, u'is_public': True, u'deleted_at': None, u'id': u'f839f331-4af0-4626-b377-e9baa9789c9d', u'projects': [], u'description': None}, u'volume_id': u'e2ed712d-fd47-4c5b-ae97-75e5834120e4', u'resource_properties': {u'status': u'creating', u'volume_type_id': u'f839f331-4af0-4626-b377-e9baa9789c9d', u'project_id': u'e92f59855c986', u'user_id': u'e92f563fa9', u'availability_zone': u'nova', u'reservations': [u'c2f86481-cc7f-47e2-91f5-6081cf4fc75d', u'e58874c0-7e52-4af2-9b92-1f14b27ceb1e', u'f30eee62-9f37-4d01-9459-5e61255dfef1', u'f2e6d81d-4728-42e3-87cc-df2687947cab'], u'multiattach': False, u'attach_status': u'detached', u'source_volid': None, u'cgsnapshot_id': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'display_description': None, u'snapshot_id': None, u'display_name': u'fingers-crossed', u'group_id': None, u'consistencygroup_id': None, u'size': 1}, u'group_id': None}, u'user_id': u'e92f563fa9', u'availability_zone': u'nova', u'volume_type': VolumeType(created_at=2018-11-08T10:33:34Z,deleted=False,deleted_at=None,description=None,extra_specs={replication_enabled='<is> True',volume_backend_name='ceph'},id=f839f331-4af0-4626-b377-e9baa9789c9d,is_public=True,name='replicated',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), u'qos_specs': None, u'retry': {u'num_attempts': 3, u'backends': [u'server.loc#ceph#ceph', u'server.loc#ceph#ceph', u'server.loc#ceph#ceph'], u'hosts': [u'server.loc#ceph#ceph', u'server.loc#ceph#ceph', u'server.loc#ceph#ceph']}, u'metadata': {}, u'resource_type': VolumeType(created_at=2018-11-08T10:33:34Z,deleted=False,deleted_at=None,description=None,extra_specs={replication_enabled='<is> True',volume_backend_name='ceph'},id=f839f331-4af0-4626-b377-e9baa9789c9d,is_public=True,name='replicated',projects=[],qos_specs=<?>,qos_specs_id=None,updated_at=None), u'size': 1}, 'context': <cinder.context.RequestContext object at 0x7fc0fe180390>, 'request_spec': RequestSpec(CG_backend=<?>,backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=<?>,group_id=None,image_id=None,resource_backend=<?>,snapshot_id=None,source_replicaid=<?>,source_volid=None,volume=Volume(e2ed712d-fd47-4c5b-ae97-75e5834120e4),volume_id=e2ed712d-fd47-4c5b-ae97-75e5834120e4,volume_properties=VolumeProperties,volume_type=VolumeType(f839f331-4af0-4626-b377-e9baa9789c9d))}, 'provides': None}
|__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-11-08T10:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='fingers-crossed',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='server.loc#ceph#ceph',id=e2ed712d-fd47-4c5b-ae97-75e5834120e4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e92f59855c986',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-11-08T10:34:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-11-08T10:34:46Z,user_id='e92f563fa9',volume_attachment=<?>,volume_type=VolumeType(f839f331-4af0-4626-b377-e9baa9789c9d),volume_type_id=f839f331-4af0-4626-b377-e9baa9789c9d), 'context': <cinder.context.RequestContext object at 0x7fc0fe180390>}, 'provides': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2018-11-08T10:34:44Z,deleted=False,deleted_at=None,display_description=None,display_name='fingers-crossed',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='server.loc#ceph#ceph',id=e2ed712d-fd47-4c5b-ae97-75e5834120e4,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='e92f59855c986',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2018-11-08T10:34:46Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2018-11-08T10:34:46Z,user_id='e92f563fa9',volume_attachment=<?>,volume_type=VolumeType(f839f331-4af0-4626-b377-e9baa9789c9d),volume_type_id=f839f331-4af0-4626-b377-e9baa9789c9d)}
|__Flow 'volume_create_manager': ReplicationError: \u041e\u0448\u0438\u0431\u043a\u0430 \u0440\u0435\u043f\u043b\u0438\u043a\u0430\u0446\u0438\u0438 \u0442\u043e\u043c\u0430 e2ed712d-fd47-4c5b-ae97-75e5834120e4: Failed to enable image replication
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager Traceback (most recent call last):
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager result = task.execute(**arguments)
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1011, in execute
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager model_update = self._create_raw_volume(volume, **volume_spec)
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_raw_volume
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager ret = self.driver.create_volume(volume)
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 795, in create_volume
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager volume_id=volume.id)
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager ReplicationError: \u041e\u0448\u0438\u0431\u043a\u0430 \u0440\u0435\u043f\u043b\u0438\u043a\u0430\u0446\u0438\u0438 \u0442\u043e\u043c\u0430 e2ed712d-fd47-4c5b-ae97-75e5834120e4: Failed to enable image replication
2018-11-08 13:34:47.369 2348 ERROR cinder.volume.manager
2018-11-08 13:34:47.376 2348 WARNING cinder.volume.manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (62ad1df9-9f38-4e4a-af1d-688f8fd793b1) transitioned into state 'REVERTED' from state 'REVERTING'
2018-11-08 13:34:47.379 2348 WARNING cinder.volume.manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (df155d8f-7861-4859-b3e0-927388c786d9) transitioned into state 'REVERTED' from state 'REVERTING'
2018-11-08 13:34:47.382 2348 WARNING cinder.volume.manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (1d6823ac-a749-4af8-be3b-4bab16f402b4) transitioned into state 'REVERTED' from state 'REVERTING'
2018-11-08 13:34:47.485 2348 INFO cinder.volume.drivers.rbd [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] volume volume-e2ed712d-fd47-4c5b-ae97-75e5834120e4 no longer exists in backend
2018-11-08 13:34:47.491 2348 WARNING cinder.volume.manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dbf019d7-c936-4c03-a5da-32800d91f80e) transitioned into state 'REVERTED' from state 'REVERTING'
2018-11-08 13:34:47.494 2348 WARNING cinder.volume.manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (c4b82318-880b-4d64-912e-3786e4b1784f) transitioned into state 'REVERTED' from state 'REVERTING'
2018-11-08 13:34:47.496 2348 WARNING cinder.volume.manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Flow 'volume_create_manager' (756bc982-2823-4d78-a803-b680e419ce01) transitioned into state 'REVERTED' from state 'RUNNING'
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Exception during message handling: ReplicationError: \u041e\u0448\u0438\u0431\u043a\u0430 \u0440\u0435\u043f\u043b\u0438\u043a\u0430\u0446\u0438\u0438 \u0442\u043e\u043c\u0430 e2ed712d-fd47-4c5b-ae97-75e5834120e4: Failed to enable image replication
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_volume
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 681, in create_volume
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server _run_flow()
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 673, in _run_flow
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server flow_engine.run()
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server failures[0].reraise()
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1011, in execute
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume(volume, **volume_spec)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_raw_volume
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 795, in create_volume
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server volume_id=volume.id)
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server ReplicationError: \u041e\u0448\u04 e2ed712d-fd47-4c5b-ae97-75e5834120e4: Failed to enable image replication
2018-11-08 13:34:47.497 2348 ERROR oslo_messaging.rpc.server
scheduler log
$ cat scheduler.log
2018-11-08 13:34:45.991 889 ERROR cinder.scheduler.filter_scheduler [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Error scheduling None from last vol-service: server.loc#ceph#ceph : [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n result = task.execute(**arguments)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1011, in execute\n model_update = self._create_raw_volume(volume, **volume_spec)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_raw_volume\n ret = self.driver.create_volume(volume)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 795, in create_volume\n volume_id=volume.id)\n', u'ReplicationError: \\u043b\\u0438\\u043a\\u0430\\u0446\\u0438\\u0438 \\u0442\\u043e\\u043c\\u0430 e2ed712d-fd47-4c5b-ae97-75e5834120e4: Failed to enable image replication\n']
2018-11-08 13:34:46.006 889 WARNING cinder.scheduler.host_manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] volume service is down. (host: server.loc#lvm)
2018-11-08 13:34:46.006 889 WARNING cinder.scheduler.host_manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] volume service is down. (host: server.loc#lvm2)
2018-11-08 13:34:46.007 889 WARNING cinder.scheduler.host_manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] volume service is down. (host: server.loc#ceph_2)
2018-11-08 13:34:46.680 889 ERROR cinder.scheduler.filter_scheduler [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Error scheduling None from last vol-service: server.loc#ceph#ceph : [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n result = task.execute(**arguments)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1011, in execute\n model_update = self._create_raw_volume(volume, **volume_spec)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_raw_volume\n ret = self.driver.create_volume(volume)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 795, in create_volume\n volume_id=volume.id)\n', u'ReplicationError: \\u041e\\u0448\\u0438\\u0431\\u043a\\u0430 \\u0440\\u0435\\u043f\\u043b\\u0438\\u043a\\u0430\\u0446\\u0438\\u0438 \\u0442\\u043e\\u043c\\u0430 e2ed712d-fd47-4c5b-ae97-75e5834120e4: Failed to enable image replication\n']
2018-11-08 13:34:46.700 889 WARNING cinder.scheduler.host_manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] volume service is down. (host: server.loc#lvm)
2018-11-08 13:34:46.701 889 WARNING cinder.scheduler.host_manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] volume service is down. (host: server.loc#lvm2)
2018-11-08 13:34:46.701 889 WARNING cinder.scheduler.host_manager [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] volume service is down. (host: server.loc#ceph_2)
2018-11-08 13:34:47.490 889 ERROR cinder.scheduler.filter_scheduler [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Error scheduling None from last vol-service: server.loc#ceph#ceph : [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n result = task.execute(**arguments)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1011, in execute\n model_update = self._create_raw_volume(volume, **volume_spec)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_raw_volume\n ret = self.driver.create_volume(volume)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 795, in create_volume\n volume_id=volume.id)\n', u'ReplicationError: \\u041e\\u0448\\u0438\\u0431\\u043a\\u0430 \\u0440\\u0435\\u043f\\u043b\\u0438\\u043a\\u043 e2ed712d-fd47-4c5b-ae97-75e5834120e4: Failed to enable image replication\n']
2018-11-08 13:34:47.491 889 INFO cinder.message.api [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Creating message record for request_id = req-7f374fa5-7d95-4dae-81c1-81b839beb083
2018-11-08 13:34:47.545 889 ERROR cinder.scheduler.flows.create_volume [req-7f374fa5-7d95-4dae-81c1-81b839beb083 e92f563fa9 e92f59855c986 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded maximum number of scheduling attempts 3 for volume None: NoValidBackend: No valid backend was found. \u041f\u0440e \u043c\u043d\u043e\u0435