RabbitMQ Shovel failed to connect - rabbitmq

Question

I have 2 rabbitmq box named : centos (192.168.1.115) and devserver (192.168.1.126)
in 'centos' I have :
I have queue named : toshovel bound to a topic exchange with routing key '#'
I test posting to the exchange and messages transfered to that queue.
In 'devserver' I have :
1. topic exchange named bino.topic
2. queue named : bino.nms.idc3d bound to bino.topic
This also tested. including using pika to publish message from 'centos' to 'devserver' so that I'm sure there is no firewall nor permition nor authentication (user/password : esx/esx) problem
Now I want to shovel from 'centos' to 'devserver'
I tried adding shovel in 'centos' per https://www.rabbitmq.com/shovel-dynamic.html
rabbitmqctl set_parameter shovel my-shovel '{"src-protocol": "amqp091", "src-uri": "amqp://esx:esx#192.168.1.115", "src-queue": "toshovel", "dest-protocol": "amqp091", "dest-uri": "amqp://esx:esx#192.168.126/", "dest-queue": "bino.nms.idc3d"}'
but the centos log said
from : /var/log/rabbitmq/rabbit\#centos.log
2018-06-20 14:03:21.800 [info] <0.735.0> terminating static worker with {timeout,{gen_server,call,[<0.763.0>,connect,60000]}}
2018-06-20 14:03:21.800 [error] <0.735.0> ** Generic server <0.735.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,{<<"/">>,<<"my-shovel">>},dynamic,#{ack_mode => on_confirm,dest => #{dest_queue => <<"bino.nms.idc3d">>,fields_fun => #Fun<rabbit_shovel_parameters.11.26683091>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_shovel_parameters.12.26683091>,resource_decl => #Fun<rabbit_shovel_parameters.10.26683091>,uris => ["amqp://esx:esx#192.168.126/"]},name => <<"my-shovel">>,reconnect_delay => 5,shovel_type => dynamic,source => #{delete_after => never,module => rabbit_amqp091_shovel,prefetch_count => 1000,queue => <<"toshovel">>,resource_decl => #Fun<rabbit_shovel_parameters.14.26683091>,source_exchange_key => <<>>,uris => ["amqp://esx:esx#192.168.1.115"]}},undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {timeout,{gen_server,call,[<0.763.0>,connect,60000]}}
2018-06-20 14:03:21.800 [warning] <0.743.0> closing AMQP connection <0.743.0> (192.168.1.115:48223 -> 192.168.1.115:5672 - Shovel my-shovel, vhost: '/', user: 'esx'):
client unexpectedly closed TCP connection
2018-06-20 14:03:21.800 [error] <0.735.0> CRASH REPORT Process <0.735.0> with 1 neighbours exited with reason: {timeout,{gen_server,call,[<0.763.0>,connect,60000]}} in gen_server2:terminate/3 line 1166
2018-06-20 14:03:21.801 [error] <0.410.0> Supervisor {<0.410.0>,rabbit_shovel_dyn_worker_sup} had child {<<"/">>,<<"my-shovel">>} started with rabbit_shovel_worker:start_link(dynamic, {<<"/">>,<<"my-shovel">>}, [{<<"dest-protocol">>,<<"amqp091">>},{<<"dest-queue">>,<<"bino.nms.idc3d">>},{<<"dest-uri">>,<<"a...">>},...]) at <0.735.0> exit with reason {timeout,{gen_server,call,[<0.763.0>,connect,60000]}} in context child_terminated
2018-06-20 14:03:21.802 [error] <0.738.0> ** Generic server <0.738.0> terminating
** Last message in was {'EXIT',<0.735.0>,{timeout,{gen_server,call,[<0.763.0>,connect,60000]}}}
** When Server state == {state,amqp_network_connection,{state,#Port<0.29325>,<<"client 192.168.1.115:48223 -> 192.168.1.115:5672">>,10,<0.744.0>,131072,<0.737.0>,undefined,false},<0.742.0>,{amqp_params_network,<<"esx">>,<<"esx">>,<<"/">>,"192.168.1.115",5672,2047,0,10,60000,none,[#Fun<amqp_uri.12.90191702>,#Fun<amqp_uri.12.90191702>],[{<<"connection_name">>,longstr,<<"Shovel my-shovel">>}],[]},2047,[{<<"capabilities">>,table,[{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,bool,true},{<<"consumer_cancel_notify">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"consumer_priorities">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"per_consumer_qos">>,bool,true},{<<"direct_reply_to">>,bool,true}]},{<<"cluster_name">>,longstr,<<"rabbit#centos">>},{<<"copyright">>,longstr,<<"Copyright (C) 2007-2018 Pivotal Software, Inc.">>},{<<"information">>,longstr,<<"Licensed under the MPL. See http://www.rabbitmq.com/">>},{<<"platform">>,longstr,<<"Erlang/OTP 20.3.4">>},{<<"product">>,longstr,<<"RabbitMQ">>},{<<"version">>,longstr,<<"3.7.5">>}],none,false}
** Reason for termination ==
** "stopping because dependent process <0.735.0> died: {timeout,\n {gen_server,call,\n [<0.763.0>,connect,\n 60000]}}"
2018-06-20 14:03:21.802 [error] <0.738.0> CRASH REPORT Process <0.738.0> with 0 neighbours exited with reason: "stopping because dependent process <0.735.0> died: {timeout,\n {gen_server,call,\n [<0.763.0>,connect,\n 60000]}}" in gen_server:handle_common_reply/8 line 726
2018-06-20 14:03:21.802 [error] <0.752.0> Supervisor {<0.752.0>,amqp_channel_sup} had child channel started with amqp_channel:start_link(network, <0.738.0>, 1, <0.753.0>, {<<"client 192.168.1.115:48223 -> 192.168.1.115:5672">>,1}) at <0.755.0> exit with reason {timeout,{gen_server,call,[<0.763.0>,connect,60000]}} in context child_terminated
2018-06-20 14:03:21.802 [error] <0.752.0> Supervisor {<0.752.0>,amqp_channel_sup} had child channel started with amqp_channel:start_link(network, <0.738.0>, 1, <0.753.0>, {<<"client 192.168.1.115:48223 -> 192.168.1.115:5672">>,1}) at <0.755.0> exit with reason reached_max_restart_intensity in context shutdown
2018-06-20 14:03:21.803 [error] <0.736.0> Supervisor {<0.736.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.737.0>, {amqp_params_network,<<"esx">>,<<"esx">>,<<"/">>,"192.168.1.115",5672,2047,0,10,60000,none,[#Fun<am..>,...],...}) at <0.738.0> exit with reason "stopping because dependent process <0.735.0> died: {timeout,\n {gen_server,call,\n [<0.763.0>,connect,\n 60000]}}" in context child_terminated
2018-06-20 14:03:21.803 [error] <0.736.0> Supervisor {<0.736.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.737.0>, {amqp_params_network,<<"esx">>,<<"esx">>,<<"/">>,"192.168.1.115",5672,2047,0,10,60000,none,[#Fun<am..>,...],...}) at <0.738.0> exit with reason reached_max_restart_intensity in context shutdown
2018-06-20 14:03:26.865 [info] <0.835.0> accepting AMQP connection <0.835.0> (192.168.1.115:47801 -> 192.168.1.115:5672)
2018-06-20 14:03:26.934 [info] <0.835.0> Connection <0.835.0> (192.168.1.115:47801 -> 192.168.1.115:5672) has a client-provided name: Shovel my-shovel
2018-06-20 14:03:26.935 [info] <0.835.0> connection <0.835.0> (192.168.1.115:47801 -> 192.168.1.115:5672 - Shovel my-shovel): user 'esx' authenticated and granted access to vhost '/'
2018-06-20 14:04:26.938 [info] <0.827.0> terminating static worker with {timeout,{gen_server,call,[<0.855.0>,connect,60000]}}
2018-06-20 14:04:26.938 [error] <0.827.0> ** Generic server <0.827.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,{<<"/">>,<<"my-shovel">>},dynamic,#{ack_mode => on_confirm,dest => #{dest_queue => <<"bino.nms.idc3d">>,fields_fun => #Fun<rabbit_shovel_parameters.11.26683091>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_shovel_parameters.12.26683091>,resource_decl => #Fun<rabbit_shovel_parameters.10.26683091>,uris => ["amqp://esx:esx#192.168.126/"]},name => <<"my-shovel">>,reconnect_delay => 5,shovel_type => dynamic,source => #{delete_after => never,module => rabbit_amqp091_shovel,prefetch_count => 1000,queue => <<"toshovel">>,resource_decl => #Fun<rabbit_shovel_parameters.14.26683091>,source_exchange_key => <<>>,uris => ["amqp://esx:esx#192.168.1.115"]}},undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {timeout,{gen_server,call,[<0.855.0>,connect,60000]}}
2018-06-20 14:04:26.939 [warning] <0.835.0> closing AMQP connection <0.835.0> (192.168.1.115:47801 -> 192.168.1.115:5672 - Shovel my-shovel, vhost: '/', user: 'esx'):
client unexpectedly closed TCP connection
2018-06-20 14:04:26.939 [error] <0.827.0> CRASH REPORT Process <0.827.0> with 1 neighbours exited with reason: {timeout,{gen_server,call,[<0.855.0>,connect,60000]}} in gen_server2:terminate/3 line 1166
2018-06-20 14:04:26.939 [error] <0.410.0> Supervisor {<0.410.0>,rabbit_shovel_dyn_worker_sup} had child {<<"/">>,<<"my-shovel">>} started with rabbit_shovel_worker:start_link(dynamic, {<<"/">>,<<"my-shovel">>}, [{<<"dest-protocol">>,<<"amqp091">>},{<<"dest-queue">>,<<"bino.nms.idc3d">>},{<<"dest-uri">>,<<"a...">>},...]) at <0.827.0> exit with reason {timeout,{gen_server,call,[<0.855.0>,connect,60000]}} in context child_terminated
2018-06-20 14:04:26.940 [error] <0.830.0> ** Generic server <0.830.0> terminating
** Last message in was {'EXIT',<0.827.0>,{timeout,{gen_server,call,[<0.855.0>,connect,60000]}}}
** When Server state == {state,amqp_network_connection,{state,#Port<0.29425>,<<"client 192.168.1.115:47801 -> 192.168.1.115:5672">>,10,<0.836.0>,131072,<0.829.0>,undefined,false},<0.834.0>,{amqp_params_network,<<"esx">>,<<"esx">>,<<"/">>,"192.168.1.115",5672,2047,0,10,60000,none,[#Fun<amqp_uri.12.90191702>,#Fun<amqp_uri.12.90191702>],[{<<"connection_name">>,longstr,<<"Shovel my-shovel">>}],[]},2047,[{<<"capabilities">>,table,[{<<"publisher_confirms">>,bool,true},{<<"exchange_exchange_bindings">>,bool,true},{<<"basic.nack">>,bool,true},{<<"consumer_cancel_notify">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"consumer_priorities">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"per_consumer_qos">>,bool,true},{<<"direct_reply_to">>,bool,true}]},{<<"cluster_name">>,longstr,<<"rabbit#centos">>},{<<"copyright">>,longstr,<<"Copyright (C) 2007-2018 Pivotal Software, Inc.">>},{<<"information">>,longstr,<<"Licensed under the MPL. See http://www.rabbitmq.com/">>},{<<"platform">>,longstr,<<"Erlang/OTP 20.3.4">>},{<<"product">>,longstr,<<"RabbitMQ">>},{<<"version">>,longstr,<<"3.7.5">>}],none,false}
** Reason for termination ==
** "stopping because dependent process <0.827.0> died: {timeout,\n {gen_server,call,\n [<0.855.0>,connect,\n 60000]}}"
2018-06-20 14:04:26.940 [error] <0.830.0> CRASH REPORT Process <0.830.0> with 0 neighbours exited with reason: "stopping because dependent process <0.827.0> died: {timeout,\n {gen_server,call,\n [<0.855.0>,connect,\n 60000]}}" in gen_server:handle_common_reply/8 line 726
2018-06-20 14:04:26.941 [error] <0.844.0> Supervisor {<0.844.0>,amqp_channel_sup} had child channel started with amqp_channel:start_link(network, <0.830.0>, 1, <0.846.0>, {<<"client 192.168.1.115:47801 -> 192.168.1.115:5672">>,1}) at <0.847.0> exit with reason {timeout,{gen_server,call,[<0.855.0>,connect,60000]}} in context child_terminated
2018-06-20 14:04:26.941 [error] <0.844.0> Supervisor {<0.844.0>,amqp_channel_sup} had child channel started with amqp_channel:start_link(network, <0.830.0>, 1, <0.846.0>, {<<"client 192.168.1.115:47801 -> 192.168.1.115:5672">>,1}) at <0.847.0> exit with reason reached_max_restart_intensity in context shutdown
2018-06-20 14:04:26.941 [error] <0.828.0> Supervisor {<0.828.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.829.0>, {amqp_params_network,<<"esx">>,<<"esx">>,<<"/">>,"192.168.1.115",5672,2047,0,10,60000,none,[#Fun<am..>,...],...}) at <0.830.0> exit with reason "stopping because dependent process <0.827.0> died: {timeout,\n {gen_server,call,\n [<0.855.0>,connect,\n 60000]}}" in context child_terminated
2018-06-20 14:04:26.942 [error] <0.828.0> Supervisor {<0.828.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.829.0>, {amqp_params_network,<<"esx">>,<<"esx">>,<<"/">>,"192.168.1.115",5672,2047,0,10,60000,none,[#Fun<am..>,...],...}) at <0.830.0> exit with reason reached_max_restart_intensity in context shutdown
from /var/log/rabbitmq/log/crash.log
2018-06-20 14:04:40 =SUPERVISOR REPORT====
Supervisor: {<0.914.0>,amqp_connection_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [{pid,<0.916.0>},{name,connection},{mfargs,{amqp_gen_connection,start_link,[<0.915.0>,{amqp_params_network,<<"esx">>,<<"esx">>,<<"/">>,"192.168.1.115",5672,2047,0,10,60000,none,[#Fun<amqp_uri.12.90191702>,#Fun<amqp_uri.12.90191702>],[{<<"connection_name">>,longstr,<<"Shovel my-shovel">>}],[]}]}},{restart_type,intrinsic},{shutdown,brutal_kill},{child_type,worker}]
Kindly please give me some clue