I see that ansible is sending some, let say, welcome command during connection.
This is how it looks in log
SSH: EXEC sshpass -d11 ssh -C -o Port=$port$ -o 'User="$username$"' -o ConnectTimeout=10 $IP$ '/bin/sh -c '"'"'echo '"'"'"'"'"'"'"'"'~$username$'"'"'"'"'"'"'"'"' && sleep 0'"'"''
This is the part which i'm intrested of '/bin/sh -c '"'"'echo '"'"'"'"'"'"'"'"'~$username$'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Can it be disabled? I have a ssh server which don't support such commands. You can't sent command during connection. Maybe it can be replaced with command sent after connection?
Related
in step 10 of tutorial
https://dzone.com/articles/kubespray-10-simple-steps-for-installing-a-product
for deploying a production ready kubernetes cluster with kubespray, an error occured when running ansible-playbook command.error is:
ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh
ssh passwordless is active between nodes and i can run ssh from each nodes without password.
can anyone help me?
thanks
this is my command and it's output:
master-node#master-node:~/kubespray$ sudo ansible all -i inventory/mycluster/hosts.ini -m ping -vvv
ansible 2.7.8
config file = /home/master-node/kubespray/ansible.cfg
configured module search path = [u'/home/master-node/kubespray/library']
ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]
Using /home/master-node/kubespray/ansible.cfg as config file
/home/master-node/kubespray/inventory/mycluster/hosts.ini did not meet host_list requirements, check plugin documentation if this is unexpected
/home/master-node/kubespray/inventory/mycluster/hosts.ini did not meet script requirements, check plugin documentation if this is unexpected
/home/master-node/kubespray/inventory/mycluster/hosts.ini did not meet yaml requirements, check plugin documentation if this is unexpected
Parsed /home/master-node/kubespray/inventory/mycluster/hosts.ini inventory source with ini plugin
META: ran handlers
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/system/ping.py
<192.168.1.107> ESTABLISH SSH CONNECTION FOR USER: worker-node
<192.168.1.107> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=worker-node -o ConnectTimeout=10 -o ControlPath=/home/master-node/.ansible/cp/e24ed02313 192.168.1.107 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/system/ping.py
<192.168.1.142> ESTABLISH SSH CONNECTION FOR USER: master-node
<192.168.1.142> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=master-node -o ConnectTimeout=10 -o ControlPath=/home/master-node/.ansible/cp/01ac2924af 192.168.1.142 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
master-node | UNREACHABLE! => {
"changed": false,
"msg": "SSH Error: data could not be sent to remote host \"192.168.1.142\". Make sure this host can be reached over ssh",
"unreachable": true
}
worker-node | UNREACHABLE! => {
"changed": false,
"msg": "SSH Error: data could not be sent to remote host \"192.168.1.107\". Make sure this host can be reached over ssh",
"unreachable": true
}
When I run this:
$ ansible -i s1, s1 -m raw -a 'echo test' -u root -k
I get:
s1 | SUCCESS | rc=0 >>
test
Shared connection to s1 closed.
But this way:
$ ansible -i s1, s1 -m command -a 'echo test' -u root -k
I don't get "Shared connection to s1 closed." part:
s1 | SUCCESS | rc=0 >>
test
Why is that?
P.S. Above is a simplified way to reproduce the issue. What I'm facing is that when running playbook I get this extra line which is in the way.
UPD The line clearly coming from ssh. And if I run raw command with -vvvv, I get:
Using /etc/ansible/ansible.cfg as config file
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc
<s1> ESTABLISH SSH CONNECTION FOR USER: root
<s1> SSH: EXEC sshpass -d13 ssh -vvv -C -o ControlMaster=auto
-o ControlPersist=60s -o User=root -o ConnectTimeout=10
-o ControlPath=/home/yuri/.ansible/cp/ansible-ssh-%h-%p-%r -tt s1
'echo test'
s1 | SUCCESS | rc=0 >>
test
OpenSSH_7.4p1, OpenSSL 1.0.2k 26 Jan 2017
debug1: Reading configuration data /home/yuri/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
debug1: Control socket "/home/yuri/.ansible/cp/ansible-ssh-s1-22-root" does not exist
<...a lot of output from ssh...>
But with command, it's just:
Using /etc/ansible/ansible.cfg as config file
Loading callback plugin minimal of type stdout, v2.0 from
/usr/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc
Using module file
/usr/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<s1> ESTABLISH SSH CONNECTION FOR USER: root
<s1> SSH: EXEC sshpass -d13 ssh -vvv -C -o ControlMaster=auto
-o ControlPersist=60s -o User=root -o ConnectTimeout=10
-o ControlPath=/home/yuri/.ansible/cp/ansible-ssh-%h-%p-%r s1
'/bin/sh -c '"'"'(
umask 77
&& mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1488989540.6-73006073289737 `"
&& echo ansible-tmp-1488989540.6-73006073289737="` echo ~/.ansible/tmp/ansible-tmp-1488989540.6-73006073289737 `"
) && sleep 0'"'"''
<s1> PUT /tmp/tmpes82wL TO
/root/.ansible/tmp/ansible-tmp-1488989540.6-73006073289737/command.py
<s1> SSH: EXEC sshpass -d13 sftp -o BatchMode=no -b - -vvv -C
-o ControlMaster=auto -o ControlPersist=60s -o User=root
-o ConnectTimeout=10
-o ControlPath=/home/yuri/.ansible/cp/ansible-ssh-%h-%p-%r '[s1]'
<s1> ESTABLISH SSH CONNECTION FOR USER: root
<s1> SSH: EXEC sshpass -d13 ssh -vvv -C -o ControlMaster=auto
-o ControlPersist=60s -o User=root -o ConnectTimeout=10
-o ControlPath=/home/yuri/.ansible/cp/ansible-ssh-%h-%p-%r s1
'/bin/sh -c '"'"'
chmod u+x /root/.ansible/tmp/ansible-tmp-1488989540.6-73006073289737/ /root/.ansible/tmp/ansible-tmp-1488989540.6-73006073289737/command.py
&& sleep 0'"'"''
<s1> ESTABLISH SSH CONNECTION FOR USER: root
<s1> SSH: EXEC sshpass -d13 ssh -vvv -C -o ControlMaster=auto
-o ControlPersist=60s -o User=root -o ConnectTimeout=10
-o ControlPath=/home/yuri/.ansible/cp/ansible-ssh-%h-%p-%r -tt s1
'/bin/sh -c '"'"'
/usr/bin/python /root/.ansible/tmp/ansible-tmp-1488989540.6-73006073289737/command.py;
rm -rf "/root/.ansible/tmp/ansible-tmp-1488989540.6-73006073289737/" > /dev/null 2>&1
&& sleep 0'"'"''
s1 | SUCCESS | rc=0 >>
test
Where is all ssh output gone?
Shared connection to s1 closed.
This message is an error message from ssh client.
With raw: echo test Ansible executes ssh <many parameters> s1 'echo test' and you get stdout/stderr from ssh command. This way message about shared connection pops up in your task result.
With command: echo test Ansible copy python-wrapper (command.py) and execute this wrapper, which in turn spawns echo test and capture stdout/stderr from echo commmand. Then command.py prints echo's result as JSON-object with stdout/stderr/rc keys. The ssh error message still occurs, but you don't see it (it is filtered by Ansible), because Ansible get task result from JSON-object key's and not from ssh plain stdout/stderr/rc.
Where is all ssh output gone?
This is related due to the same difference in handling raw/command. To see detailed ssh output set ANSIBLE_DEBUG=1 environment variable.
If you want to hide this error message, you can use ansible_ssh_extra_args='-o LogLevel=QUIET' inventory variable. But I'm not sure if this can give some other unexpected results.
My SSH using keys is set up properly.
ssh admin#192.168.1.111
admin#DiskStation:~$
But Ansible returns an error:
TASK [setup] *******************************************************************
<192.168.1.111> ESTABLISH SSH CONNECTION FOR USER: admin
<192.168.1.111> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 -o ControlPath=/Users/Shared/Jenkins/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.1.111 '/bin/sh -c '"'"'( umask 22 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1479205446.3-33100049148171 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1479205446.3-33100049148171 `" )'"'"''
<192.168.1.111> PUT /var/folders/pd/8q63k3z93nx_78dggb9ltm4c00007x/T/tmpNJvc43 TO /var/services/homes/admin/.ansible/tmp/ansible-tmp-1479205446.3-33100049148171/setup
<192.168.1.111> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 -o ControlPath=/Users/Shared/Jenkins/.ansible/cp/ansible-ssh-%h-%p-%r '[192.168.1.111]'
fatal: [192.168.1.111]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", "unreachable": true}
Can someone help me?
Ansible returns "unreachable" for the SFTP connection, not SSH.
Either enable SFTP on the target node (or a firewall in-between), or configure Ansible to use SCP in ansible.cfg:
scp_if_ssh = True
I had a similar "unreachable" error, but in my case it was because my playbook file specified the host this way:
[webservers]
ubuntu#123.456.789.111
This worked for us in the past, so presumably this works with some Ansible versions, but not with my version (2.0.0.2). Instead I changed this to what the documentation recommends:
[webservers]
123.456.789.111 ansible_user=ubuntu
and now the SFTP connection does not fail.
After many years of try and error, now I always have these setting on my ansible.cfg:
[defaults]
host_key_checking = false
[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=20
scp_if_ssh = True
[connection]
pipelining = true
The pipelining is my personal preference when dealing with multiple
hosts.
The ssh_args deals with hangs and timeouts, useful when your target remote has unstable connection.
Please check if python is installed on the target machines. It's a prerequisite.
sudo apt install python3
sudo apt install python
sudo apt install python-minimal
My Ansible playbook connects to a remote node using a Proxy.
When the Ansible play book runs; it gives the following ERROR while doing the ssh step.
[root#vm1-msdp ANSIBLE]# ansible-playbook fend_file.yaml -i env/target -vvvvv
PLAY [LAB1] *******************************************************************
GATHERING FACTS ***************************************************************
<10.169.99.222> ESTABLISH CONNECTION FOR USER: msdp
<10.169.99.222> REMOTE_MODULE setup
<10.169.99.222> EXEC sshpass -d9 ssh -C -tt -vvv -o ProxyCommand="nc -x 142.133.134.161:1088 %h %p" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o User=msdp -o ConnectTimeout=10 10.169.99.222 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1473708903.98-28407509853006 && echo $HOME/.ansible/tmp/ansible-tmp-1473708903.98-28407509853006'
fatal: [10.169.99.222] => SSH Error: ssh_exchange_identification: Connection closed by remote host
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
But when I run the ssh command myself, I am able to successfully connect.
[root#vm1-msdp ANSIBLE]# ssh -C -tt -o ProxyCommand="nc -x 142.133.134.161:1088 %h %p" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o User=root -o ConnectTimeout=10 10.169.99.222
root#10.169.99.222's password:
Last login: Mon Sep 12 12:28:19 2016 from 10.169.102.6
root#IC02 ~ #
Do I need to clear any ansible files ?
When you run the SSH command manually, you are specifying the root user. Your Ansible playbook is using your local user of msdp. Try setting your ansible_user variable in your inventory file. Maybe something like:
10.169.99.22 ansible_user=root
I run a computer with Ubuntu 15.10 and I try to run Vagrant with Ansible.
Before start, I like to say that I don't have any idea about server management and especialy the Ansible.
The reason I am going to run my system this way, is because I have start working on a project that requires this installation.
After all, the problem I have is that while provisioning the Vagrant I get the following message
<aaa.dev> ESTABLISH CONNECTION FOR USER: vagrant
<aaa.dev> REMOTE_MODULE setup
<aaa.dev> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o IdentityFile=/media/merianos/Large Internal/Vagrant/ansible-project/.vagrant/machines/default/virtualbox/private_key -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/merianos/.ansible/cp/%h-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 aaa.dev /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1446622406.54-199921739516776 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1446622406.54-199921739516776 && echo $HOME/.ansible/tmp/ansible-tmp-1446622406.54-199921739516776'
fatal: [aaa.dev] => SSH Error: ControlPath too long
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
So, is it posible to help me someone with this issue ?
Just to say that I have try this article : https://github.com/ansible/ansible/issues/11536 and I changed the control_path in my ansible.cfg to control_path = %(directory)s/%%h-%%r but still not working.
Note My installation path contains a space that I can't remove it because are running many other projects on the same HDD and the configuration will be huge for all the projects. I don't know if that space is the problem, but just I say about it.
UPDATE #1
Result before I change anything:
<aaa.dev> ESTABLISH CONNECTION FOR USER: vagrant
<aaa.dev> REMOTE_MODULE setup
<aaa.dev> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o IdentityFile=/media/merianos/Large Internal/Vagrant/ansible-project/.vagrant/machines/default/virtualbox/private_key -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/merianos/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 aaa.dev /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1446628138.53-155680153347939 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1446628138.53-155680153347939 && echo $HOME/.ansible/tmp/ansible-tmp-1446628138.53-155680153347939'
fatal: [aaa.dev] => SSH Error: ControlPath too long
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
Result with control_path = %(directory)s/%%h-%%r :
<aaa.dev> ESTABLISH CONNECTION FOR USER: vagrant
<aaa.dev> REMOTE_MODULE setup
<aaa.dev> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o IdentityFile=/media/merianos/Large Internal/Vagrant/ansible-project/.vagrant/machines/default/virtualbox/private_key -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/merianos/.ansible/cp/%h-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 aaa.dev /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1446628320.4-231606404275563 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1446628320.4-231606404275563 && echo $HOME/.ansible/tmp/ansible-tmp-1446628320.4-231606404275563'
fatal: [aaa.dev] => SSH Error: ControlPath too long
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
UPDATE #2
After I set the ssh_args = -o ControlMaster=off I get the following result:
<aaa.dev> ESTABLISH CONNECTION FOR USER: vagrant
<aaa.dev> REMOTE_MODULE setup
<aaa.dev> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o IdentityFile=/media/merianos/Large Internal/Vagrant/ansible-project/.vagrant/machines/default/virtualbox/private_key -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/merianos/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 aaa.dev /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1446628489.4-10074395967553 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1446628489.4-10074395967553 && echo $HOME/.ansible/tmp/ansible-tmp-1446628489.4-10074395967553'
fatal: [aaa.dev] => SSH Error: ControlPath too long
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
In general for each modification I did, the error message seems to be the same, and maybe the configuration it happens from some other level, but not the ansible.cfg.
Unfortunatelly I don't know where to find that location :(
I described the problem in similar question.
You need to change it to something shorter (if you have long hostname). For test case you can try just ./master, but for real use case, you should use at least ./s/%%h-%%r.
The very first sentence of the documentation for OpenSSH specific settings in Ansible says:
Under the [ssh_connection] header, the following settings are tunable for SSH connections.
So you need to place the ssh_args variable in [ssh_connection] section of the ansible.cfg, for example:
[defaults]
timeout = 600
[ssh_connection]
ssh_args = -o ControlMaster=off
In fact overriding the ssh_args with an empty value disables the defaults for ControlMaster/ControlPersistent/ControlPath in Ansible, so it should simply be:
[ssh_connection]
ssh_args =
Short Answers
Pass this as an argument to ansible or ansible-playbook commands:
-e "ansible_ssh_common_args='-o ControlPath=/tmp/ssh-%r#%h:%p'"
Or as env variable
export ANSIBLE_SSH_ARGS="-o ControlPath=/tmp/ssh-%r#%h:%p'"
Or as an argument to the host definition in the inventory file
[web_servers]
host ansible_ssh_common_args='-o ControlPath=/tmp/ssh-%r#%h:%p'
Avoid using /tmp directory as its not secure
Long Answer
ControlPath too long is the error that belongs to SSH. SSH creates a control (unix) socket to reuse a TCP connection. The Control path is where this socket is saved.
Ansible may use its local config directory as the location to the ControlPath and if this local config directory is long would raise this error. Alternatively if the home directory itself is at a long path this problem may raise.
Typically, this can be fixed at SSH end by simply using a shorter path in the local ssh config file (~/.ssh/config):
Host *
ControlPath /tmp/ssh-%r#%h:%p
This will create socket file in the /tmp directory (very short path and is world writeable) using the username (%r), hostname (%h), and port number (%p) from the SSH connection as part of the filename
/tmp is world writeable and is not safe. Hackers can use the socket file from the /tmp to login to target SSH server and is dangerous. Please prefer using home directory if it is at short path
Ansible typically should read the file ~/.ssh/config and load ControlPath settings. However in some cases if it isn't, then the following can be done:
Use ANSIBLE_SSH_CONFIG environment variable like:
export ANSIBLE_SSH_CONFIG=~/.ssh/config
OR use ansible configuration file. Create an ansible.cfg file in the ansible project directory and add this code to it:
[ssh_connection]
ssh_args = -F /full/path/to/ssh_config
For me, ~ or \~ in the cfg file didn't work. So ~/.ssh/config wouldn't work and we need to specify full path to the .ssh/config for ex: /home/xxxx/.ssh/config