I have successfully set up a master and a minion using the Salt tutorial, between two hosted VPS (Debian 7).
I am trying to set-up a second minion on my laptop (Ubuntu 14.04), but following the same steps fails.
I am suspecting that my ISP is blocking some ports used by Salt.
I ma be wrong but that wouldn't be the first time my problems are related to that (I have some wireless connection included in my housing contract and live in some kind of young worker residence).
Is there a way to tell which ports my ISP is blocking ?
Can I tunnel my salt minion connection through ssh ?
Note : ssh runs fine, if that can help, and I have access to remote servers (the other master and minion).
Anonymised command output below :
$ salt-minion -l debug
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: nha-Ub
[DEBUG ] Configuration file path: /etc/salt/minion
[INFO ] Setting up the Salt Minion "my_machine_name"
[DEBUG ] Created pidfile: /var/run/salt-minion.pid
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Attempting to authenticate with the Salt Master at X.X.X.X
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[ERROR ] Attempt to authenticate with the salt master failed
I'd run a tcpdump on each side to see what packets are being sent and received, an example command for tcpdump:
tcpdump -i $INTERFACE -s 15000 -w testing.pcap
Where interface is eth0 etc.. (ifconfig) to confirm.
Then you can look at this in wireshark.
Another thing to look at is firewall, I'd put an allow rule in iptables for each WAN IP, just to make sure that's not causing any issues also.
By default, salt needs ports 4505 and 4506 open (TCP)
Related
Currently, I have built a small datacenter environment in OTC with Terraform. based on Ubuntu 20.04 images.
The idea is to have a jump host in the setup phase and for operational purposes that allows spontaneous access to service frontends via ssh proxy jumps without permanently routing them to the public net.
Basic setup works fine so far - I can access the jump host with ssh, and can access the internal machines from there with ssh when I put the private key onto the jump host. So, cloudwise the security seems to be fine. Key pair is generated with ed25519, I use the same key for jump host and internal servers (for now).
What I cannot achieve is the proxy jump as a chained command from my outside machine.
On the jump host, I set AllowTcpForwarding to "yes" in /etc/ssh/sshd_config and restarted ssh and sshd services.
My current local ssh config looks like this:
Host otc
User ubuntu
Hostname <FloatingIP-Address>
Port 22
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
IdentityFile= ~/.ssh/ssh_access
ControlPath ~/.ssh/cm-%r#%h:%p
ControlMaster auto
ControlPersist 10m
Host 10.*
User ubuntu
Port 22
IdentityFile=~/.ssh/ssh_access
ProxyJump otc
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
I can use this to ssh otc to the jump host.
What I would expect is that I could use e.g. ssh 10.0.0.56 to reach an internal host without further ado. As well I should be able to use commands like ssh -L 8080:10.0.0.56:8080 10.0.0.56 -N to map an internal server's port to a localhost port on my external machine. This is how I managed that successfully on other hosting scenarios in the public cloud.
All I get is:
Stdio forwarding request failed: Session open refused by peer
kex_exchange_identification: Connection closed by remote host
Journal on the Jump host says:
Jul 30 07:19:04 dev-nc-o-bastion sshd[2176]: refused local port forward: originator 127.0.0.1 port 65535, target 10.0.0.56 port 22
What I checked as well:
ufw is off on the Jump Host.
replaced ProxyJump configuration with ProxyCommand
So I am at the end of my knowledge. Has anyone a hint what else could be the reason? Any help welcome!
Ok, cause is found (but not yet fully explained).
My local ssh setting was allowing multiplexed forwards (ControlMaster auto ) which caused the creation of a unix socket file for the Controlpath in ~/.ssh.
I had to login to the jump host to AllowTcpForwarding in the first place.
After rebooting the sshd, I returned to the local machine and the failure occured when trying to forward to the remote internal machine.
After deleting the socket file in ~/.ssh, the connection can now be established as needed. Obviously, the persistent tunnel was not impacted by the restarted daemon on the jump host and simply refused to follow the new directive.
This cost me two days. On the bright side, I learned a lot about ssh :o
Background
I have a machine in production running an elixir application (no access to iex, only to erl) and I am tasked with running an analysis on why we are consuming so much CPU. The idea here would be to launch observer, check the processes tab and see the processes with the most reductions.
How am I connecting?
To connect I am following a tutorial from a blog:
https://sgeos.github.io/elixir/erlang/observer/2016/09/16/elixir_erlang_running_otp_observer_remotely.html 1
Their instructions are as follows:
launch the app in the production machine with a cookie and a name
from local run: ssh user#public_ip "epmd -names" to get the name of the app and the port used
from local create a ssh tunnel to the remote machine: ssh -L 4369:user#public_ip:4369 -L 42877:user#public_ip:42877 user#public_ip (4369 is the epmd port by default, 42877 is the port of the app)
from local connect to the remote machine using the node's name: erl -name "user#app_name" -setcookie "mah_cookie" -hidden -run observer
Problem
And now in theory I should be able to use observer on the machine. Instead however I am greeted with the following error:
Protocol ‘inet_tcp’: register/listen error: epmd_close
So, after scouring the dark side of internet, I decided to use sudo journalctl -f to check all the logs of the machine and I found this:
channel 3: open failed: administratively prohibited: open failed
my_app_name sshd[8917]: error: connect_to flame#99.999.99.999: unknown host (Name or service not known)
/scripts/watchdog.sh")
my_app_name CRON[9985]: pam_unix(cron:session): session closed for user flame
Where:
erlang -name: my_app_name
machine user: flame
machine public ip: 99.999.99.999 (obviously not real)
so it tells me, unknown host ?? I am confused since 99.999.99.999 is the public IP of the machine itself!
Questions
What am I doing wrong?
I read that in older versions of erlang I can’t monitor a machine with observer if they are in different networks (which is the case, because I want to monitor this machine from my localhost) but I didn’t find any information regarding this in modern days.
If this is in fact impossible, what alternatives do I have?
Solution
After 3 days of non-stop searching, I finally found something that works.
To summarize I am putting it here everything I did.
All steps in local machine:
get the ports from the remote server:
> ssh remote-user#remote-ip "epmd -names"
epmd: up and running on port 4369 with data:
name super_duper_app at port 43175
create a ssh tunel with the ports:
ssh remote-user#remote-ip -L4369:localhost:4369 -L43175:localhost:43175
On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:
iex --name observer#127.0.0.1 --cookie super_duper_cookie
Node.connect :"super_duper_app#127.0.0.1"
> true
:observer.start
With observer started, select the machine from the Nodes menu.
Possible setbacks
If you have tried this and it didn't work there are a few things you can check for:
Check if the EPMD port on your local machine is free, if not, kill the process using it and free it.
Check your ssh tunneling keys and configurations for permissions. As #Roberto Aloi pointed out this link can be useful: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open
I am using elastic cache single node shard redis 4.0 later version.
I enabled In-Transit Encryption and gave redis auth token.
I created one bastion host with stunnal using this link
https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-connect-redis-node/
I am able to connect to elastic cache redis node using following way
redis-cli -h hostname -p 6379 -a mypassword
and i can do telnet also.
BUT
when I ping (expected response "PONG") on redis-cli after connection it is giving
"Error: Connection reset by peer "
I checked security group of both side.
Any idea ?
Bastion Host ubuntu 16.04 machine
As I mentioned in question, I was running the command like this:
redis-cli -h hostname -p 6379 -a mypassword
The correct way to connect into a ElastiCache cluster through stunnel should be using "localhost" as the host address,like this:
redis-cli -h localhost -p 6379 -a mypassword
There is explanation for using the localhost address:
when you create a tunnel between your bastion server and the ElastiCache host through stunnel, the program will start a service that listen to a local TCP port (6379), encapsulate the communication using the SSL protocol and transfer the data between the local server and the remote host.
you need to start the stunnel, check if the service is listening on the localhost address (127.0.0.1), and connect using the "localhost" as the destination address: "
Start stunnel. (Make sure you have installed stunnel using this link https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-connect-redis-node/)
$ sudo stunnel /etc/stunnel/redis-cli.conf
Use the netstat command to confirm that the tunnels have started:
$ netstat -tulnp | grep -i stunnel
You can now use the redis-cli to connect to the encrypted Redis node using the local endpoint of the tunnel:
$redis-cli -h localhost -p 6379 -a MySecretPassword
localhost:6379>set foo "bar"
OK
localhost:6379>get foo
"bar"
Most probably ElastiCache Redis Instance is using Encryption in-transit and Encryption at-rest and by design, the Redis CLI is not compatible with the encryption.
You need to setup stunnel to connect redis cluster
https://datanextsolutions.com/blog/how-to-fix-redis-cli-error-connection-reset-by-peer/
"Error: Connection reset by peer" indicates that Redis is killing your connection without sending any response.
One possible cause is you are trying to connect to the Redis node without using SSL, as your connection will get rejected by the Redis server without a response [1]. Make sure you are connecting through the correct port in your tunnel proxy. If you are connecting directly from the bastion host, you should be using local host.
Another option is that you have incorrectly configured your stunnel to not include a version of SSL that is supported by Redis. You should double check the config file is exactly the same as the one provided in the support doc.
It that doesn't solve your problem, you can try to build the cli included in AWS open source contribution.[2] You'll need to check out the repository, follow the instructions in the readme, and then do make BUILD_SSL=yes make redis-cli.
[1] https://github.com/madolson/redis/blob/unstable/src/ssl.c#L464
[2] https://github.com/madolson/redis/blob/unstable/SSL_README.md
Let me explain my very strange problem. I have one server (Linux Debian Jessie) which had access to my git repository on gitlab.com
Two days ago, I tried to pull some modifications on this server with a simple git pull. I received an error message :
ssh: connect to host gitlab.com port 22: Connection timed out
Si I have done some tests
1. TELNET
To understand why, I have tried a telnet on 22 port = TIMEOUT
2. IPTABLES
I checked my iptables to be sure that SSH port was allowed. It is. If I try a telnet on another service for example like github.com, it works. So I'm allowed in OUTPUT on this port.
3. PING
I thought a ip translation problem. I have done a ping, I obtain this message :
PING 104.210.2.228 (104.210.2.228) 56(84) bytes of data.
--- 104.210.2.228 ping statistics ---
87 packets transmitted, 0 received, 100% packet loss, time 86534ms
4. FAIL2BAN
I use fail2ban, so I have checked if gitlab was in jail address, but it seems not.
So my problem is that I can't reach gitlab.com
If I try from my local machine or from another server, I don't have this problem. It works.
I can't reach gitlab.com only from this server but I don't know why. Maybe someone has an idea which cans be very precious to help me ?
Probably some modification of firewall caused this. For a quick solution use http protocol instead of ssh. Change your url in the git config file to http.
git config --local -e
change entry of
url = git#gitlab.com:username/repo.git , to
url = https://gitlab.com/username/repo.git
You need to give your username and password to authenticate yourself while making a push or pull though as it's http based.
Can I use salt-master in VM(Ubuntu) control salt-minion in VM(Ubuntu) on different computers?
Because when I try to set public ip in /etc/salt/minion and run salt-minion & salt-master, minion always says '[ERROR]Attempt to authenticate with the salt master failed'
Given that you've set the public IP in /etc/salt/minion (and I'm assuming you've set that on the "master: " line, and uncommented that line), and that you've noted the master can't see the minion's key when running salt-key -A, it's likely that something is interrupting communications between the minion and the salt master.
I'd first check the minion can ping the master, and then do a nc -v X.X.X.X 4505 to see if you get tcp connection through.
It's possible you'll need to refer to http://docs.saltstack.com/en/latest/topics/tutorials/firewall.html to make firewall changes to permit the communication.