Using RamNode's Turnkey Linux distro I noticed my websites have expired certificates and https links are flagged with a "Proceed" dialogue. Looking further into the issue, Turnkey Linux uses confconsole with Let's Encrypt to request new certs. Multiple [virtual] domains requires the user to manually use the dehydrated-wrapper which the cronjob calls daily under /etc/cron.daily/confconsole-dehydrated but results in an error:
/var/log/confconsole/letsencrypt.log
[2019-03-09 05:35:04] dehydrated-wrapper: FATAL: An unexpected service is listening on port 80: nginx:
[2019-03-09 05:35:04] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert & key.
Restarting SSL tunnels: [stopped: /etc/stunnel/stunnel.conf] [Started: /etc/stunnel/stunnel.conf] stunnel.
[2019-03-09 05:35:09] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
[2019-03-09 05:35:09] cron: ERROR: dehydrated-wrapper exited with a non-zero exit code.
[2019-03-10 05:35:04] cron: /etc/ssl/private/cert.pem has expired or will do so within 30 days. Attempting renewal.
Solution:
Update /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper with
REPLACE:
netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 | sed -e 's [[:space:]].*$||'
WITH:
netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 \
| sed -e 's|[[:space:]].*$||; s|[^a-zA-Z0-9]||'
like in this commit https://github.com/turnkeylinux/confconsole/commit/d1e61c4767c2148663429d63bc3a42925af8cbcd
Then manually run the cronjob again or wait for tomorrow:
/etc/cron.daily/confconsole-dehydrated
[2019-03-31 19:26:45] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for DOMAIN1 to /etc/ssl/private
[2019-03-31 19:26:52] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for DOMAIN2 to /etc/ssl/private
[2019-03-31 19:26:59] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for DOMAIN3 to /etc/ssl/private
Thank you and I hope it saves you some time
Related Link: https://github.com/turnkeylinux/tracker/issues/976
I've answered this question in the Question Post itself, however, here's the answer again for the robots.
Solution: Update /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper with
REPLACE:
netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 | sed -e 's [[:space:]].*$||'
WITH:
netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 \
| sed -e 's|[[:space:]].*$||; s|[^a-zA-Z0-9]||'
I could further expand on the solution by discussing the root problem.
Because the line in dehydrated-wrapper had been assigning the WEBSERVER variable to nginx:, dehydrated was unable to stop nginx before requesting and updating the certs. Adding nginx: to the case statement would allow you to stop nginx, but would not be able to start the nginx: process, because it does not exist, to host the .well-known/acme-challenges location, resulting in a 404 when dehydrated tries to verify the https ssl certs are correctly configured.
How can I get a list of all connected IP-adresses that do have a connection to the server using SSH?
I want to get a list of all IP addresses that do have a connection, including bots.
sudo netstat -tanp | grep ESTABLISHED | grep ssh | awk '{print $5}' | sed 's/:.*//'
netstat for listing all connections with ports and daemons
filter only ESTABLISHED connections
filter only SSH connections
print only remote IP address
remove port part
Team,
I am facing some difficulties running commands on a remote machine. I am unable to understand why ssh is trying to think that the command I pass is a host.
ssh -tt -i /root/.ssh/teamuser.pem teamuser#myserver 'cd ~/bin && ./ssh-out.sh'
|-----------------------------------------------------------------|
| This system is for the use of authorized users only. |
| Individuals using this computer system without authority, or in |
| excess of their authority, are subject to having all of their |
| activities on this system monitored and recorded by system |
| personnel. |
| |
| In the course of monitoring individuals improperly using this |
| system, or in the course of system maintenance, the activities |
| of authorized users may also be monitored. |
| |
| Anyone using this system expressly consents to such monitoring |
| and is advised that if such monitoring reveals possible |
| evidence of criminal activity, system personnel may provide the |
| evidence of such monitoring to law enforcement officials. |
|-----------------------------------------------------------------|
ssh: Could not resolve hostname cd: No address associated with hostname
Connection to myserver closed.
It works absolutely fine if I don't pass a command. It simply logs me in. Any ideas?
Man ssh says:
If command is specified, it is executed on the remote host instead of
a login shell.
The thing is that cd is a bash built-in (run type cd in your terminal). So, ssh tries to run cd as a shell, but can not find it in PATH.
You should invoke ssh something like this:
ssh user#host -t 'bash -l -c "cd ~/bin && ./ssh-out.sh"'
Is there any way to Remove all Redis Client Connections with one command?
I know that it's possible to remove by IP:PORT
CLIENT KILL addr:port
Also I found that is possible to do this since Redis 2.8.12.
But
I couldn't find anything about.
CLIENT KILL can receive TYPE argument that can be one of a three connection types; normal, slave and pubsub.
You can kill all open connections by sending the following three commands:
CLIENT KILL TYPE normal
CLIENT KILL TYPE slave
CLIENT KILL TYPE pubsub
Note that you can skip the later two if you do not use them (slave and pubsub connections).
You can also add a SKIPME no for a kamikaze connections killer.
So SHUTDOWN is definitely the easiest way, especially in dev.
However, although Redis doesn't have a CLIENT KILL * variant, you can script it. AFAIR you could even do it in Lua but I checked now and CLIENT LIST errs so I'm guessing that's changed. Still, it is fairly easy to do this with the CLI - this appears to do the trick:
redis-cli CLIENT LIST | cut -d ' ' -f 2 | cut -d = -f 2 | awk -e '{ print "CLIENT KILL " $0 }' | redis-cli -x
You can use the following command to check your connection numbers:
netstat -an | grep :6379 | grep ESTABLISHED | wc -l
Then try Redis Client command to kill connection:
http://redis.io/commands/client-kill
I have written a small bash script which needs an ssh tunnel to draw data from a remote server, so it prompts the user:
echo "Please open an ssh tunnel using 'ssh -L 6000:localhost:5432 example.com'"
I would like to check whether the user had opened this tunnel, and exit with an error message if no tunnel exist. Is there any way to query the ssh tunnel, i.e. check if the local port 6000 is really tunneled to that server?
Netcat is your friend:
nc -z localhost 6000 || echo "no tunnel open"
This is my test. Hope it is useful.
# $COMMAND is the command used to create the reverse ssh tunnel
COMMAND="ssh -p $SSH_PORT -q -N -R $REMOTE_HOST:$REMOTE_HTTP_PORT:localhost:80 $USER_NAME#$REMOTE_HOST"
# Is the tunnel up? Perform two tests:
# 1. Check for relevant process ($COMMAND)
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
# 2. Test tunnel by looking at "netstat" output on $REMOTE_HOST
ssh -p $SSH_PORT $USER_NAME#$REMOTE_HOST netstat -an | egrep "tcp.*:$REMOTE_HTTP_PORT.*LISTEN" \
> /dev/null 2>&1
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi
Autossh is best option - checking process is not working in all cases (e.g. zombie process, network related problems)
example:
autossh -M 2323 -c arcfour -f -N -L 8088:localhost:80 host2
This is really more of a serverfault-type question, but you can use netstat.
something like:
# netstat -lpnt | grep 6000 | grep ssh
This will tell you if there's an ssh process listening on the specified port. it will also tell you the PID of the process.
If you really want to double-check that the ssh process was started with the right options, you can then look up the process by PID in something like
# ps aux | grep PID
Use autossh. It's the tool that's meant for monitoring the ssh connection.
We can check using ps command
# ps -aux | grep ssh
Will show all shh service running and we can find the tunnel service listed
These are more detailed steps to test or troubleshoot an SSH tunnel. You can use some of them in a script. I'm adding this answer because I had to troubleshoot the link between two applications after they stopped working. Just grepping for the ssh process wasn't enough, as it was still there. And I couldn't use nc -z because that option wasn't available on my incantation of netcat.
Let's start from the beginning. Assume there is a machine, which will be called local with IP address 10.0.0.1 and another, called remote, at 10.0.3.12. I will prepend these hostnames, to the commands below, so it's obvious where they're being executed.
The goal is to create a tunnel that will forward TCP traffic from the loopback address on the remote machine on port 123 to the local machine on port 456. This can be done with the following command, on the local machine:
local:~# ssh -N -R 123:127.0.0.1:456 10.0.3.12
To check that the process is running, we can do:
local:~# ps aux | grep ssh
If you see the command in the output, we can proceed. Otherwise, check that the SSH key is installed in the remote. Note that excluding the username before the remote IP, makes ssh use the current username.
Next, we want to check that the tunnel is open on the remote:
remote:~# netstat | grep 10.0.0.1
We should get an output similar to this:
tcp 0 0 10.0.3.12:ssh 10.0.0.1:45988 ESTABLISHED
Would be nice to actually see some data going through from the remote to the host. This is where netcat comes in. On CentOS it can be installed with yum install nc.
First, open a listening port on the local machine:
local:~# nc -l 127.0.0.1:456
Then make a connection on the remote:
remote:~# nc 127.0.0.1 123
If you open a second terminal to the local machine, you can see the connection. Something like this:
local:~# netstat | grep 456
tcp 0 0 localhost.localdom:456 localhost.localdo:33826 ESTABLISHED
tcp 0 0 localhost.localdo:33826 localhost.localdom:456 ESTABLISHED
Better still, go ahead and type something on the remote:
remote:~# nc 127.0.0.1 8888
Hallo?
anyone there?
You should see this being mirrored on the local terminal:
local:~# nc -l 127.0.0.1:456
Hallo?
anyone there?
The tunnel is working! But what if you have an application, called appname, which is supposed to be listening on port 456 on the local machine? Terminate nc on both sides then run your application. You can check that it's listening on the correct port with this:
local:~# netstat -tulpn | grep LISTEN | grep appname
tcp 0 0 127.0.0.1:456 0.0.0.0:* LISTEN 2964/appname
By the way, running the same command on the remote should show sshd listening on port 127.0.0.1:123.
#!/bin/bash
# Check do we have tunnel to example.com server
lsof -i tcp#localhost:6000 > /dev/null
# If exit code wasn't 0 then tunnel doesn't exist.
if [ $? -eq 1 ]
then
echo ' > You missing ssh tunnel. Creating one..'
ssh -L 6000:localhost:5432 example.com
fi
echo ' > DO YOUR STUFF < '
stunnel is a good tool to make semi-permanent connections between hosts.
http://www.stunnel.org/
If you are using ssh in background, use this:
sudo lsof -i -n | egrep '\<ssh\>'