How to shorten an inittab process entry, a.k.a., where to put environment variables that will be seen by init? - process

I am setting up a Debian Etch server to host ruby and php applications with nginx. I have successfully configured inittab to start the php-cgi process on boot with the respawn action. After serving 1000 requests, the php-cgi worker processes die and are respawned by init. The inittab record looks like this:
50:23:respawn:/usr/local/bin/spawn-fcgi -n -a 127.0.0.1 -p 8000 -C 3 -u someuser -- /usr/bin/php-cgi
I initially wrote the process entry (everything after the 3rd colon) in a separate script (simply because it was long) and put that script name in the inittab record, but because the script would run its single line and die, the syslog was filled with errors like this:
May 7 20:20:50 sb init: Id "50" respawning too fast: disabled for 5 minutes
Thus, I got rid of the script file and just put the whole line in the inittab. Henceforth, no errors show up in the syslog.
Now I'm attempting the same with thin to serve a rails application. I can successfully start the thin server by running this command:
sudo thin -a 127.0.0.1 -e production -l /var/log/thin/thin.log -P /var/run/thin/thin.pid -c /path/to/rails/app -p 8010 -u someuser -g somegroup -s 2 -d start
It works apparently exactly the same whether I use the -d (daemonize) flag or not. Command line control comes immediately back (the processes have been daemonized) either way. If I put that whole command (minus the sudo and with absolute paths) into inittab, init complains (in syslog) that the process entry is too long, so I put the options into an exported environment variable in /etc/profile. Now I can successfully start the server with:
sudo thin $THIN_OPTIONS start
But when I put this in an inittab record with the respawn action
51:23:respawn:/usr/local/bin/thin $THIN_OPTIONS start
the logs clearly indicate that the environment variable is not visible to init; it's as though the command were simply "thin start."
How can I shorten the inittab process entry? Is there another file than /etc/profile where I could set the THIN_OPTIONS environment variable? My earlier experience with php-cgi tells me I can't just put the whole command in a separate script.

And why don't you call a wrapper who start thin whith your options?
start_thin.sh:
#!/bin/bash
/usr/local/bin/thin -a 127.0.0.1 -e production -l /var/log/thin/thin.log -P /var/run/thin/thin.pid -c /path/to/rails/app -p 8010 -u someuser -g somegroup -s 2 -d start
and then:
51:23:respawn:/usr/local/bin/start_thin

init.d script
Use a script in
/etc/rc.d/init.d
and set the runlevel
Here are some examples with thin, ruby, apache
http://articles.slicehost.com/2009/4/17/centos-apache-rails-and-thin
http://blog.fiveruns.com/2008/9/24/rails-automation-at-slicehost
http://elwoodicious.com/2008/07/15/nginx-haproxy-thin-fastcgi-php5-load-balanced-rails-with-php-support/
Which provide example initscripts to use.
edit:
Asker pointed out this will not allow respawning. I suggested forking in the init script and disowning the process so init doesn't hang (it might fork() the script itself, will check). And then creating an infinite loop that waits on the server process to die and restarts it.
edit2:
It seems init will fork the script. Just a loop should do it.

Related

Why does MSSQL in Docker return "The last operation was terminated because the user pressed CTRL+C" on sql queries?

I'm on Archlinux 64x (4.17.4-1-ARCH) with Docker (version 18.06.0-ce, build 0ffa8257ec). I'm using Microsoft's MSSQL docker container CU7. Each time I'm trying to enter a query or to run a SQL file I get this warning message:
Sqlcmd: Warning: The last operation was terminated because the user pressed CTRL+C.
Then when I check in the database with Datagrip, the query hasn't been executed! Here are my commands :
docker pull microsoft/mssql-server-linux:2017-CU7
docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=GitGood*0987654321" -e "MSSQL_PID=Developer" -p 1433:1433 --name beep_boop_boop -d microsoft/mssql-server-linux:2017-CU7
# THIS
sudo echo "CREATE DATABASE test;" > /test.sql
docker exec beep_boop_boop /opt/mssql-tools/bin/sqlcmd -U SA -P GitGood*0987654321 < test.sql
# OR
docker exec beep_boop_boop /opt/mssql-tools/bin/sqlcmd -U SA -P GitGood*0987654321 -Q "CREATE DATABASE test;"
My question is How to avoid Warning operation was terminated by user warning on MSSQL queries?
You should use docker-compose, I'm sure it will make your life easier. My guess is you're getting an error without actually knowing it. First time I tried, I used an unsafe password which didn't meet security requirements and I got this error:
ERROR: Unable to set system administrator password: Password validation failed. The password does not meet SQL Server password policy requirements because it is not complex enough. The password must be at least 8 characters long and contain characters from three of the following four sets: Uppercase letters, Lowercase letters, Base 10 digits, and Symbols..
I see your password is strong, but note that you have a * in your password, which may be executed if not correctly escaped.
Or the server is just not started when running with your command line, example:
# example of a failing attempt
docker run -it --rm -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=GitGood*0987654321' -p 1433:1433 microsoft/mssql-server-linux:2017-CU7 bash
# wait until you're inside the container, then check if server is running
apt-get update && apt-get install -y nmap
nmap -Pn localhost -p 1433
If it's not running, you'll see something like that:
Starting Nmap 7.01 ( https://nmap.org ) at 2018-08-27 06:12 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000083s latency).
Other addresses for localhost (not scanned): ::1
PORT STATE SERVICE
1433/tcp closed ms-sql-s
Nmap done: 1 IP address (1 host up) scanned in 0.38 seconds
Enough with the intro, here's a working solution:
docker-compose.yml
version: '2'
services:
db:
image: microsoft/mssql-server-linux:2017-CU7
container_name: beep-boop-boop
ports:
- 1443:1443
environment:
ACCEPT_EULA: Y
SA_PASSWORD: GitGood*0987654321
Then run the following commands and wait until the image is ready:
docker-compose up -d
docker-compose logs -f &
up -d to demonize the container so it keeps running in the background.
logs -f will read logs and follow (similar to what tail -f does)
& to run the command in the background so we don't need to use a new shell
Now get a bash running inside that container like this:
docker-compose exec db bash
Once inside the image, you can run your commands
/opt/mssql-tools/bin/sqlcmd -U SA -P $SA_PASSWORD -Q "CREATE DATABASE test;"
/opt/mssql-tools/bin/sqlcmd -U SA -P $SA_PASSWORD -Q "SELECT name FROM master.sys.databases"
Note how I reused the SA_PASSWORD environment variable here so I didn't need to retype the password.
Now enjoy the result
name
--------------------------------------------------------------------------------------------------------------------------------
master
tempdb
model
msdb
test
(5 rows affected)
For a proper setup, I recommend replacing the environment key with the following lines in docker-compose.yml:
env_file:
- .env
This way, you can store your secrets outside of your docker-compose.yml and also make sure you don't track .env in your version control (you should add .env to your .gitignore and provide a .env.example in your repository with proper documentation.
Here's an example project which confirms it works in Travis-CI:
https://github.com/GabLeRoux/mssql-docker-compose-example
Other improvements
There are probably some other ways to accomplish this with one liners, but for readability, it's often better to just use some scripts. In the repo, I took a few shortcuts such as sleep 10 in run.sh. This could be improved by actually waiting until the db is up with a proper way. The initialization script could also be part of an entrypoint.sh, etc. Hope it gets you started 🍻

How to add EnvironmentFile directive to systemctl using Docker with centos7/httpd base image

I am not sure if this is possible without creating my own base image, but I use environment variables in /etc/environment on our servers and typically make them accessible to apache by doing the following:
$ printf "HTTP_VAR1=var1-value\n\
HTTP_VAR2=var2-value"\
>> /etc/environment
$ mkdir /usr/lib/systemd/system/httpd.service.d
$ printf "[Service]\n\
EnvironmentFile=/etc/environment"\
> /usr/lib/systemd/system/httpd.service.d/environment.conf
$ systemctl daemon-reload
$ systemctl restart httpd
$ reboot
The variables are then available in any PHP calls to getenv('HTTP_VAR1'); and etc. However, in running this from a docker file I get dbus errors on the systemctl commands. Without the systemctl commands it seems the variables are not available to apache as it seems the new EnvironmentFile directive doesn't take effect. My docker file snippet:
FROM centos/httpd:latest
RUN printf "HTTP_VAR1=var1-value\n\
HTTP_VAR2=var2-value"\
>> /etc/environment
RUN mkdir /usr/lib/systemd/system/httpd.service.d &&\
printf "[Service]\n\
EnvironmentFile=/etc/environment"\
> /usr/lib/systemd/system/httpd.service.d/environment.conf
RUN systemctl daemon-reload &&\
systemctl restart httpd
COPY entrypoint.sh /entrypoint.sh
So I happened upon the answer to the issue today. It seems that systemd drops backslashes inside single quotes, but it may effect double-quotes too from what I saw in testing. I found the systemd development mailing list thread from April 2014 where patching the issue was being discussed. It seems as though the fix never made it in. So we have to work around it.
In attempting to work around it I noticed some issues with actually reading the variables at all. It seemed as though either Apache or php-cli would get the correct variables, and sometimes not at all, it took a bit of sleuthing to figure out what was going on. Then I started reading into the systemd's EnvironmentFile directive to see if there was more to gain from the docs. It turns out it does not evaluate bash so export won't work. It expects a text file with variable assignments and herein lies one of the main issues that might keep this from being resolved.
I then devised a workable solution. Utilizing systemd's ExecStartPre directive I am able to run a script on startup of the httpd service. I then read in the environment file and write a new plain text one that will then be used by httpd's systemd unit. Here is the code:
Firstly, I moved my variables to /etc/profile.d/ directory rather than /etc/environment file.
file: /etc/profile.d/environment.sh
This is where we store all our environment variables, this gets easily sourced on all interactive shell logins. In the rarer cases where we need to have these variables available non-interactively we can either provide --login flag to /bin/bash or source it manually.
export HTTP_VAR1=var1-value-with-a-back\slash
export HTTP_VAR2=var2-value
file: /usr/lib/systemd/system/httpd.service.d/environment.conf
Our drop-in unit file to extend how the httpd service works. I add in a script that runs before httpd starts up. This gets ran on all httpd restarts and starts. The script that runs generates a plain text file at /etc/profile.d/environment.env which we subsequently tell systemd to load as an EnvironmentFile.
[Service]
ExecStartPre=/usr/bin/bash -c "/usr/local/bin/generate-plain-environment-file"
EnvironmentFile=/etc/profile.d/environment.env
file: /usr/local/bin/generate-plain-environment-file
Here is the script I am using, I whipped this together really fast, I really don't think it is that robust and it could be better. It just removes the export from the beginning of the lines and then escapes any backslashes since systemd drops single backslashes. A more proper solution might be to use bash to evaluate each line and obtain the variable value in case of usage of variables or other bash in the actual bash variables, then output them as plain text name=value assignments, however this is not part of my use-case so I didn't bother.
#!/bin/bash
cd /etc/profile.d/
rm -rf "./environment.env"
while IFS='' read -r line || [[ -n "$line" ]]; do
echo $(echo "${line}" | sed 's/^export //' | sed 's/\\/\\\\/g') >> "./environment.env";
done < "./environment.sh"
file: /etc/profile.d/environment.env
This is the resulting file generated by the described script.
HTTP_VAR1=var1-value-with-a-back\\slash
HTTP_VAR2=var2-value
Conclusion is that the I now have two files with the same thing in them but I only need to maintain the one, the other is generated each time we restart httpd. Also, we fix the backslash issue in the process. Hurray!

SSH – Force Command execution on login even without Shell

I am creating a restricted user without shell for port forwarding only and I need to execute a script on login via pubkey, even if the user is connected via ssh -N user#host which doesn't asks SSH server for a shell.
The script should warn admin on connections authenticated with pubkey, so the user connecting shouldn't be able to skip the execution of the script (e.g., by connecting with ssh -N).
I have tried to no avail:
Setting the command at /etc/ssh/sshrc.
Using command="COMMAND" in .ssh/authorized_keys (man authorized_keys)
Setting up a script with the command as user's shell. (chsh -s /sbin/myscript.sh USERNAME)
Matching user in /etc/ssh/sshd_config like:
Match User MYUSERNAME
ForceCommand "/sbin/myscript.sh"
All work when user asks for shell, but if logged only for port forwarding and no shell (ssh -N) it doesn't work.
The ForceCommand option runs without a PTY unless the client requests one. As a result, you don't actually have a shell to execute scripts the way you might expect. In addition, the OpenSSH SSHD_CONFIG(5) man page clearly says:
The command is invoked by using the user's login shell with the -c option.
That means that if you've disabled the user's login shell, or set it to something like /bin/false, then ForceCommand can't work. Assuming that:
the user has a sensible shell defined,
that your target script is executable, and
that your script has an appropriate shebang line
then the following should work in your global sshd_config file once properly modified with the proper username and fully-qualified pathname to your custom script:
Match User foo
ForceCommand /path/to/script.sh
If you only need to run a script you can rely on pam_exec.
Basically you reference the script you need to run in the /etc/pam.d/sshd configuration:
session optional pam_exec.so seteuid /path/to/script.sh
After some testing you may want to change optional to required.
Please refer to this answer "bash - How do I set up an email alert when a ssh login is successful? - Ask Ubuntu" for a similar request.
Indeed in the script only a limited subset on the environment variables is available:
LANGUAGE=en_US.UTF-8
PAM_USER=bitnami
PAM_RHOST=192.168.1.17
PAM_TYPE=open_session
PAM_SERVICE=sshd
PAM_TTY=ssh
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8
PWD=/
If you want to get the user info from authorized_keys this script could be helpful:
#!/bin/bash
# Get user from authorized_keys
# pam_exec_login.sh
# * [ssh - What is the SHA256 that comes on the sshd entry in auth.log? - Server Fault](https://serverfault.com/questions/888281/what-is-the-sha256-that-comes-on-the-sshd-entry-in-auth-log)
# * [bash - How to get all fingerprints for .ssh/authorized_keys(2) file - Server Fault](https://serverfault.com/questions/413231/how-to-get-all-fingerprints-for-ssh-authorized-keys2-file)
# Setup log
b=$(basename $0| cut -d. -f1)
log="/tmp/${b}.log"
function timeStamp () {
echo "$(date '+%b %d %H:%M:%S') ${HOSTNAME} $b[$$]:"
}
# Check if opening a remote session with sshd
if [ "${PAM_TYPE}" != "open_session" ] || [ $PAM_SERVICE != "sshd" ] || [ $PAM_RHOST == "::1" ]; then
exit $PAM_SUCCESS
fi
# Get info from auth.log
authLogLine=$(journalctl -u ssh.service |tail -100 |grep "sshd\[${PPID}\]" |grep "${PAM_RHOST}")
echo ${authLogLine} >> ${log}
PAM_USER_PORT=$(echo ${authLogLine}| sed -r 's/.*port (.*) ssh2.*/\1/')
PAM_USER_SHA256=$(echo ${authLogLine}| sed -r 's/.*SHA256:(.*)/\1/')
# Get details from .ssh/authorized_keys
authFile="/home/${PAM_USER}/.ssh/authorized_keys"
PAM_USER_authorized_keys=""
while read l; do
if [[ -n "$l" && "${l###}" = "$l" ]]; then
authFileSHA256=$(ssh-keygen -l -f <(echo "$l"))
if [[ "${authFileSHA256}" == *"${PAM_USER_SHA256}"* ]]; then
PAM_USER_authorized_keys=$(echo ${authFileSHA256}| cut -d" " -f3)
break
fi
fi
done < ${authFile}
if [[ -n ${PAM_USER_authorized_keys} ]]
then
echo "$(timeStamp) Local user: ${PAM_USER}, authorized_keys user: ${PAM_USER_authorized_keys}" >> ${log}
else
echo "$(timeStamp) WARNING: no matching user in authorized_keys" >> ${log}
fi
I am the author of the OP; I came to the conclusion that what I need to achieve is not possible using SSH only to the date (OpenSSH_6.9p1 Ubuntu-2, OpenSSL 1.0.2d 9 Jul 2015), but I found a great piece of software that uses encrypted SPAuthentication to open SSH port and it's new version (to the date of this post, it's GitHub master branch) has a feature to execute a command always that a user authorizates successfully.
FWKNOP - Encrypted Single Packet Authorization
FWKNOP set iptables rules that allow access to given ports upon a single packet encrypted which is sent via UDP. Then after authorization it allow access for the authorized user for a given time, for example 30 seconds, closing the port after this, leaving the connection open.
1. To install on an Ubuntu linux:
The current version (2.6.0-2.1build1) on Ubuntu repositories to the date still doesn't allow command execution on successful SPA; (please use 2.6.8 from GitHub instead)
On client machine:
sudo apt-get install fwknop-client
On server side:
sudo apt-get install fwknop-server
Here is a tutorial on how to setup the client and server machines
https://help.ubuntu.com/community/SinglePacketAuthorization
Then, after it is set up, on server side:
Edit /etc/default/fwknop-server
Change the line START_DAEMON="no" to START_DAEMON="yes"
Then run:
sudo service fwknop-server stop
sudo service fwknop-server start
2. Warning admin on successful SPA (email, pushover script etc)
So, as stated above the current version present in Ubuntu repositories (2.6.0-2.1build1) cannot execute command on successful SPA. If you need this feature as of the OP, but it will be released at fwknop version (2.6.8), as can it is stated here:
https://github.com/mrash/fwknop/issues/172
So if you need to use it right now you can build from github branch master which have the CMD_CYCLE_OPEN option.
3. More resources on fwknop
https://help.ubuntu.com/community/SinglePacketAuthorization
https://github.com/mrash/fwknop/ (project on GitHub)
http://www.cipherdyne.org/fwknop/ (project site)
https://www.digitalocean.com/community/tutorials/how-to-use-fwknop-to-enable-single-packet-authentication-on-ubuntu-12-04 (tutorial on DO's community)
I am the author of the OP. Also, you can implement a simple logwatcher as the following written in python3, which keeps reading for a file and executes a command when line contains pattern.
logwatcher.python3
#!/usr/bin/env python3
# follow.py
#
# Follow a file like tail -f.
import sys
import os
import time
def follow(thefile):
thefile.seek(0,2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.5)
continue
yield line
if __name__ == '__main__':
logfilename = sys.argv[1]
pattern_string = sys.argv[2]
command_to_execute = sys.argv[3]
print("Log filename is: {}".format(logfilename))
logfile = open(logfilename, "r")
loglines = follow(logfile)
for line in loglines:
if pattern_string in line:
os.system(command_to_execute)
Usage
Make the above script executable:
chmod +x logwatcher.python3
Add a cronjob to start it after reboot
crontab -e
Then write this line there and save it after this:
#reboot /home/YOURUSERNAME/logwatcher.python3 "/var/log/auth.log" "session opened for user" "/sbin/myscript.sh"
The first argument of this script is the log file to watch, and the second argument is the string for which to look in it. The third argument is the script to execute when the line is found in file.
It is best if you use something more reliable to start/restart the script in case it crashes.

Nagios - NRPE: Command '...' not defined

In /usr/local/nagios/etc/nrpe.cfg I added a new command check_this_process to the already pre-defined ones:
command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/$
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s$
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
command[check_this_process]=/usr/local/nagios/libexec/check_procs -w 15 -c 20 -C name
This works:
define service{
use generic-service
host_name my_host
service_description CPU Load
check_command check_nrpe!check_load
}
This doesn't:
define service{
use local-service
host_name my_host
service_description cron
check_command check_nrpe!check_this_process
}
and returns: NRPE: Command 'check_this_process' not defined
The terminology used in the supplied docs is a little confusing, but I'll put it like this:
As written in Page 10 of https://assets.nagios.com/downloads/nagioscore/docs/nrpe/NRPE.pdf, you need to modify /usr/local/nagios/etc/commands.cfg on your Nagios server and add the following to define the check_nrpe command:
define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}
On your Nagios server, define your service definition as you've already done:
define service{
use local-service
host_name my_host
service_description cron
check_command check_nrpe!check_this_process
}
On your remote host to be monitored, the following is going to be different depending on whether you installed NRPE:
using the tarball and xinetd as in
https://assets.nagios.com/downloads/nagioscore/docs/nrpe/NRPE.pdf
or using a package manager like yum as in
http://sharadchhetri.com/2013/03/02/how-to-install-and-configure-nagios-nrpe-in-centos-and-red-hat/
If you used the tarball / xinetd method, your NRPE configuration file will likely be located at /usr/local/nagios/etc/nrpe.cfg on your remote-host-to-be-monitored. (To avoid typing that all the time, I'll just call it "my_host").
So, on my_host, modify /usr/local/nagios/etc/nrpe.cfg.
Add
command[check_this_process]=/usr/local/nagios/libexec/check_procs -w 15 -c 20 -C name
So that it looks like:
command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/$
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s$
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
command[check_this_process]=/usr/local/nagios/libexec/check_procs -w 15 -c 20 -C name
(Note: the above is assuming you have a process called name. If not, replace name with your real process name: i.e. crond)
Restart xinetd:
service xinetd restart
(NOTE: restarting xinted might not be necessary, but I don't use it so I'm a little fuzzy on this one.)
However, if you installed NRPE on my_host using a package manager like yum, your NRPE configuration file will likely be located at /etc/nagios/nrpe.cfg.
So, on my_host, modify /etc/nagios/nrpe.cfg.
Add
command[check_this_process]=/usr/local/nagios/libexec/check_procs -w 15 -c 20 -C name
So that it looks like:
command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/$
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s$
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200
command[check_this_process]=/usr/local/nagios/libexec/check_procs -w 15 -c 20 -C name
Restart the nrpe service:
service nrpe restart
Back on your Nagios server, run a verification of your Nagios configuration settings:
nagios -v /usr/local/nagios/etc/nagios.cfg
Check the output for errors.
If there are no errors, restart Nagios:
service nagios restart
On your Nagios server you should have a check_nrpe utility installed somehwere as a result of installing the "check_nrpe plugin" on your Nagios server.
See pages 9 and 10 of: https://assets.nagios.com/downloads/nagioscore/docs/nrpe/NRPE.pdf
This check_nrpe utility will most likely be located at: /usr/local/nagios/libexec/check_nrpe
Using the host information for my_host manually test your NRPE connection from the Nagios server.
Execute the following:
/usr/local/nagios/libexec/check_nrpe -H <IP Address of my_host> -c check_this_process
If everything is setup correctly, you should get some output on the command line.
My trouble-shooting guide for 'NRPE: Command ... not found.' Ordered from most common to least common - in my environment.
Was the NRPE daemon restarted AFTER adding the new command? If it is a new command, then NRPE MUST be restarted.
Typos/spelling errors. Does the configured command name on the Nagios side, match that the one in the NRPE config?
Permissions issues. Does the USER that NRPE runs as, have READABLE and EXECUTABLE access to the actual command being ran? Did you test run the command, as the NRPE user? On that same system? TIP: Use the dash (-) when changing to the NRPE user on Linux (su - ...) so you import said users environment as well.
Path issues. Was the FULL PATH to the actual command put into the NRPE config file? Doing this will normally eliminate issues with PATHs, so don't do it any other way.
Bad commands. Does the actual command really execute? Or is it simply throwing an error and exiting? Do you have the correct version of (INSERT SOMETHING HERE) to run the command, installed on the remote system? You should be able to run any command defined in the nrpe.cfg from the command line, and all new commands should be checked BEFORE being added to the nrpe.cfg.
IF ALL THE ABOVE FAILS: Enable DEBUGGING in NRPE and check the log files (on the remote host). This is a bit of a drawn out process - described in the documentation - read it. It is important to disable DEBUGGING as soon as you get output that looks like it would be useful.
This checklist ASSUMES that you've done the needful things to the various Nagios and NRPE configs to get it working in the first place. Hopefully others will read this before posting yet another question as to why they are seeing this error.

ssh client (dropbear on a router) does no output when put in background

I'm trying to automate some things on remote Linux machines with bash scripting on Linux machine and have a working command (the braces are a relict from cmd concatenations):
(ssh -i /path/to/private_key user#remoteHost 'sh -c "echo 1; echo 2; echo 3; uname -a"')
But if an ampersand is concatenated to execute it in background, it seems to execute, but no output is printed, neither on stdout, nor on stderr, and even a redirection to a file (inside the braces) does not work...:
(ssh -i /path/to/private_key user#remoteHost 'sh -c "echo 1; echo 2; echo 3; uname -a"') &
By the way, I'm running the ssh client dropbear v0.52 in BusyBox v1.17.4 on Linux 2.4.37.10 (TomatoUSB build on a WRT54G).
Is there a way to get the output either? What's the reason for this behaviour?
EDIT:
For convenience, here's the plain ssh help output (on my TomatoUSB):
Dropbear client v0.52
Usage: ssh [options] [user#]host[/port][,[user#]host/port],...] [command]
Options are:
-p <remoteport>
-l <username>
-t Allocate a pty
-T Don't allocate a pty
-N Don't run a remote command
-f Run in background after auth
-y Always accept remote host key if unknown
-s Request a subsystem (use for sftp)
-i <identityfile> (multiple allowed)
-L <listenport:remotehost:remoteport> Local port forwarding
-g Allow remote hosts to connect to forwarded ports
-R <listenport:remotehost:remoteport> Remote port forwarding
-W <receive_window_buffer> (default 12288, larger may be faster, max 1MB)
-K <keepalive> (0 is never, default 0)
-I <idle_timeout> (0 is never, default 0)
-B <endhost:endport> Netcat-alike forwarding
-J <proxy_program> Use program pipe rather than TCP connection
Amendment after 1 day:
The braces do not hurt, with and without its the same result. I wanted to put the ssh authentication to background, so the -f option is not a solution. Interesting side note: if an unexpected option is specified (like -v), the error message WARNING: Ignoring unknown argument '-v' is displayed - even when put in background, so getting output from background processes generally works in my environment.
I tried on x86 Ubuntu regular ssh client: it works. I also tried dbclient on x86 Ubuntu: works, too. So this problem seems to be specific to the TomatoUSB build - or inside the "dropbear v0.52" was an unknown fix between the build in TomatoUSB and the one Ubuntu provides (difference in help output is just the double-sized default receive window buffer on Ubuntu)... how can a process know if it was put in background? Is there a solution to the problem?
I had the similar problem on my OpenWRT router. Dropbear SSH client does not write anything to output if there is no stdin, e.g. when run by cron. I presume that & has the same effect on process stdin (no input).
I found some workaround on author's bugtracker. Try to redirect input from /dev/zero.
Like:
ssh -i yourkey user#remotehost "echo 123" </dev/zero &
It worked for me as I tried to describe at my blog page.