Jenkins version: 1.574
I created a simple job which performs the following:
Using "Execute shell script on remote host using SSH" as one of the BUILD steps, I'm just calling a shell script. This shell script performs stop and start operations on Tomcat to restart an application on the target machine.
I have a valid username, password, port defined for the target SSH server in Jenkins Global settings.
I saw this behavior that when I run a Jenkins job and call the restart script (which gets the application name as parameter $1), it works fine, but as soon as "Execute shell script on remote host using SSH" step completes, I see the new process dies on the remote/target application server.
If I run the script from the target/remote server itself, everything works fine and the new process/PID remains live forever, but running the same script from Jenkins, though I don't see any errors and everything works as expected, the new process dies as soon as the above mentioned SSH step is complete and control comes back to the next BUILD step in Jenkins job OR the Jenkins job is complete.
I saw a few posts/blogs and tried setting: BUILD_ID=dontKillMe in the Jenkins job (in various places i.e. Prepare Environment variables and also using Inject Environment variables...). When the job's particular build# is complete, I can see Environment Variables for that build# does say BUILD_ID=dontKillMe as its value (instead of the default Timestamp tag value).
I tried putting nohup before calling the restart script, i.e.,
nohup restart_tomcat.sh "${app}"
I also tried:
BUILD_ID=dontKillMe nohup restart_tomcat.sh "${app}"
This doesn't give any error and creates a nohup.out file on the remote server (but I'm not worried about it as the restart_tomcat.sh script itself creates its own LOG file which I'm "cat"ing after the restart_tomcat.sh script is complete. cat'ing on the log file is performed using another "Execute shell script on remote host using SSH" build step, and it successfully shows the log file created by the restart script).
I don't know what I'm missing at this point, but as soon as the restart_tomcat.sh step is complete, the new PID/process on the remote/target server dies.
How can I fix this?
I've been through this myself.
On my first iteration, before I knew about Jenkins ProcessTreeKiller, I ended up just daemonizing Tomcat. The Apache Tomcat documentation includes a section on running as a daemon.
You can also try disabling the ProcessTreeKiller for your whole Jenkins instance, if it's relatively small (read the first link for information).
The BUILD_ID=dontKillMe should be passed to the shell, and therefore it should be in your command line, not in Jenkins global configuration or job parameters.
BUILD_ID=dontKillMe restart_tomcat.sh "${app}" should have worked without problems.
You can also try nohup restart_tomcat.sh "${app}" & with the & at the end.
My solution (it worked after trying everything else) in Ubuntu 14.04 (Trusty Tahr) (Amazon AWS - Amazon EC2), Jenkins 1.601:
Exec command: (setsid COMMAND < /dev/null > /dev/null 2>&1 &);
Exec in PTY: DISABLED
// Example COMMAND=socat TCP4-LISTEN:1337,fork TCP4:127.0.0.1:1338
I created this Transfer as my last one.
#!/bin/ksh
export BUILD_ID=dontKillMe
I added the above line to the start of my script and the issue was resolved.
Related
I have a unique problem using jmeter SSH command.
I use this step to run spark jobs.
the problem is that one of the commands not working, to clarify it connects and not get response and just wait and wait for hours, and nothing displayed on screen.
I know how to work with the tool, and this behavior is special for this script alone.
All other script worked, I duplicate one that worked for example
sudo /run_stg.sh this command worked
sudo /run_off2-stg.sh this command not worked
if I run the job manually via jenkins it worked
if I entered to command line and use plik ssh it worked,
the problem is just Jmeter, that is waiting and waiting and I can not understand for what?
the job is about 3 minutes, and I wait for response in Jmeter for 4 hours and nothing Jmeter just waiting.
in the console log I set to trace level and nothing, absolutely no idea how to start handle this issue in Jmeter.
an anyone please assists how to make Jmeter to write what happened?
or just to know if he connect or anything
since this behavior all the test can not be performed
Most probably you are as usual misconfiguring the SSH Command sampler.
The idea is not to run the script per se, you need to delegate the script execution to the Unix Shell, for example Bash this way you will be able to combine several commands together, see the output, amend debugging level, etc.
So I would recommend setting your command to something like /bin/bash -c -x /your/script.sh
Another guess, given you use sudo it might be the case that the sudo command simply waits for the password (which JMeter never provides), if this is the case try amending your script permissions using chmod command and allowing your user its execution without root privileges.
And finally, given you're able to run your command using "plik ssh" (whatever it is) you can run it using OS Process Sampler
More information: How to Run External Commands and Programs Locally and Remotely from JMeter
I have a project made and tested with Visual Studio. It works.
Then I uploaded it into Ubuntu server.
Then ran it with dotnet run. Works, remote machines see it (via nginx proxy).
Then I tried dotnet run &. The process seems like started, but nothing listens on port specified. Then, according to the example, I tried sudo nohup dotnet run kestrel > /dev/null 2>&1 &. This time it listened for a while, then died with:
Application started. Press Ctrl+C to shut down.
fail:
Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
An unhandled exception has occurred: Can not find compilation library location for package 'google.protobuf'
System.InvalidOperationException: Can not find compilation library
location for package 'google.protobuf' at
Microsoft.Extensions.DependencyModel.CompilationLibrary.ResolveReferencePaths()
(A fragment of output from nohup.out, first linens, I skipped project details as irrelevant and private).
Any clues what's happening? I still get no errors when running it in foreground.
Here's what I found out: I can't run it (it gives same error messages) when I run it as root. On my test server I have a special user account named "dotnet". When I log in as dotnet I can run the app. As root I can't.
I don't want to run my app with root privileges.
Next try: I run dotnet restore as root. Then I go with nohup dotnet run kestrel > /dev/null 2>&1 & and it works.
Nice. Now is there a way to start my app with limited privileges?
I found the answer myself, so I'll share.
First: do not run .NET core projects on Linux as root if you don't intend to give them full root privileges. I think web applications with root privileges are bad idea and sort of asking for troubles.
But well, when something doesn't work, it's tempting to use sudo from time to time. And it turns out it was the cause of the problem:
dotnet restore and dotnet run must be executed with the same privileges, by the same user. When I issued dotnet restore as root, it worked. It's even harder the other way, when you want to run the project as user with lower privileges. You have to remove all temporary files before issuing dontet restore. So, generally nohup dotnet run kestrel > /dev/null 2>&1 & works as charm, no sudo is needed here, running this as root can be harmful.
Now I always create special dotnet user to run dotnet apps on servers. It's safer this way. The user cannot sudo. When I need to perform administrative tasks I just start separate root session. I use the same approach with database access. The app has only rights to execute procedures, not even select allowed.
I use tmux (tmux 1.8) from Ubuntu 14.04.
I wanted to configure it a bit via ~/.tmux.conf. But whatever I set inside this file my tmux session looks the same. Then I tried a fresh new /etc/tmux.conf but I still get the same display.
It seems that my config is hardcoded and that I cannot change it.
If I remove these two files (~/.tmux.conf and /etc/tmux.conf) my tmux session is still the same. Tmux runs but I can not configure it. But it should be so simple...
Does anybody have already seen this? And how I could solve that? Do I need to compile a fresh new release of tmux?
Today, I have more details :
on one machine it works as expected. It's OK. But I did not changed anything! Strange...
But on another machine (also running Ubuntu same release and up2date like the first machine) it does not work.
The file /etc/tmux.conf does not exist on none of these 2 machines. I put this little config file (~/.tmux.conf) :
# start Window Numbering at 2
set -g base-index 2
When I launch tmux on this second machine, window numbering starts at 0. On the first machine with the same config file, it behaves correctly : it starts at 2.
I'm going crazy!
After you make changes to ~/.tmux.conf make sure tmux sources them with the tmux source-file ~/.tmux.conf shell command.
Try removing all sessions before running tmux. I have noticed that if you have sessions still running, tmux will still load the previous .tmux.config file.
Executing tmux kill-server can stop the server and then try to run the server again using tmux command.
Please note that after killing the server you will lose all open sessions / tabs.
Please note that this is an AIX related question.
I have a jenkins server running on Redhat which is running a node via SSH on an AIX server.
The commands are run non-interactively using SSH to a user on the AIX machine who has ksh as its standard shell.
The problem is that this build needs a number of environment variables, and i can't seem to get it to work.
I have tried:
Jenkins allows me to set some environment variables for the session. So i tried:
ENV="$HOME/.profile"
I tried creating a .kshrc file containing
. .profile
But none of these approaches seems to make KSH run the .profile script.
The .profile script contains the environment setup for the user i need.
How do i get an AIX implementation of KSH to run my .profile script before executing commands?
You need to specifically tell Jenkins that you want to execute them in ksh shell.
By default, Jenkins runs as sh <commands>.
Add a shebang in your shell command as first line,
#!/bin/ksh
Most shells don't source their .profile files on non-interactive sessions. A simple solution is to source the .profile yourself as part of the command you are sending.
So instead of
yourcommand1; yourcommand2
you should send
. ~/.profile; yourcommand1; yourcommand2
over ssh
UPDATE after reading the comment about Jenkins controlling the ssh command
In the case your ssh command is performed by Jenkins you should have a look at https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin, especially the 'Login profile files' paragraph.
I'd say one of these solutions is best
Set all environment variables from Jenkins using the node's configure page. Install the EnvInject plugin to do this.
Write a wrapper around the java command on the slave that sources your profile script and adjust the JavaPath (also on the node's configure page) to point to that wrapper.
The only way I know of for setting environment variables that will apply for non-interactive shells on AIX is via /etc/environment. I believe this is the correct place, but it will of course then apply to all users and all shells.
Hello people.
I'm using Jenkins as CI server and I need to run some performance test using Jmeter. I've setup the plugin and configured my workspace and everything works ok, but I have to do some steps manually and I want a bit more of "automation".
Currently i have some small programs in a remote server. These programs make some specific validations, for instance (just to explain): validates e-mail addresses, phone numbers, etc.
So, before I run the build in jenkins, I have to manually start the program (file.sh) I want:
I have to use putty (or any othe ssh client) to conect to the server and then run, for instance, the command
./email_validation.sh
And the Jmeter test runs in a correct way, and when the test is done I have to manually "shut down" the program I started. But what I want is trying to start the program I need in Jenkins configuration (not manually outside Jenkins, but in "execute shell" or "execute remote shell using ssh" build step).
I have tried to start it, but it get stuck, because when Jenkins build finds the command
./email_validation.sh
the build stops, it waits for the command to finish and then it will continue the other build steps, but obviously, I need this step not to finish until the test is executed.
Is there a way to achieve this? Thanks
Run your command as a background process by adding the & symbol at the end of the command and use the nohup command in case the parent process gets a hangup signal, e.g.
nohup /path/to/email_validation.sh &
If the script produces any output, it will go by default to the file nohup.out in the current directory when the script was launched.
You can kill the process at the end of the build by running:
pkill email_validation.sh