When I run
strace -f strace /bin/ls
to know how strace work it failed with
ptrace(PTRACE_TRACEME, 0, 0, 0) = -1 EPERM (Operation not permitted)
even with root account.
It there any solution for this?
Docker
When running strace within Docker container, to enable ptrace, run with SYS_PTRACE param:
docker run -it --cap-add SYS_PTRACE ubuntu
See: Running Strace in Docker.
ptrace system call is limited only one tracing application per process.
man ptrace:
EPERM The specified process cannot be traced. This could be because the tracer has insuffi‐
cient privileges (the required capability is CAP_SYS_PTRACE); unprivileged processes
cannot trace processes that they cannot send signals to or those running set-user-
ID/set-group-ID programs, for obvious reasons. Alternatively, the process may already
be being traced, or (on kernels before 2.6.26) be init(1) (PID 1).
This means only a debug application can attach to same process. When you done strace -f you tell it to attach all process started by program debugged. In your case strace call fork to create a new process and setups the new process for debugging using ptrace system call. Then it calls exec with parameters you provide to the call. This then start strace again which tries to do fork and ptrace again. But the second ptrace fails with EPERM because first strace has already attached to the process.
Running first strace without -f parameter allows you to trace the first thread from second strace while second strace is tracing the ls.
strace strace -f ls
There is -b to detach from lwp when a specific syscall is made but it only supports execve. If there was a ptrace call support it would be perfect. That means strace either needs a small patch to support ptrace call.
Alternative potential hacks include preloaded library which implements detaching with some trickery.
Better alternative would be using tracing tool systemtap or trace-cmd which can use kernel provided tracing infrastructure instead of ptrace.
I mention this and more helpful tips in a recent blog post about strace.
You need to enable support for gdb, strace, and similar tools to attach to processes on the system.
You can do this temporarily by running command to set a setting proc:
sudo bash -c 'echo 0 > /proc/sys/kernel/yama/ptrace_scope'
You can persist that setting between system reboots by modifying /etc/sysctl.d/10-ptrace.conf and setting kernel.yama.ptrace_scope = 0.
If your system does not have /etc/sysctl.d/10-ptrace.conf, you can modify /etc/sysctl.conf and set kernel.yama.ptrace_scope = 0.
Related
We have just started using monit for process monitor and pretty much new in monit. I have a bash script at /home/ubuntu/launch_example.sh. This is continuously running. is it possible to monitor this using monit? monit should start the script if it bash scripts terminates. What should be syntax.I tried below syntax but all the commands are not being executed as ubuntu user, like shell script calls some python scripts.
check process launch_example
matching "launch_example"
start program = "/bin/bash -c '/home/ubuntu/launch_example.sh'"
as uid ubuntu and gid ubuntu
stop program = "/bin/bash -c '/home/ubuntu/launch_example.sh'"
as uid ubuntu and gid ubuntu
The simple answer is "no". Monit is just for monitoring and is not some kind of supervisor/process manager. So if you want to monitor your long running executable, you have to wrap it.
check process launch_example with pidfile /run/launch.pid
start program = "/bin/bash -c 'nohup /home/ubuntu/launch_example.sh &'"
as uid ubuntu and gid ubuntu
stop program = "/bin/bash -c 'kill $(cat /run/launch.pid)'"
as uid ubuntu and gid ubuntu
This quick'n'dirty way also needs an additional line for your launch_example.sh to write the pidfile (pidfile matching should always be preferred over string matching) - it could be just the first line after she shebang. It simply writes the current process ID to the pidfile. Nothing fancy here ;)
echo $$ > /run/launch.pid
In fact, it's not even hard to convert your script into a systemd unit. Here is an example on how to. User binding, restarts, pidfile, and "start-on-boot" can then be managed through systemd (eg. start program = "/usr/bin/systemctl start my_unit").
WSL2 clock goes out of sync after resuming from sleep/hibernate.
A workaround was shared on GitHub sudo hwclock -s to resync clock in WSL, but you have to do this every time you resume from sleep/hibernate.
UPDATE: THIS BUG IS FIXED, just check for updates! See the Clock Sync fix
In case anyone finds this via search and doesn't notice that there is actually a solution listed in the question, you can fix WSL clock drift via.
sudo hwclock -s
If you just need to do it occasionally, this is a fine solution. If you need to do it more frequently, consider #piouson's solution
Update
The fix is now in WSL2 Linux kernel 5.10.16.3 and newer! Note you may need to install WSL2 from the Windows Store to get the latest kernel version per this thread with Craig from Microsoft.
Older Answer
sudo hwclock -s gets you kind of there, but for some reason doesn't get the exact time - I often find it's a minute or so in the future!
sudo ntpdate pool.ntp.org should get you the correct time.
But this is all because of a bug in the Linux kernel which should be included in a Windows update at some point...
There are a number of hacks referenced in the the GitHub issue which can work around this, mostly, but not always, in my experience...
just restart wsl, it works fine for me
wsl --shutdown
then
wsl
in PowerShell
UPDATE: as mentioned by drkvogel, the Clock Sync fix was released in WSL2 kernel version 5.10.16.3
OBSOLETE
At time of writing, this GitHub Issue was open for the bug.
The workaround I chose for my situation (single distro in WSL2) is to use Windows Task Scheduler to run hwclock in WSL whenever Windows resyncs hardware clock.
Windows: Open PowerShell as Administrator
schtasks /create /tn WSLClockSync /tr "wsl.exe sudo hwclock -s" /sc onevent /ec system /mo "*[System[Provider[#Name='Microsoft-Windows-Kernel-General'] and (EventID=1)]]"
Set-ScheduledTask WSLClockSync -Settings (New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries)
WSL2: Run sudo visudo and add hwclock to sudoers to skip password prompt
# bottom of my file looks like this
...
...
#includedir /etc/sudoers.d
<username> ALL=(ALL) NOPASSWD:/usr/sbin/hwclock, /usr/bin/apt update, /usr/bin/apt upgrade
Results
See image for how to get Event XPath from Windows Event filtering. Use as provided to let task scheduler auto-display scheduled triggers.
Use cron to schedule sudo hwclock -s
As others said before sudo hwclock -s syncs the clock,
but you will need to do this after every sleep/hibernate.
Solution is to add an hourly cron task to sync the clock.
Open crontab with sudo (must open with sudo since the command uses sudo):
sudo crontab -e
and add this code with a new line after the task (it's a cron requirement):
PATH=/sbin:/bin:/usr/bin
#hourly hwclock -s
You must either set PATH since root-cron do not has it or use absolute paths e.g. /usr/sbin/hwclock.
cron troubleshooting:
To verify cron is working you may add a dummy task (don't forget to add a new line):
* * * * * date > /tmp/log.txt
If no file is created, verify cron is working: pgrep cron.
If no PID shows, start cron with: sudo service cron start.
To learn cron timing method: cron timing generator
Necro'ing this: As of May 2022, this issue persists to a degree.
There are two components.
First, Windows time sync needs to be decent to start with. It's not, out of the box, on machines that aren't domain-joined.
Change w32time to start automatically. In Administrator cmd, but not PowerShell, sc triggerinfo w32time start/networkon stop/networkoff. Verify with sc qtriggerinfo w32time. To get into cmd that way, you can start Admin PowerShell and then just type cmd.
Make a few changes in regedit.
In Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\w32time\Config, set MaxPollInterval to hex c, decimal 12.
Check Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\w32time\Parameters\NtpServer. If it ends in 0x9 you are done. If it ends in 0x1 you need to adjust SpecialPollInterval in Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\w32time\TimeProviders\NtpClient to read 3600
Reboot, then from Powershell run w32tm /query /status /verbose to verify that w32time service did start. If it didn't, check triggers again. If all else fails, set it to Automatic Delayed startup
Second, WSL2 needs to actually stay in sync. MS will likely release another kernel fix. In the meantime a scheduled task can bring it back into sync periodically:
schtasks /Create /TN WSL2TimeSync /TR "wsl -u root hwclock -s" /SC ONEVENT /EC System /MO "*[System[Provider[#Name='Microsoft-Windows-Kernel-Power'] and (EventID=107 or EventID=507) or Provider[#Name='Microsoft-Windows-Kernel-General'] and (EventID=1)]]" /F
This GitHub Issue was closed
You can also run the below command in Powershell Terminal so sync it.
wsl.exe sudo hwclock -s
You can manually update the WSL2 kernel to 5.10.16 by following the method in this comment: #5650 (comment). I have fixed the issue by this method.
I've added this to Windows Task Scheduler, set to run every 12 hours:
wsl.exe -d ubuntu -u root -- ntpdate time.windows.com
To install ntpdate:
sudo apt install ntpdate
For me this issue seems to be happening when the system goes to sleep. So I have registered a bash command to call whenever, it goes out of sync. I did it by adding a function.sh file and sourced it in ~/.bashrc.
function.sh:
YELLOW='\033[0;33m'
NC='\033[0m'
TIME_SERVER=ntp.ubuntu.com
# Sync wsl time
sync_date () {
echo -e "${RED}sudo ntpdate $TIME_SERVER ${NC}"
echo
sudo ntpdate $TIME_SERVER
}
~/.bashrc:
source ~/Linux/funtions.sh
Note that I have added a bit of color and some customizations (TIME_SERVER: [windows time server is other option]).
You can sync the time using sync_date command in cli.
I am not sure if this is possible without creating my own base image, but I use environment variables in /etc/environment on our servers and typically make them accessible to apache by doing the following:
$ printf "HTTP_VAR1=var1-value\n\
HTTP_VAR2=var2-value"\
>> /etc/environment
$ mkdir /usr/lib/systemd/system/httpd.service.d
$ printf "[Service]\n\
EnvironmentFile=/etc/environment"\
> /usr/lib/systemd/system/httpd.service.d/environment.conf
$ systemctl daemon-reload
$ systemctl restart httpd
$ reboot
The variables are then available in any PHP calls to getenv('HTTP_VAR1'); and etc. However, in running this from a docker file I get dbus errors on the systemctl commands. Without the systemctl commands it seems the variables are not available to apache as it seems the new EnvironmentFile directive doesn't take effect. My docker file snippet:
FROM centos/httpd:latest
RUN printf "HTTP_VAR1=var1-value\n\
HTTP_VAR2=var2-value"\
>> /etc/environment
RUN mkdir /usr/lib/systemd/system/httpd.service.d &&\
printf "[Service]\n\
EnvironmentFile=/etc/environment"\
> /usr/lib/systemd/system/httpd.service.d/environment.conf
RUN systemctl daemon-reload &&\
systemctl restart httpd
COPY entrypoint.sh /entrypoint.sh
So I happened upon the answer to the issue today. It seems that systemd drops backslashes inside single quotes, but it may effect double-quotes too from what I saw in testing. I found the systemd development mailing list thread from April 2014 where patching the issue was being discussed. It seems as though the fix never made it in. So we have to work around it.
In attempting to work around it I noticed some issues with actually reading the variables at all. It seemed as though either Apache or php-cli would get the correct variables, and sometimes not at all, it took a bit of sleuthing to figure out what was going on. Then I started reading into the systemd's EnvironmentFile directive to see if there was more to gain from the docs. It turns out it does not evaluate bash so export won't work. It expects a text file with variable assignments and herein lies one of the main issues that might keep this from being resolved.
I then devised a workable solution. Utilizing systemd's ExecStartPre directive I am able to run a script on startup of the httpd service. I then read in the environment file and write a new plain text one that will then be used by httpd's systemd unit. Here is the code:
Firstly, I moved my variables to /etc/profile.d/ directory rather than /etc/environment file.
file: /etc/profile.d/environment.sh
This is where we store all our environment variables, this gets easily sourced on all interactive shell logins. In the rarer cases where we need to have these variables available non-interactively we can either provide --login flag to /bin/bash or source it manually.
export HTTP_VAR1=var1-value-with-a-back\slash
export HTTP_VAR2=var2-value
file: /usr/lib/systemd/system/httpd.service.d/environment.conf
Our drop-in unit file to extend how the httpd service works. I add in a script that runs before httpd starts up. This gets ran on all httpd restarts and starts. The script that runs generates a plain text file at /etc/profile.d/environment.env which we subsequently tell systemd to load as an EnvironmentFile.
[Service]
ExecStartPre=/usr/bin/bash -c "/usr/local/bin/generate-plain-environment-file"
EnvironmentFile=/etc/profile.d/environment.env
file: /usr/local/bin/generate-plain-environment-file
Here is the script I am using, I whipped this together really fast, I really don't think it is that robust and it could be better. It just removes the export from the beginning of the lines and then escapes any backslashes since systemd drops single backslashes. A more proper solution might be to use bash to evaluate each line and obtain the variable value in case of usage of variables or other bash in the actual bash variables, then output them as plain text name=value assignments, however this is not part of my use-case so I didn't bother.
#!/bin/bash
cd /etc/profile.d/
rm -rf "./environment.env"
while IFS='' read -r line || [[ -n "$line" ]]; do
echo $(echo "${line}" | sed 's/^export //' | sed 's/\\/\\\\/g') >> "./environment.env";
done < "./environment.sh"
file: /etc/profile.d/environment.env
This is the resulting file generated by the described script.
HTTP_VAR1=var1-value-with-a-back\\slash
HTTP_VAR2=var2-value
Conclusion is that the I now have two files with the same thing in them but I only need to maintain the one, the other is generated each time we restart httpd. Also, we fix the backslash issue in the process. Hurray!
It looks like in OS X 10.11 El Capitan, dtruss and dtrace can no longer do what they're meant to do. This is the error I get when I try to run sudo dtruss curl ...:
dtrace: failed to execute curl: dtrace cannot control executables signed with restricted entitlements
I've come across people noticing this problem but so far no solutions.
Is there a way to fix this or work around this?
Following up to Alexander Ushakov and Charles' answers:
Once you csrutil enable --without dtrace, there is an alternative to copying the binary: run the binary in one Terminal window and trace the Terminal process itself in another Terminal window.
In the first terminal window, find its PID:
$ echo $$
1154
In the second terminal window, begin the trace:
$ sudo dtruss -p 1154 -f
Back, in the first terminal window, run the process you want to trace:
$ ls
At this point, you should see the trace in the second window. Ignore the entries for the PID you are tracing (e.g., 1154), and the rest are for the process (and its descendants) you are interested in.
1154/0x1499: sigprocmask(0x3, 0x7FFF53E5C608, 0x0) = 0x0 0
1154/0x1499: sigprocmask(0x1, 0x7FFF53E5C614, 0x7FFF53E5C610) = 0x0 0
3100/0xa9f3: getpid(0x7FFF82A35344, 0x7FFF82A35334, 0x2000) = 3100 0
3100/0xa9f3: sigprocmask(0x3, 0x10BE32EF8, 0x0) = 0x0 0
For those who want to dtrace system shipped binary after csrutil disable, copyit to a directory that is not "restricted", for example, /tmp
CC#~ $ csrutil status
System Integrity Protection status: disabled.
CC#~ $ cp /bin/echo /tmp
CC#~ $ sudo dtruss /tmp/echo
SYSCALL(args) = return
thread_selfid(0x0, 0x0, 0x0) = 46811 0
csops(0x0, 0x0, 0x7FFF51B6CA20) = 0 0
issetugid(0x0, 0x0, 0x7FFF51B6CA20) = 0 0
shared_region_check_np(0x7FFF51B6A918, 0x0, 0x7FFF51B6CA20) = 0 0
stat64("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x7FFF51B6BEA8, 0x7FFF51B6CA20 = 0 0
See #J.J's comment: https://apple.stackexchange.com/questions/208762/now-that-el-capitan-is-rootless-is-there-any-way-to-get-dtrace-working/224731#224731
As Andrew notices it's because of System Integrity Protection, also known as "rootless".
You can disable it completely or partially (enable just dtrace with some limitations).
Completely disable SIP
Although not recommended by Apple, you can entirely disable System
Integrity Protection on you Mac. Here's how:
Boot your Mac into Recovery Mode: reboot it and hold cmd+R until a progress bar appears.
Go to Utilities menu. Choose Terminal there.
Enter this command to disable System Integrity Protection:
$ csrutil disable
It will ask you to reboot — do so and you're free from SIP!
Partially disable SIP
Fortunately, SIP is not monolithic: it's built from many different
modules we can disable/enable separately.
Repeat steps 1 and 2 from «Completely disable SIP» section above. Now
in Terminal enter these commands:
$ csrutil clear # restore the default configuration first
$ csrutil enable --without dtrace # disable dtrace restrictions *only*
Reboot and enjoy your OS again.
Dtrace starts to work but you're still unable to attach dtrace to restricted processes
I would post this as a comment but I'm not allowed.
Disabling SIP is not necessary. Just copy the binary to an alternate location and it works just fine:
$ sudo dtruss ping google.com
dtrace: system integrity protection is on, some features will not be available
dtrace: failed to execute ping: dtrace cannot control executables signed with restricted entitlements
$ sudo cp $(which ping) .
$ sudo dtruss ./ping google.com
dtrace: system integrity protection is on, some features will not be available
SYSCALL(args) = return
PING google.com (172.217.10.78): 56 data bytes
^C
$ csrutil status
System Integrity Protection status: enabled.
For binaries that can still function normally after being copied, this is the best option as it captures the entire lifetime of the process and doesn't require disabling any protections.
Looks like completely disabling SIP still blocks dtruss for restricted processes:
$ /usr/bin/csrutil status
System Integrity Protection status: disabled.
$ sudo dtruss /bin/echo "blah"
dtrace: failed to execute /bin/echo: dtrace cannot control executables signed with restricted entitlements
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.11.2
BuildVersion: 15C50
See my answer on related question "How can get dtrace to run the traced command with non-root priviledges?" [sic].
DTrace can snoop processes that are already running. So, start a background process which waits 1sec for DTrace to start up (sorry for race condition), and snoop the PID of that process.
sudo true && \
(sleep 1; ps) & \
sudo dtrace -n 'syscall:::entry /pid == $1/ {#[probefunc] = count();}' $! \
&& kill $!
Full explanation in linked answer.
I'm attempting to setup a new compute cluster, and currently experiencing errors when using the qsub command in the SGE. Here's a simple experiment that shows the problem:
test.sh
#!/usr/bin/zsh
test="hello"
echo "${test}"
test.sh.eXX
test=hello: Command not found.
test: Undefined variable.
test.sh.oXX
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
If I ran the script on the head node (sh test.sh), the output is correct. I submit the job to the SGE by typing "qsub test.sh".
If I submit the exact same script job in the same way on an established compute cluster like HPC, it works perfectly as expected. What setting could be causing this problem?
Thanks for any help on this matter.
Most likely the queues on your cluster are set to posix_compliant mode with a default shell of /bin/csh. The posix_compliant setting means your #! line is ignored. You can either change the queues to unix_behavior or specify the required shell using qsub's -S option.
#$ -S /bin/sh