Here is an interesting mystery ...
This code ...
shuf $TRAINING_UNSHUFFLED > $TRAINING_SHUFFLED
wc -l $TRAINING_UNSHUFFLED
wc -l $TRAINING_SHUFFLED
shuf $VALIDATION_UNSHUFFLED > $VALIDATION_SHUFFLED
wc -l $VALIDATION_UNSHUFFLED
wc -l $VALIDATION_SHUFFLED
generates this error ...
shuf: read error: Bad file descriptor
8122 /nfs/digits/datasets/com-aosvapps-distracted-driving3/databases/TrainImagePathsAndLabels_AlpineTest1.csv
0 /nfs/digits/datasets/com-aosvapps-distracted-driving3/databases/TrainImagePathsAndLabels_AlpineTest1_Shuffled.csv
shuf: read error: Bad file descriptor
882 /nfs/digits/datasets/com-aosvapps-distracted-driving3/databases/ValImagePathsAndLabels_AlpineTest1.csv
0 /nfs/digits/datasets/com-aosvapps-distracted-driving3/databases/ValImagePathsAndLabels_AlpineTest1_Shuffled.csv
but ONLY when I run it as a background job like so ...
tf2$nohup ./shuffle.sh >> /tmp/shuffle.log 2>&1 0>&- &
[1] 6897
When I run it directly in an interactive shell, it seems to work fine.
tf2$./shuffle.sh > /tmp/shuffle.log
I am guessing that this has something to do with the fact that both the input and output files reside on an nfs share on a different aws ec2 instance.
The severing of stdin, stderr and stdin in the background process example is suspicious. This is done so that the process will not die if the terminal session is closed. I have many other commands that read and write from this share without any problems at all. Only the shuf command is being difficult.
I am curious as to what might be causing this and if it is fixable without seeking an alternative to shuf?
I am using shuf (GNU coreutils) 8.21 on Ubuntu 14.04.5 LTS.
tf2$which shuf
/usr/bin/shuf
tf2$shuf --version
shuf (GNU coreutils) 8.21
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Paul Eggert.
tf2$lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
UPDATE: eliminating the severing of STDIN makes the problem go away
ie. if instead of doing this ...
$nohup ./shuffle.sh > /tmp/shuffle.log 2>&1 0>&- &
I do this ...
$nohup ./shuffle.sh > /tmp/shuffle.log 2>&1 &
the "Bad descriptor" error goes away.
However, the severing of stdin/stdout/stderr is there to ensure that killing the terminal session will not kill the process, so this solution is not entirely satisfactory.
Furthermore, it only seems be be necessary to do this for shuf. None of other commands which read files from this file system cause any errors.
This turned out to be a bug in glibc.
The details are here:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25029
The work-around is simple:
instead of
shuf $TRAINING_UNSHUFFLED > $TRAINING_SHUFFLED
do
shuf < $TRAINING_UNSHUFFLED > $TRAINING_SHUFFLED
Thanks to Pádraig Brady on the coreutils team.
Related
Background:
I'm trying to figure out how to use ConfigFS to set up an HID device on BeagleBoneBlack.
I found the following example (www.isticktoit.net/?p=1383) on the web and tried it. The sample runs on a Raspberry Pi Zero. However, the sample does not work on my BBB. The following is the script that I wrote and which is executed as root. The script attempts to define a keyboard hid device.
#!/bin/bash
cd /sys/kernel/config/usb_gadget/
modprobe libcomposite
modprobe usb_f_hid
mkdir -p isticktoit
cd isticktoit
echo 0x1d6b > idVendor # Linux Foundation
echo 0x0104 > idProduct # Multifunction Composite Gadget
echo 0x0100 > bcdDevice # v1.0.0
echo 0x0200 > bcdUSB # USB2
mkdir -p strings/0x409
echo "fedcba9876543210" > strings/0x409/serialnumber
echo "Tobias Girstmair" > strings/0x409/manufacturer
echo "iSticktoit.net USB Device" > strings/0x409/product
mkdir -p configs/c.1/strings/0x409
echo "Config 1: ECM network" > configs/c.1/strings/0x409/configuration
echo 250 > configs/c.1/MaxPower
# Add functions here
pwd
mkdir -p functions/hid.xyz
echo 1 > functions/hid.xyz/protocol
echo 1 > functions/hid.xyz/subclass
echo 8 > functions/hid.xyz/report_length
echo -ne \\x05\\x01\\x09\\x06\\xa1\\x01\\x05\\x07\\x19\\xe0\\x29\\xe7\\x15\\x00\\x25\\x01\\x75\\x01\\x95\\x08\\x81\\x02\\x95\\x01\\x75\\x08\\x81\\x03\\x95\\x05\\x75\\x01\\x05\\x08\\x19\\x01\\x29\\x05\\x91\\x02\\x95\\x01\\x75\\x03\\x91\\x03\\x95\\x06\\x75\\x08\\x15\\x00\\x25\\x65\\x05\\x07\\x19\\x00\\x29\\x65\\x81\\x00\\xc0 > functions/hid.xyz/report_desc
ln -s functions/hid.xyz configs/c.1/
# End functions
ls /sys/class/udc > UDC
The error that I get is "ls: write error: Devicew or resource busy".
I am running Debian Jessie - Linux version 4.4.9-ti-r25
I did an lsmod and libcomposite and usb_f_hid are loaded.
The usb device controller, musb-hdrc-0.auto, is loaded.
Questions:
How can I tell which device is busy?
Where can I find the USB configfs defect bug list for BBB.
Is there a logging file and enabling parameter that would give me a clue as to what is happening?
Thanks for any help
David Glaser
The problem you are likely having with the beaglebone black is the cdc_acm driver. It is difficult to remove (well, not really now that I KNOW how to do it) if you don't know how because the steps aren't just laid out to find anywhere yet. I found this: https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/DEFCON-23-Phil-Polstra-One-device-to-Pwn-them-all.pdf
which led me to the following solution
#!/usr/bin/env bash
function checkModule(){
MODULE="$1"
if lsmod | grep "$MODULE" &> /dev/null ; then
echo "$MODULE" found.
return 0
else
echo "$MODULE" not found.
return 1
fi
}
if which 'systemctl' | grep "systemctl" &> /dev/null ; then
systemctl stop serial-getty#ttyGS0.service >/dev/null
fi
if checkModule "g_serial" == 0; then
modprobe -r g_serial
fi
if checkModule "usb_f_acm" == 0; then
modprobe -r usb_f_acm
fi
if ! checkModule "libcomposite" == 0; then
modprobe libcomposite
fi
basically, it stops the serial-getty service which allows you to remove the g_serial device and then this allows you to remove usb_f_acm. This in turn removes the libcomposite device, which you actually want to keep. Once this is done, you can likely do all the things you needed to do. I got a nice HID keyboard working this way (well, okay I guess its a KeygleBone Black now)... It is pretty dirt simple once you understand ALL of the pieces, but I'm having a little trouble tearing my device back down. I might not need to eventually, but I'd like to be able to do that and it seems that certain directories cannot be removed (namely the "strings" directories that I've created). This means I can't really fully tear down the device, but maybe I only need to:
echo "" > /sys/kernel/config/usb_gadget/my_gadget/UDC
to actually tear it down. I haven't worked that part out yet. There are also some C libraries, but I've got a bunch of python scripts that I want to use and I don't yet have python wrappers for those. But that probably isn't too much work.
I didn't want to forget to mention, that I tried to throw the above script into rc.local so the beaglebone black I am using would be "HID ready" on boot. There are probably better locations and methods to do this, but I just wanted to use rc.local because the above is a script, rc.local is a script, it should run on boot... But it doesn't... You have to make sure to make rc.local runable ( chmod 755 /etc/rc.local ) as well as modifying the default shell it wants to run (well, it always runs bash, but its method for running bash is the "POSIX" method, and that doesn't seem to work, so you have to force it to run bash in non-POSIX mode with:
#!/usr/bin/bash
Again, there are probably other better methods (I was lazy here and, well, I'm just old school), especially if your device is going to be an IoT device or anything linked to the net, so you might want to consider something else if you need this script to run on boot.
I did stupidly leave out one thing: I made sure the beaglebone black doesn't present its usual "disk" portion as well. I would put the details here, but frankly, those I'd have to track back down. I basically googled around for how to disable the beaglebone black disk. It isn't hard and amounted to me moving some file to another name so it doesn't find the "USB disk" configuration on boot. You can also change a line in the uboot config somewhere I believe, but I didn't really want to do that.
Found the file: /var/local/bb_usb_mass_storage.img
Well, it might be bbg_usb_mass_storage.img if it is a beaglebone green, but I just moved this file so it wouldn't present the mass storage device. That should allow you to do what you want.
I am writing a Makefile and need to access the HOSTNAME environment variable.
None of the environment variables are available. Here's the Makefile:
all:
echo $(HOSTNAME) "b"
The output is:
echo "b"
b
I am running on ubuntu 14.04 in a VM. gmake version:
make -v
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for x86_64-pc-linux-gnu
And yes, the HOSTNAME environment variable is set:
/tmp$ echo $HOSTNAME
ubuntu
/tmp$
This Makefile works in cygwin.
Do you know why this is not working?
HOSTNAME is set by Bash, but when Make runs a shell, it calls sh; even if this is a symlink to Bash on your system, it runs in POSIX mode, which disables many Bash-specific behaviors.
Workaround 0; always call make with Bash with HOSTNAME exported.
alias make='export HOSTNAME; make'
Workaround 1; use Bash instead of sh explicitly.
SHELL=/bin/bash
HOSTNAME=$(shell echo "$$HOSTNAME")
Workaround 2; call Bash explicitly.
HOSTNAME=$(shell bash -c 'echo $$HOSTNAME')
Workaround 3; run an external command.
HOSTNAME=$(shell uname -n)
Out of these, the last one is probably the least obscure as well as the most efficient.
It looks like in OS X 10.11 El Capitan, dtruss and dtrace can no longer do what they're meant to do. This is the error I get when I try to run sudo dtruss curl ...:
dtrace: failed to execute curl: dtrace cannot control executables signed with restricted entitlements
I've come across people noticing this problem but so far no solutions.
Is there a way to fix this or work around this?
Following up to Alexander Ushakov and Charles' answers:
Once you csrutil enable --without dtrace, there is an alternative to copying the binary: run the binary in one Terminal window and trace the Terminal process itself in another Terminal window.
In the first terminal window, find its PID:
$ echo $$
1154
In the second terminal window, begin the trace:
$ sudo dtruss -p 1154 -f
Back, in the first terminal window, run the process you want to trace:
$ ls
At this point, you should see the trace in the second window. Ignore the entries for the PID you are tracing (e.g., 1154), and the rest are for the process (and its descendants) you are interested in.
1154/0x1499: sigprocmask(0x3, 0x7FFF53E5C608, 0x0) = 0x0 0
1154/0x1499: sigprocmask(0x1, 0x7FFF53E5C614, 0x7FFF53E5C610) = 0x0 0
3100/0xa9f3: getpid(0x7FFF82A35344, 0x7FFF82A35334, 0x2000) = 3100 0
3100/0xa9f3: sigprocmask(0x3, 0x10BE32EF8, 0x0) = 0x0 0
For those who want to dtrace system shipped binary after csrutil disable, copyit to a directory that is not "restricted", for example, /tmp
CC#~ $ csrutil status
System Integrity Protection status: disabled.
CC#~ $ cp /bin/echo /tmp
CC#~ $ sudo dtruss /tmp/echo
SYSCALL(args) = return
thread_selfid(0x0, 0x0, 0x0) = 46811 0
csops(0x0, 0x0, 0x7FFF51B6CA20) = 0 0
issetugid(0x0, 0x0, 0x7FFF51B6CA20) = 0 0
shared_region_check_np(0x7FFF51B6A918, 0x0, 0x7FFF51B6CA20) = 0 0
stat64("/usr/lib/dtrace/libdtrace_dyld.dylib\0", 0x7FFF51B6BEA8, 0x7FFF51B6CA20 = 0 0
See #J.J's comment: https://apple.stackexchange.com/questions/208762/now-that-el-capitan-is-rootless-is-there-any-way-to-get-dtrace-working/224731#224731
As Andrew notices it's because of System Integrity Protection, also known as "rootless".
You can disable it completely or partially (enable just dtrace with some limitations).
Completely disable SIP
Although not recommended by Apple, you can entirely disable System
Integrity Protection on you Mac. Here's how:
Boot your Mac into Recovery Mode: reboot it and hold cmd+R until a progress bar appears.
Go to Utilities menu. Choose Terminal there.
Enter this command to disable System Integrity Protection:
$ csrutil disable
It will ask you to reboot — do so and you're free from SIP!
Partially disable SIP
Fortunately, SIP is not monolithic: it's built from many different
modules we can disable/enable separately.
Repeat steps 1 and 2 from «Completely disable SIP» section above. Now
in Terminal enter these commands:
$ csrutil clear # restore the default configuration first
$ csrutil enable --without dtrace # disable dtrace restrictions *only*
Reboot and enjoy your OS again.
Dtrace starts to work but you're still unable to attach dtrace to restricted processes
I would post this as a comment but I'm not allowed.
Disabling SIP is not necessary. Just copy the binary to an alternate location and it works just fine:
$ sudo dtruss ping google.com
dtrace: system integrity protection is on, some features will not be available
dtrace: failed to execute ping: dtrace cannot control executables signed with restricted entitlements
$ sudo cp $(which ping) .
$ sudo dtruss ./ping google.com
dtrace: system integrity protection is on, some features will not be available
SYSCALL(args) = return
PING google.com (172.217.10.78): 56 data bytes
^C
$ csrutil status
System Integrity Protection status: enabled.
For binaries that can still function normally after being copied, this is the best option as it captures the entire lifetime of the process and doesn't require disabling any protections.
Looks like completely disabling SIP still blocks dtruss for restricted processes:
$ /usr/bin/csrutil status
System Integrity Protection status: disabled.
$ sudo dtruss /bin/echo "blah"
dtrace: failed to execute /bin/echo: dtrace cannot control executables signed with restricted entitlements
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.11.2
BuildVersion: 15C50
See my answer on related question "How can get dtrace to run the traced command with non-root priviledges?" [sic].
DTrace can snoop processes that are already running. So, start a background process which waits 1sec for DTrace to start up (sorry for race condition), and snoop the PID of that process.
sudo true && \
(sleep 1; ps) & \
sudo dtrace -n 'syscall:::entry /pid == $1/ {#[probefunc] = count();}' $! \
&& kill $!
Full explanation in linked answer.
When I try to start a demo jsvc implementation I get the following error output of jsvc:
jsvc -cp ApacheDeamonDemo.jar -pidfile /mypath/pid.txt -outfile /mypath/log.txt -errfile /mypath/err.log net.example.deamon.DemoDeamon
I get the following error ouput:
Cannot find any VM in Java Home /Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home
Cannot locate JVM library file
Service exit with a return value of 1
Actually the path is correct. Therefore I do not understand why jsvc is telling me this. I'm using a mac.
Almost five years later, so probably too late to help the original asker, but I had the same problem today trying to run jsvc with open-jdk-11 for AMD64, so this might help someone later.
To diagnose the problem, I ran jsvc with the --debug flag, and that told me that it was choking on trying to find libjvm.so. I ran find /usr/lib/jvm/java-11-openjdk-amd64 -name libjvm.so and found it at /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so, but jsvc was looking for it at /usr/lib/jvm/java-11-openjdk-amd64//lib/amd64/server/libjvm.so. So, I did this, and then jscv worked:
sudo mkdir /usr/lib/jvm/java-11-openjdk-amd64/lib/amd64
sudo ln -s /usr/lib/jvm/java-11-openjdk-amd64/lib/server /usr/lib/jvm/java-11-openjdk-amd64/lib/amd64/
It turns out the problem is fixed in later versions of jsvc. I experienced the issue with jsvc version 1.0.6, which is the one you get if you run apt install jsvc on Ubuntu 18.04. After I downloaded the 1.2.0 version commons-daemon src from Apache and compiled jsvc myself, the issue is fixed and I didn't need the symlink anymore.
Don't know why jsvc would try to locate all the dylib files and load them with dlopen, but apparently, this doesn't work well with Apple's Java release. While fixing jsvc might not be too hard, I just went firing up the JVM myself like so,
export JAVA_HOME=$(/usr/libexec/java_home)
export CATALINA_HOME=/Users/rong/Projects/apache-tomcat-8.0.12
export CATALINA_BASE=$CATALINA_HOME
java \
-server \
-classpath $CATALINA_HOME/bin/bootstrap.jar:$CATALINA_HOME/bin/tomcat-juli.jar \
-Dcatalina.home=$CATALINA_HOME \
-Dcatalina.base=$CATALINA_BASE \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
-Djava.util.logging.config.file=$CATALINA_BASE/conf/logging.properties \
org.apache.catalina.startup.Bootstrap \
> $CATALINA_BASE/logs/catalina.out \
2> $CATALINA_BASE/logs/catalina.err
By wrapping this in a bash script and adding a bit of forking, changing UID stuff, you can forget about jsvc completely.
GNU make gives me a strange error message, which I do not understand.
gao#L8470-130213 ~
$ make
echo Test
C:\Program: C:\Program: is a directory
make: *** [test] Error 126
This is what I thought of verifying:
gao#L8470-130213 ~
$ less makefile
test:
echo Test
gao#L8470-130213 ~
$ which make
/c/Programx86/GnuWin32/bin/make
gao#L8470-130213 ~
$ /c/Progra~2/GnuWin32/bin/make.exe test
echo Test
C:\Program: C:\Program: is a directory
make: *** [test] Error 126
gao#L8470-130213 ~
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-pc-mingw32
It feels like some other program is trying to run at the end, and that its path includes some spaces. In that case, what program could it be, and how can I prevent it from running?
I have seen this thread and tried to disable my antivirus, which did not help.
I have also looked into permissions, but I am not sure if makefile needs execution rights. I can't seem to be able to change that anyway (running in bash on windows. makefile is not read-only when I check in explorer):
gao#L8470-130213 ~
$ ls -l makefile
-rw-r--r-- 1 gao Administ 21 Apr 15 14:53 makefile
gao#L8470-130213 ~
$ chmod +x makefile
gao#L8470-130213 ~
$ ls -l makefile
-rw-r--r-- 1 gao Administ 21 Apr 15 14:53 makefile
What is going on with make, what can I do?
It's not "some other program" that's trying to run, it's the echo command. Make prints the command to be run, echo test, but you never see the output (test) so that means it failed trying to find the echo program. Unfortunately I'm not very familiar with the vagaries of running GNU make on Windows: there are lots of different options. One possibility would be to get a newer version of GNU make; 3.81 is very old. 3.82 is now available and might work better for you.
Good info you added above about your environment re: using bash; that wasn't clear from the original question and on Windows there are many different ways to do things. You're using the mingw version of make; that version (as I understand it) does NOT use bash as the shell to run commands in: it's supposed to be used with native Windows environments which do not, certainly, have bash available. I believe that the version of make you have is invoking commands directly, and/or using command.com. Certainly not a UNIX shell like bash.
If you want to use bash you should set the SHELL make variable to the path of your bash.exe program. If you're using a Cygwin environment you can use the GNU make that comes with Cygwin which behaves more like a traditional make + shell.
Otherwise you'll need to write your commands using Windows command.com statements.
Again, I don't use Windows so this is mostly hearsay.
PS. The makefile does not need to be executable.
What is going on is that make doesn't like file names or directory names with spaces in them, such as Program Files. Neither do most of the utilities that makefiles typically rely on, such as the shell to execute commands with.
I create a junction from Program Files to ProgramFiles and use the latter whenever I encounter cases like this.