I am wondering if it is possible for CMake to run tests like one might run with a configure script. Specifically I want to test if the system I am compiling on has support for the rdtscp instruction.
I am using Linux and if I were using a configure script I could do something like:
cat /proc/cpuinfo | head -n 19 | tail -1 | grep -c rdtscp
which would give me 0 if the rdtscp feature was not present or a 1 if it were. I could then use this to determine whether to #define RDTSCP. I'm wondering if it's possible to do something similar with CMake even if it's not completely portable (I'm only running under Linux I'm not using Visual Studio, etc.).
execute_process(COMMAND cat /proc/cpuinfo
COMMAND head -n 19
COMMAND tail -1
COMMAND grep -c rdtscp
OUTPUT_VARIABLE OUT)
Selecting line 19 exactly makes this brittle. On my desktop (Linux 4.20 on i7-6700k), that line is
wp : yes
Instead use grep's pattern-matching ability to check for the flags\t\t: line.
grep -l '^flags[[:space:]]*:.*rdtscp' /proc/cpuinfo prints the filename and exits with success after the first match. Or prints nothing and exists with failure status if it doesn't find a match.
I don't know CMake, but based on the other answer presumably you'd use
execute_process(COMMAND grep -l '^flags[[:space:]]*:.*rdtscp' /proc/cpuinfo
OUTPUT_VARIABLE OUT)
The simpler version of this is just grep -l rdtscp /proc/cpuinfo, but requiring a match in the flags : line will prevent any possible false-positive. (To be even more belt-and-suspenders, you could require space or end of line before/after, maybe with PCREgrep for zero-width assertions. In case some future feature flag like XYZrdtscpABC that can be present without RDTSCP support becomes a thing in the future. Or like broken_rdtscp). Or we could just assume that rdtscp is never at the end of the a line and look for ^flags.*:.* rdtscp.
Using -l gets grep to exit after the first match, in case you were using head/tail as an optimization to avoid processing more lines on massively multi-core systems like Xeon Phi? It will still read the whole file if there's no match for rdtscp, but probably any massively-multi-core system will have RDTSCP. And grep is very fast anyway.
Related
I'm trying to use the --immediate-submit on a PBSPro cluster. I tried using an in-place modification of the dependencies string to adapt it to PBSPro, similar to what is done here.
snakemake --cluster "qsub -l wd -l mem={cluster.mem}GB -l ncpus={threads} -e {cluster.stderr} -q {cluster.queue} -l walltime={cluster.walltime} -o {cluster.stdout} -S /bin/bash -W $(echo '{dependencies}' | sed 's/^/depend=afterok:/g' | sed 's/ /:/g')"
This last part gets converted into, for example:
-W depend=afterok: /g/data1a/va1/dk0741/analysis/2018-03-25_marmo_test/.snakemake/tmp.cyrhf51c/snakejob.trimmomatic_pe.7.sh
There are two problems here:
How can I get the dependencies string to output job ID instead of the script path? The qsub command normally outputs the job ID to stdout, so I'm not sure why it's not doing so here.
How do I get rid of the space after afterok:? I've tried everything!
As an aside, it would be helpful if there were some option to debug the submission or not to delete the tmp.cyrhf51c directory in .snakemake -- is there some way to do this?
Thanks,
David
I suggest to use a profile for this, instead of trying to find an ad-hoc solution. This will also help with debugging. E.g., there is already a pbs-torque profile available (https://github.com/Snakemake-Profiles/pbs-torque), probably there is not much to change towards pbspro?
GNU make gives me a strange error message, which I do not understand.
gao#L8470-130213 ~
$ make
echo Test
C:\Program: C:\Program: is a directory
make: *** [test] Error 126
This is what I thought of verifying:
gao#L8470-130213 ~
$ less makefile
test:
echo Test
gao#L8470-130213 ~
$ which make
/c/Programx86/GnuWin32/bin/make
gao#L8470-130213 ~
$ /c/Progra~2/GnuWin32/bin/make.exe test
echo Test
C:\Program: C:\Program: is a directory
make: *** [test] Error 126
gao#L8470-130213 ~
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-pc-mingw32
It feels like some other program is trying to run at the end, and that its path includes some spaces. In that case, what program could it be, and how can I prevent it from running?
I have seen this thread and tried to disable my antivirus, which did not help.
I have also looked into permissions, but I am not sure if makefile needs execution rights. I can't seem to be able to change that anyway (running in bash on windows. makefile is not read-only when I check in explorer):
gao#L8470-130213 ~
$ ls -l makefile
-rw-r--r-- 1 gao Administ 21 Apr 15 14:53 makefile
gao#L8470-130213 ~
$ chmod +x makefile
gao#L8470-130213 ~
$ ls -l makefile
-rw-r--r-- 1 gao Administ 21 Apr 15 14:53 makefile
What is going on with make, what can I do?
It's not "some other program" that's trying to run, it's the echo command. Make prints the command to be run, echo test, but you never see the output (test) so that means it failed trying to find the echo program. Unfortunately I'm not very familiar with the vagaries of running GNU make on Windows: there are lots of different options. One possibility would be to get a newer version of GNU make; 3.81 is very old. 3.82 is now available and might work better for you.
Good info you added above about your environment re: using bash; that wasn't clear from the original question and on Windows there are many different ways to do things. You're using the mingw version of make; that version (as I understand it) does NOT use bash as the shell to run commands in: it's supposed to be used with native Windows environments which do not, certainly, have bash available. I believe that the version of make you have is invoking commands directly, and/or using command.com. Certainly not a UNIX shell like bash.
If you want to use bash you should set the SHELL make variable to the path of your bash.exe program. If you're using a Cygwin environment you can use the GNU make that comes with Cygwin which behaves more like a traditional make + shell.
Otherwise you'll need to write your commands using Windows command.com statements.
Again, I don't use Windows so this is mostly hearsay.
PS. The makefile does not need to be executable.
What is going on is that make doesn't like file names or directory names with spaces in them, such as Program Files. Neither do most of the utilities that makefiles typically rely on, such as the shell to execute commands with.
I create a junction from Program Files to ProgramFiles and use the latter whenever I encounter cases like this.
When I run the following command in csh, I got nothing, but it works in bash.
Is there any equivalent in csh which can redirect the standard error to standard out?
somecommand 2>&1
The csh shell has never been known for its extensive ability to manipulate file handles in the redirection process.
You can redirect both standard output and error to a file with:
xxx >& filename
but that's not quite what you were after, redirecting standard error to the current standard output.
However, if your underlying operating system exposes the standard output of a process in the file system (as Linux does with /dev/stdout), you can use that method as follows:
xxx >& /dev/stdout
This will force both standard output and standard error to go to the same place as the current standard output, effectively what you have with the bash redirection, 2>&1.
Just keep in mind this isn't a csh feature. If you run on an operating system that doesn't expose standard output as a file, you can't use this method.
However, there is another method. You can combine the two streams into one if you send it to a pipeline with |&, then all you need to do is find a pipeline component that writes its standard input to its standard output. In case you're unaware of such a thing, that's exactly what cat does if you don't give it any arguments. Hence, you can achieve your ends in this specific case with:
xxx |& cat
Of course, there's also nothing stopping you from running bash (assuming it's on the system somewhere) within a csh script to give you the added capabilities. Then you can use the rich redirections of that shell for the more complex cases where csh may struggle.
Let's explore this in more detail. First, create an executable echo_err that will write a string to stderr:
#include <stdio.h>
int main (int argc, char *argv[]) {
fprintf (stderr, "stderr (%s)\n", (argc > 1) ? argv[1] : "?");
return 0;
}
Then a control script test.csh which will show it in action:
#!/usr/bin/csh
ps -ef ; echo ; echo $$ ; echo
echo 'stdout (csh)'
./echo_err csh
bash -c "( echo 'stdout (bash)' ; ./echo_err bash ) 2>&1"
The echo of the PID and ps are simply so you can ensure it's csh running this script. When you run this script with:
./test.csh >test.out 2>test.err
(the initial redirection is set up by bash before csh starts running the script), and examine the out/err files, you see:
test.out:
UID PID PPID TTY STIME COMMAND
pax 5708 5364 cons0 11:31:14 /usr/bin/ps
pax 5364 7364 cons0 11:31:13 /usr/bin/tcsh
pax 7364 1 cons0 10:44:30 /usr/bin/bash
5364
stdout (csh)
stdout (bash)
stderr (bash)
test.err:
stderr (csh)
You can see there that the test.csh process is running in the C shell, and that calling bash from within there gives you the full bash power of redirection.
The 2>&1 in the bash command quite easily lets you redirect standard error to the current standard output (as desired) without prior knowledge of where standard output is currently going.
I object the above answer and provide my own. csh DOES have this capability and here is how it's done:
xxx |& some_exec # will pipe merged output to your some_exec
or
xxx |& cat > filename
or if you just want it to merge streams (to stdout) and not redirect to a file or some_exec:
xxx |& tee /dev/null
As paxdiablo said you can use >& to redirect both stdout and stderr. However if you want them separated you can use the following:
(command > stdoutfile) >& stderrfile
...as indicated the above will redirect stdout to stdoutfile and stderr to stderrfile.
xxx >& filename
Or do this to see everything on the screen and have it go to your file:
xxx | & tee ./logfile
What about just
xxx >& /dev/stdout
???
I think this is the correct answer for csh.
xxx >/dev/stderr
Note most csh are really tcsh in modern environments:
rmockler> ls -latr /usr/bin/csh
lrwxrwxrwx 1 root root 9 2011-05-03 13:40 /usr/bin/csh -> /bin/tcsh
using a backtick embedded statement to portray this as follows:
echo "`echo 'standard out1'` `echo 'error out1' >/dev/stderr` `echo 'standard out2'`" | tee -a /tmp/test.txt ; cat /tmp/test.txt
if this works for you please bump up to 1. The other suggestions don't work for my csh environment.
I am working on a program written by several folks with largely varying skill level. There are files in there that have never changed (and probably never will, as we're afraid to touch them) and others that are changing constantly.
I wonder, are there any tools out there that would look at the entire repo history (git) and produce analysis on how frequently a given file changes? Or package? Or project?
It would be of value to recognize that (for example) we spent 25% of our time working on a set of packages, which would be indicative or code's fragility, as compared with code that "just works".
If you're looking for an OS solution, I'd probably consider starting with gitstats and look at extending it by grabbing file logs and aggregating that data.
I'd have a look at NChurn:
NChurn is a utility that helps asses the churn level of your files in
your repository. Churn can help you detect which files are changed the
most in their life time. This helps identify potential bug hives, and
improper design.The best thing to do is to plug NChurn into your build
process and store history of each run. Then, you can plot the
evolution of your repository's churn.
I wrote something that we use to visualize this information successfully.
https://github.com/bcarlso/defect-density-heatmap
Take a look at the project and you can see what the output looks like in the readme.
You can do what you need by first getting a list of files that have changed in each commit from Git.
~ $ git log --pretty="format:" --name-only | grep -v ^$ > file-changes.txt
~ $ for i in `cat file-changes.txt | cut -d"." -f1,2 | uniq`; do num=`cat file-changes.txt | grep $i | wc -l`; if (( $num > 1 )); then echo $num,0,$i; fi; done | heatmap > results.html
This will give you a tag cloud with files that churn more will show up larger.
I suggest using a command like
git log --follow -p file
That will give you all the changes that happened to the file in the history (including renames). If you want to get the number of commits that changed the file then you can do on a UNIX-based OS :
git log --follow --format=oneline Gemfile | wc -l
You can then create a bash script to apply this to multiple files with the name aside.
Hope it helped !
Building on a previous answer I suggest the following script to parse all project files
#!/bin/sh
cd $1
find . -path ./.git -prune -o -name "*" -exec sh -c 'git log --follow --format=oneline $1 | wc -l | awk "{ print \$1,\"\\t\",\"$1\" }" ' {} {} \; | sort -nr
cd ..
If you call the script as file_churn.sh you can parse your git project directory calling
> ./file_churn.sh project_dir
Hope it helps.
I seem to remember being able to print out (or locate) the specific switches that each -O<num> option turns on. Can you remind?
Thanks!
The list of new features on gcc 4.3 shows a way to do it, via an extension to the --help command line option:
gcc -c -Q -O3 --help=optimizers > /tmp/O3-opts
gcc -c -Q -O2 --help=optimizers > /tmp/O2-opts
diff /tmp/O2-opts /tmp/O3-opts | grep enabled
Note, however that I never tried that, only read about it. The documentation about this command line option is at http://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html#Overall-Options
If you ever read the list of new features on gcc 4.3, perhaps this was what you were recalling.
You may also try the good ol' manual
$ man gcc
at the subsection "Options That Control Optimization".
On many machines, 'info gcc' will produce a wealth of information. Using 'gcc -v --help' produced a very long listing of options from sub-processes (actually, 1001 lines on stdout, and 14 on stderr) on my Mac (PPC G4 and MacOS X 10.4.11).