Maven + Mercurial for Build Numbers - maven-2

I can't figure out how to get a Mercurial revision id put into my Maven build (ideally I would like it in the MANIFEST of my jars and war).
The closest solution I could find is:
mvn -DbuildNumber=`hg id -i`
Which won't really work for Windows or my Hudson server.
Luckily Hudson tags my builds but I would like some more assurance if the builds were also tagged with the Mercurial changset id.

Have a look at this previous question and the link from the accepted answer. Basically, you want to do the same thing except that you'll want to use the buildnumber:hgchangeset goal with Mercurial to get a changeset property with the content of hg id -i.

Unfortunately, hg id -i is too long for use. I created a script that will calculate an accurate build number. However, there are two exceptions. If there was not previous release on the branch, then it cannot be valid. If there are changes in the local repo, then it cannot be valid. In my build script I mark the build as "x.x.UNSTABLE" whenever that happens.
I use a REL_PATTERN to pick up the last tag in the current branch that was marked as an actual release. Then I calculate the build number by tracking the commit log count from that release + all commits to the branch since that release.
#!/bin/bash
REL_PATTERN="release-[0-9]*\.[0-9]*\.[0-9]*"
BRANCH=$( hg branch )
CURR_REV=$( hg id -n )
if [ "${CURR_REV: -1}" = "+" ] ; then
echo "ERROR: This workspace contains uncommitted code. Cannot calculate build number" >&2
echo "UNSTABLE"
exit 1
fi
RELEASE=$( hg log --rev="branch($BRANCH) and tag() and 1:$CURR_REV" -T "{tags} {rev}\n"|grep "${REL_PATTERN} "|tail -1 )
if [ "$RELEASE" = "" ] ; then
echo "ERROR: Unable to locate version tag" >&2
echo "UNSTABLE"
exit 1
fi
RELEASE_REV=$( echo $RELEASE|cut -f 2 -d ' ' )
RELEASE_TAG=$( echo $RELEASE|cut -f 1 -d ' ' )
REVS=$( hg log -P $RELEASE_REV -b $BRANCH -T "{rev}\n"|wc -l )
BUILD=$( hg log -r1:$CURR_REV -P $RELEASE_REV -b $BRANCH -T "{rev}\n"|wc -l )
echo "BRANCH=$BRANCH" >&2
echo "CURR_REV=$CURR_REV" >&2
echo "RELEASE_REV=$RELEASE_REV" >&2
echo "RELEASE_TAG=$RELEASE_TAG" >&2
echo "BUILD=$BUILD" >&2
echo $BUILD

Related

Running PMD in GitLab CI Script doesn't work unless echo command is added after the script runs

This is an interesting issue. I have a GitLab project, and I've created a .gitlab-ci.yml to run a PMD that will scan my code after every commit. The ci.yml file looks like this:
image: "node:latest"
stages:
- preliminary-testing
apex-code-scan:
stage: preliminary-testing
allow_failure: false
script:
- install_java
- install_pmd
artifacts:
paths:
- pmd-reports/
####################################################
# Helper Methods
####################################################
.sfdx_helpers: &sfdx_helpers |
function install_java() {
local JAVA_VERSION=11
local JAVA_INSTALLATION=openjdk-$JAVA_VERSION-jdk
echo "Installing ${JAVA_INSTALLATION}"
apt update && apt -y install $JAVA_INSTALLATION
}
function install_pmd() {
local PMD_VERSION=6.52.0
local RULESET_PATH=ruleset.xml
local OUTPUT_DIRECTORY=pmd-reports
local SOURCE_DIRECTORY=force-app
local URL=https://github.com/pmd/pmd/releases/download/pmd_releases%2F$PMD_VERSION/pmd-bin-$PMD_VERSION.zip
# Here I would download and unzip the PMD source code. But for now I have the PMD source already in my project for testing purposes
# apt update && apt -y install unzip
# wget $URL
# unzip -o pmd-bin-$PMD_VERSION.zip
# rm pmd-bin-$PMD_VERSION.zip
echo "Installed PMD!"
mkdir -p $OUTPUT_DIRECTORY
echo "Going to run PMD!"
ls
echo "Start"
pmd-bin-$PMD_VERSION/bin/run.sh pmd -d $SOURCE_DIRECTORY -R $RULESET_PATH -f xslt -P xsltFilename=pmd_report.xsl -r $OUTPUT_DIRECTORY/pmd-apex.html
echo "Done"
rm -r pmd-bin-$PMD_VERSION
echo "Remove pmd"
}
before_script:
- *sfdx_helpers
When I try to run this pipeline, it will fail after starting the PMD:
However, if I make a small change to the PMD's .sh file and add an echo command at the very end. Then the pipeline succeeds:
PMD /bin/run.sh before (doesn't work):
...
java ${HEAPSIZE} ${PMD_JAVA_OPTS} $(jre_specific_vm_options) -cp "${classpath}" "${CLASSNAME}" "$#"
PMD /bin/run.sh after (does work):
...
java ${HEAPSIZE} ${PMD_JAVA_OPTS} $(jre_specific_vm_options) -cp "${classpath}" "${CLASSNAME}" "$#"
echo "Done1" // This is the last line in the file
I don't have the slightest idea why this is the case. Does anyone know why adding this echo command at the end of the .sh file would cause the pipeline to succeed? I could keep it as is with the echo command, but I would like to understand why it is behaving this way. I don't want to be that guy that just leaves a comment saying Hey don't touch this line of code, I don't know why, but without it the whole thing fails. Thank you!
PMD exits with a specific exit code depending whether it found some violations or not, see https://pmd.github.io/latest/pmd_userdocs_cli_reference.html#exit-status
I guess, your PMD run finds some violations, and PMD exits with exit code 4 - which is not a success exit code.
In general, this is used to make the CI build fail, in case any PMD violations are present - forcing to fix the violations before you get a green build.
If that is not what you want, e.g. you only want to report the violations but not fail the build, then you need to add the following command line option:
--fail-on-violation false
Then PMD will exit with exit code 0, even when there are violations.
So it appears that the java command that the PMD runs for some reason returns a non-zero exit code (even though the script is successful). Because I was adding an echo command at the end of that bash script, the last line in the script returned a success exit code, which is why the GitLab CI pipeline succeeded when the echo command was there.
In order to work around the non-zero exit code being returned by the java PMD command, I have changed this line in my .gitlab-ci.yml file to catch the non-zero exit code and proceed.
function install_pmd() {
// ... For brevity I'm just including the line that was changed in this method
pmd-bin-$PMD_VERSION/bin/run.sh pmd -d $SOURCE_DIRECTORY -R $RULESET_PATH -f xslt -P xsltFilename=pmd_report.xsl -r $OUTPUT_DIRECTORY/pmd-apex.html || echo "PMD Returned Exit Code"
// ...
}

Is there a simple test in Bash to test if a logical path is a directory

The question is best explained through a small example:
Create an infrastructure of directories as follows:
rm -rf /tmp/work
mkdir -p /tmp/work/a/b/c
cd /tmp/work
ln -s a/b/c s
mkdir t
tree
This results in the following infrastructure:
There is a very simple way to check for the existence of a directory:
cd /tmp/work
if test -d s/../t; then echo EXISTS; else echo DOES NOT EXIST; fi
However, it seems that the '-d' test only checks for the physical path (a/b/t); The following clearly shows that the logical path/directory does exist:
cd s/../t
pwd
I am hoping for a simple test to get this situation checked...
Checking for similar questions in StackOverflow, I haven't found an answer that differentiates logical paths from physical paths...
I did a brute force (silly) attempt to get some more insight:
for o in -b -c -d -e -f -g -G -k -h -L -O -p -r -S -s -t -u -w -x; do
if test $o s/../t; then echo EXISTS; else echo DOES NOT EXIST; fi
done
As expected, all of the tests indicate failure...
The closest I get to a simple solution is the following test:
if (unset CDPATH; cd -L s/../t &>/dev/null); then echo EXISTS; else echo DOES NOT EXIST; fi
If using '-P' rather than '-L', then this syntax tests for physical paths (the equivalent of 'test -d').
Note that if we're sure that we're using builtins, then the syntax is just a bit simpler:
if (unset CDPATH; cd s/../t &>/dev/null); then echo EXISTS; else echo DOES NOT EXIST; fi
But there must be a better way...

Why does this fish while loop terminate after a single iteration?

This is a fish function used to upgrade JavaScript packages for a project. Strangely, it terminates after a single iteration with an exit status of 0. Why?
function yarn-upgrade-all --description "Upgrade JavaScript packages"
yarn outdated | sed '1,/^Package/d;/^Done/d' | awk '{print $1, $4}' | while read -l PACKAGE VERSION
echo
set_color brwhite
echo -n "==>"
set_color yellow
echo -n " "$PACKAGE
set_color brblue
echo -n " "$VERSION
set_color brwhite
echo -n " <=="
set_color normal
echo
echo
yarn upgrade --latest $PACKAGE
and yarn run test
and yarn run build
and git commit -am "Upgrade to "$PACKAGE" "$VERSION
or begin
set_color red
echo "last command exited with status $status" >&2
set_color normal
return 1
end
end
end
On the other hand, this second function, which contains only a stub body, runs through all packages piped into the loop.
function yarn-upgrade-all-debug --description "Upgrade JavaScript packages"
yarn outdated | sed '1,/^Package/d;/^Done/d' | awk '{print $1, $4}' | while read -l PACKAGE VERSION
echo $PACKAGE $VERSION
end
end
fish --version
fish, version 3.0.2
You're running fish 3.0.0, and hitting https://github.com/fish-shell/fish-shell/issues/5513 - return from a while doesn't actually set the status correctly.
However, the return still causes it to terminate the while-loop.
Upgrade to 3.0.2.
The loop terminates after a single iteration because the yarn run invocations in the loop body slurp up the rest of stdin. (Credits to #glenn-jackman.)
A possible workaround is to redirect stdin to /dev/null for these commands:
and yarn run test < /dev/null
and yarn run build < /dev/null
The culprit is run-s from the npm-run-all package, which is invoked by both yarn run commands.
https://github.com/mysticatea/npm-run-all/issues/166

Busybox syslog enable/disable klogd

I know I can configure busybox to be built with klogd (CONFIG_KLOGD) and it works out of the box.
I'd like to know, though, if I can enable/disable this feature at run-time by a command-line switch or syslog-startup.conf option without need to recompile busybox.
Thanks in advance.
Well, I really rather have anyone else's response but since no one else came, I think my own solution is better than no solution, just in case someone else needs it.
I have dug into the sources and it looks like there is no command-line switch or config option to selectively enable/disable klogd.
The init-script, though, provides a very nice way of doing that.
1.Start/Stop of syslogd and klogd are separated lines:
start)
echo -n "Starting syslogd/klogd: "
start-stop-daemon -S -b -n syslogd -a /sbin/syslogd -- -n $SYSLOG_ARGS
start-stop-daemon -S -b -n klogd -a /sbin/klogd -- -n
echo "done"
The ugliest think one could think now is that, if klogd is not to be started, simply comment that line out.
/etc/syslog-startup.conf is simply sourced by /etc/init.d/syslog, before the start) case:
(basically the first thing in script)
if [ -f /etc/syslog-startup.conf ]; then
. /etc/syslog-startup.conf
Well, I can add a variable to /etc/syslog-startup.conf that I can test in the init script to selectively enable/disable the start-stop-daemon ... klog.
start)
echo -n "Starting syslogd/klogd: "
start-stop-daemon -S -b -n syslogd -a /sbin/syslogd -- -n $SYSLOG_ARGS
[ "$KLOGD_DISABLE" = "yes" ] || \
start-stop-daemon -S -b -n klogd -a /sbin/klogd -- -n
echo "done"
Since I am adding a non-standard variable to a config file, it is still very ugly but I feel it less ugly than the first one, so that's what I propose as solution.

How to ignore certain files when branching / checking out?

I'd like to compare a few files from the bazaar branch lp:ubuntu/nvidia-graphics-drivers. I'm mainly interested in the debian subdirectory inside that branch, but due to the binary blob in http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/nvidia-graphics-drivers/oneiric/files, it takes ages to get just the text files. I've already downloaded 555MB and it's still counting.
Is it possible to retrieve a bazaar branch, including or excluding certain files by one of the following properties:
file size
file extension
file name (include only debian/ for example)
I do not need to push back any changes, nor do I need to view the history of a file. I just want to compare two files in the debian/ directory, files with the .in extension and files without.
As far as I'm aware, no. You're downloading the branch history, not just the individual files. And each file is an integral part of the branch's history.
On the bright side, you only have to check it out once. Unless those binary files change, they'll be skipped the next time you pull from Launchpad.
Depending on the branch's history, you may be able to cut down on the download size if you use a lightweight checkout (bzr checkout --lightweight). But of course, that may come back and bite you later, as it means you won't get a local copy of the branch, only the checked-out files. So it'll work much like SVN, where every operation has to go through the server. And as long as you don't need to look at the branch history, or commit your changes, that should serve you just fine, I believe.
I ended up doing some dirty grep-ing on the HTTP response since bzr info "$branch" and bzr ls -d "$branch" "$directory" did not provide enough information to me.
The below Bash script relies on the working of Launchpads front-end Loggerhead. It recursively downloads from a given URL. Currently, it ignores *.run files. Save it as bzrdl in a directory available from $PATH and run it with bzrdl http://launchpad.net/~ubuntu-branches/ubuntu/oneiric/nvidia-graphics-drivers/oneiric/files/head:/debian/. All files will be saved in the current directory, be sure that it's empty to avoid conflicts.
#!/bin/bash
max_retries=5
rooturl="$1"
if ! [[ $rooturl =~ /$ ]]; then
echo "Usage: ${0##*/} URL"
echo "URL must end with a slash. Example URL:"
echo "http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/nvidia-graphics-drivers/oneiric/files/head:/"
exit 1
fi
tmpdir="$(mktemp -d)"
target="$(pwd)"
# used for holding HTTP response before extracting data
tmp="$(mktemp)"
# url_filter reads download URLs from stdin (piped)
url_filter() {
grep -v '\.run$'
}
get_files_from_dir() {
local slash=/
local dir="$1"
# to avoid name collision: a/b/c/ -> a.d/b.d/c.d/
local storedir="${dir//$slash/.d${slash}}"
mkdir -p "$tmpdir/$storedir" "$target/$dir"
local i subdir
for ((i=0; i<$max_retries; i++ )); do
if wget -O "$tmp" "$rooturl$dir"; then
# store file list
grep -F -B 1 '<img src="/static/images/ico_file_download.gif" alt="Download File" />' "$tmp" |\
grep '^<a' | cut -d '"' -f 2 | url_filter \
> "$tmpdir/$storedir/files"
IFS=$'\n'
for subdir in $(grep -F -B 1 '<img src="/static/images/ico_folder.gif" ' "$tmp" | \
grep -F '<a ' | rev | cut -d / -f 2 | rev); do
IFS=$' \t\n'
get_files_from_dir "$dir$subdir/"
done
return
fi
done
echo "Failed to download directory listing of: $dir" >> "$tmpdir/errors"
}
download_files() {
local slash=/
local dir="$1"
# to avoid name collision: a/b/c/ -> a.d/b.d/c.d/
local storedir="${dir//$slash/.d${slash}}"
local done=false
local subdir
cd "$tmpdir/$storedir"
for ((i=0; i<$max_retries; i++)); do
if wget -B "$rooturl$dir" -nc -i files -P "$target/$dir"; then
done=true
break
fi
done
$done || echo "Failed to download all files from $dir" >> "$tmpdir/errors"
for subdir in *.d; do
download_files "$dir${subdir%%.d}/"
done
}
get_files_from_dir ''
# make *.d expand to nothing if no directories are found
shopt -s nullglob
download_files ''
echo "TMP dir: $tmpdir"
echo "Errors : $(wc -l "$tmpdir/errors" 2>/dev/null | cut -d ' ' -f 2 || echo 0)"
The temporary directory and file is not removed afterwards, that must be done manually. Any errors (failures to download) will be written to $tmpdir/errors
It's confirmed to work with:
bzrdl http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/nvidia-settings/oneiric/files/head:/debian/
Feel free to correct any mistakes or add improvements.
There is no way to selectively check out a specific directory from a Bazaar branch at the moment, although we do have plans to add such support in the future.
There is definitely too much traffic for the clone you are doing, considering the size of the branch. It's probably a bug in the client implementation.
Here on bzr 2.4 it is still quite slow but not too bad (60s):
localhost:/tmp% bzr branch http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/nvidia-settings/oneiric
Most recent Ubuntu Oneiric version: 275.09.07-0ubuntu1
Packaging branch status: CURRENT
Branched 37 revision(s).
From the log:
[11866] 2011-07-31 00:56:57.007 INFO: Branched 37 revision(s).
56.786 Transferred: 5335kB (95.8kB/s r:5314kB w:21kB)