I ran bazel test syntaxnet/... util/utf8/... and it gave me this output:
FAIL: //syntaxnet:parser_trainer_test (see /home/me/.cache/bazel/_bazel_rushat/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/testlogs/syntaxnet/parser_trainer_test/test.log).
INFO: Elapsed time: 2179.396s, Critical Path: 1623.00s
//syntaxnet:arc_standard_transitions_test PASSED in 0.7s
//syntaxnet:beam_reader_ops_test PASSED in 24.1s
//syntaxnet:graph_builder_test PASSED in 14.6s
//syntaxnet:lexicon_builder_test PASSED in 6.1s
//syntaxnet:parser_features_test PASSED in 5.8s
//syntaxnet:reader_ops_test PASSED in 9.4s
//syntaxnet:sentence_features_test PASSED in 0.2s
//syntaxnet:shared_store_test PASSED in 41.7s
//syntaxnet:tagger_transitions_test PASSED in 5.2s
//syntaxnet:text_formats_test PASSED in 6.1s
//util/utf8:unicodetext_unittest PASSED in 0.4s
//syntaxnet:parser_trainer_test FAILED in 0.5s
/home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/testlogs/syntaxnet/parser_trainer_test/test.log
Executed 12 out of 12 tests: 11 tests pass and 1 fails locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
If you want the output of --test_verbose_timeout_warnings, please ask.
Test.log output is below because Stackoverflow tells me I have too much code in my post :/
Thanks!
test.log output:
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
+ BINDIR=/home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_trainer_test.runfiles/syntaxnet
+ CONTEXT=/home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_trainer_test.runfiles/syntaxnet/testdata/context.pbtxt
+ TMP_DIR=/tmp/syntaxnet-output
+ mkdir -p /tmp/syntaxnet-output
+ sed s=OUTPATH=/tmp/syntaxnet-output=
+ sed s=SRCDIR=/home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_trainer_test.runfiles= /home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_trainer_test.runfiles/syntaxnet/testdata/context.pbtxt
sed: can't read /home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_trainer_test.runfiles/syntaxnet/testdata/context.pbtxt: No such file or directory
+ PARAMS=128-0.08-3600-0.9-0
+ /home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_trainer_test.runfiles/syntaxnet/parser_trainer --arg_prefix=brain_parser --batch_size=32 --compute_lexicon --decay_steps=3600 --graph_builder=greedy --hidden_layer_sizes=128 --learning_rate=0.08 --momentum=0.9 --output_path=/tmp/syntaxnet-output --task_context=/tmp/syntaxnet-output/context --training_corpus=training-corpus --tuning_corpus=tuning-corpus --params=128-0.08-3600-0.9-0 --num_epochs=12 --report_every=100 --checkpoint_every=1000 --logtostderr
syntaxnet/parser_trainer_test: line 36: /home/me/.cache/bazel/_bazel_me/cc4d67663fbe887a603385d628fdf383/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_trainer_test.runfiles/syntaxnet/parser_trainer: No such file or directory
This is a bug in the syntaxnet test, it's looking for the wrong path. It needs the following patch:
diff --git a/syntaxnet/syntaxnet/parser_trainer_test.sh b/syntaxnet/syntaxnet/parser_trainer_test.sh
index ba2a6e7..977c89c 100755
--- a/syntaxnet/syntaxnet/parser_trainer_test.sh
+++ b/syntaxnet/syntaxnet/parser_trainer_test.sh
## -22,7 +22,7 ##
set -eux
-BINDIR=$TEST_SRCDIR/syntaxnet
+BINDIR=$TEST_SRCDIR/$TEST_WORKSPACE/syntaxnet
CONTEXT=$BINDIR/testdata/context.pbtxt
TMP_DIR=/tmp/syntaxnet-output
You need to have the correct bazel version installed for syntaxnet to compile. I had the latest build and it didn;t work. So I removed it by deleting the folder
rm -fr .cache/bazel
and then reinstalled the correct version bazel==0.2.2b by downloading the correct installer from the download page
and running it on my machine
sudo chmod +x bazel-version.sh
./bazel-version.sh --user
Related
This is an interesting issue. I have a GitLab project, and I've created a .gitlab-ci.yml to run a PMD that will scan my code after every commit. The ci.yml file looks like this:
image: "node:latest"
stages:
- preliminary-testing
apex-code-scan:
stage: preliminary-testing
allow_failure: false
script:
- install_java
- install_pmd
artifacts:
paths:
- pmd-reports/
####################################################
# Helper Methods
####################################################
.sfdx_helpers: &sfdx_helpers |
function install_java() {
local JAVA_VERSION=11
local JAVA_INSTALLATION=openjdk-$JAVA_VERSION-jdk
echo "Installing ${JAVA_INSTALLATION}"
apt update && apt -y install $JAVA_INSTALLATION
}
function install_pmd() {
local PMD_VERSION=6.52.0
local RULESET_PATH=ruleset.xml
local OUTPUT_DIRECTORY=pmd-reports
local SOURCE_DIRECTORY=force-app
local URL=https://github.com/pmd/pmd/releases/download/pmd_releases%2F$PMD_VERSION/pmd-bin-$PMD_VERSION.zip
# Here I would download and unzip the PMD source code. But for now I have the PMD source already in my project for testing purposes
# apt update && apt -y install unzip
# wget $URL
# unzip -o pmd-bin-$PMD_VERSION.zip
# rm pmd-bin-$PMD_VERSION.zip
echo "Installed PMD!"
mkdir -p $OUTPUT_DIRECTORY
echo "Going to run PMD!"
ls
echo "Start"
pmd-bin-$PMD_VERSION/bin/run.sh pmd -d $SOURCE_DIRECTORY -R $RULESET_PATH -f xslt -P xsltFilename=pmd_report.xsl -r $OUTPUT_DIRECTORY/pmd-apex.html
echo "Done"
rm -r pmd-bin-$PMD_VERSION
echo "Remove pmd"
}
before_script:
- *sfdx_helpers
When I try to run this pipeline, it will fail after starting the PMD:
However, if I make a small change to the PMD's .sh file and add an echo command at the very end. Then the pipeline succeeds:
PMD /bin/run.sh before (doesn't work):
...
java ${HEAPSIZE} ${PMD_JAVA_OPTS} $(jre_specific_vm_options) -cp "${classpath}" "${CLASSNAME}" "$#"
PMD /bin/run.sh after (does work):
...
java ${HEAPSIZE} ${PMD_JAVA_OPTS} $(jre_specific_vm_options) -cp "${classpath}" "${CLASSNAME}" "$#"
echo "Done1" // This is the last line in the file
I don't have the slightest idea why this is the case. Does anyone know why adding this echo command at the end of the .sh file would cause the pipeline to succeed? I could keep it as is with the echo command, but I would like to understand why it is behaving this way. I don't want to be that guy that just leaves a comment saying Hey don't touch this line of code, I don't know why, but without it the whole thing fails. Thank you!
PMD exits with a specific exit code depending whether it found some violations or not, see https://pmd.github.io/latest/pmd_userdocs_cli_reference.html#exit-status
I guess, your PMD run finds some violations, and PMD exits with exit code 4 - which is not a success exit code.
In general, this is used to make the CI build fail, in case any PMD violations are present - forcing to fix the violations before you get a green build.
If that is not what you want, e.g. you only want to report the violations but not fail the build, then you need to add the following command line option:
--fail-on-violation false
Then PMD will exit with exit code 0, even when there are violations.
So it appears that the java command that the PMD runs for some reason returns a non-zero exit code (even though the script is successful). Because I was adding an echo command at the end of that bash script, the last line in the script returned a success exit code, which is why the GitLab CI pipeline succeeded when the echo command was there.
In order to work around the non-zero exit code being returned by the java PMD command, I have changed this line in my .gitlab-ci.yml file to catch the non-zero exit code and proceed.
function install_pmd() {
// ... For brevity I'm just including the line that was changed in this method
pmd-bin-$PMD_VERSION/bin/run.sh pmd -d $SOURCE_DIRECTORY -R $RULESET_PATH -f xslt -P xsltFilename=pmd_report.xsl -r $OUTPUT_DIRECTORY/pmd-apex.html || echo "PMD Returned Exit Code"
// ...
}
I have a CI stage with the following command, which has to be executed remotely and checks if the mentioned file exists, if yes it creates a backup for it.
script: |
ssh ${USER}#${HOST} '([ -f "${PATH}/test_1.txt" ] && cp -v "${PATH}/test_1.txt" ${PATH}/test_1_$CI_COMMIT_TIMESTAMP.txt)'
The issue is, this job always fails whether the file exists or not with the following output:
ssh user#hostname '([ -f /etc/file/path/test_1.txt ] && cp -v /etc/file/path/test_1.txt /etc/file/path/test_1_$CI_COMMIT_TIMESTAMP.txt)'
Cleaning up project directory and file based variables
ERROR: Job failed: exit status 1
Running the same command manually, just works fine. So,
How can I make sure that this job succeeds as long as command logic is executed successfully and only fail incase there are some genuine failures?
There is no way for the job to know if the command you ran remotely worked or not. It can only know if the ssh instruction worked or not. You can force it to always succeed by appending || true to any instruction.
However, if you want to see and save the output of your remote instruction, you can do something like this:
ssh user#host command 2>&1 | tee ssh-session.log
I'm trying to debug a CI pipeline and want to create a custom logger stage that dumps a bunch of information about the environment in which the pipeline is running.
I tried adding this:
stages:
- logger
logger-commands:
stage: logger
allow_failure: true
script:
- echo 'Examining environment'
- echo PWD=$(pwd) Using image ${CI_JOB_IMAGE}
- git --version
- echo --------------------------------------------------------------------------------
- env
- echo --------------------------------------------------------------------------------
- npm --version
- node --version
- echo java -version
- mvn --version
- kanico --version
- echo --------------------------------------------------------------------------------
The problem is that the Java command is failing because java isn't installed. The error says:
/bin/sh: eval: line 217: java: not found
I know I could remove the line java -version, but I'm trying to come up with a canned logger that I could use in all my CI-Pipelines, so it would include: Java, Maven, Node, npm, python, and whatever else I want to include and I realize that some of those commands will fail because some of the commands are not found.
Searching for the above solution got me close.
GitLab CI: How to continue job even when script fails - Which did help. By adding allow_failure: true I found that even if the logger job failed the remaining stages would run (which is desirable). The answer also suggests a syntax to wrap commands in which is:
./script_that_fails.sh > /dev/null 2>&1 || FAILED=true
if [ $FAILED ]
then ./do_something.sh
fi
So that is helpful, but my question is this.
Is there anything built into gitlab's CI-pipeline syntax (or bash syntax) that allows all commands in a given step to run even if one command fails?
Is it possible to allow for a script in a CI/CD job to fail? - suggests adding the UNIX bash OR syntax as shown below:
- npm --version || echo nmp failed
- node --version || echo node failed
- echo java -version || echo java failed
That is a little cleaner (syntax) but I'm trying to make it simpler.
The answers already mentioned are good, but I was looking for something simpler so I wrote the following bash script. The script always returns a zero exit code so the CI-pipeline always thinks the command was successful.
If the command did fail, the command is printed along with the non-zero exit code.
# File: runit
#!/bin/sh
"$#"
EXITCODE=$?
if [ $EXITCODE -ne 0 ]
then
echo "CMD: $#"
echo "Ignored exit code ($EXITCODE)"
fi
exit 0
Testing it as follows:
./runit ls "/bad dir"
echo "ExitCode = $?"
Gives this output:
ls: cannot access /bad dir: No such file or directory
CMD: ls /bad dir
Ignored exit code (2)
ExitCode=0
Notice even though the command failed the ExitCode=0 shows what the ci-pipeline will see.
To use it in the pipeline, I have to have that shell script available. I'll research how to include it, but it must be in the CI runner job. For example,
stages:
- logger-safe
logger-safe-commands:
stage: logger-safe
allow_failure: true
script:
- ./runit npm --version
- ./runit java -version
- ./runit mvn --version
I don't like this solution because it requires extra file in the repo but this is in the spirit of what I'm looking for. So far the simplest built in solution is:
- some_command || echo command failed $?
I am trying to run gem5 in FS mode by using command as : "build/ARM/gem5.opt configs/example/fs.py --disk-image=/home/coep/gem5%202/full_system_images/aarch32-ubuntu-natty-headless.img --arm=/home/coep/gem5 2/full_system_images/vmlinux.arm.smp.fb.3.2/vmlinux.arm.smp.fb.3.2"
and getting error as : "Usage: fs.py [options] fs.py: error: option --arm-iset: invalid choice: '/home/coep/gem5' (choose from 'arm', 'thumb', 'aarch64')"
please help me to solve this error.
Thank you.
I assume the --arm=/home/coep/gem5...vmlinux.arm.smp.fb.3.2 argument specifies the path to the guest kernel, in which case it should be --kernel=...:
build/ARM/gem5.opt \
configs/example/fs.py \
--disk-image=/home/coep/gem5\ 2/full_system_images/aarch32-ubuntu-natty-headless.img \
--kernel=/home/coep/gem5\ 2/full_system_images/vmlinux.arm.smp.fb.3.2/vmlinux.arm.smp.fb.3.2
Arguments and their explanations are found in configs/common/Options.py
There can be multiple reasons why are getting this error, One of them can be an incorrect path to the disk image files.
I have run the gem5 in the FS mode and have booted Linux on top of it on Ubuntu 18.04 LTS
You can follow the below steps, the first step is to download and install the full-system binary and disk image files.
1. $ mkdir full_system_image
2. $ cd full_system_image/
3. $ wget http://www.m5sim.org/dist/current/arm/aarch-system-2014-10.tar.bz2
4. $ tar jxf aarch-system-2014-10.tar.bz2
5. $ echo "export M5_PATH=/Path to the full_system_image directory/full_system_images/" >> ~/.bashrc
6. $ source ~/.bashrc
7. $ echo $M5_PATH (- check if the path is set correct)
Now the path has been set, the next step is to run the gem5 in FS mode.
1. connect to gem5 base directory
2. $ ./build/ARM/gem5.opt configs/example/fs.py --disk-image=/home/full_system_image/disks/aarch32-ubuntu-natty-headless.img
3. Note: --disk-image=path to the full_system_image/disks/aarch32-ubuntu-natty-headless.img
4. open a new terminal and listen to port 3456
5. $ telnet localhost 3456
6. Here 3456 is a port number on the gem5 terminal
7. this will take around 30 mins depending on the machine performance.
8. After this, at the end you will get something like this
input: AT Raw Set 2 keyboard as /devices/smb.14/motherboard.15/iofpga.17/1c060000.kmi/serio0/input/input0
input: touchkitPS/2 eGalax Touchscreen as
/devices/smb.14/motherboard.15/iofpga.17/1c070000.kmi/serio1/input/input2
kjournald starting. Commit interval 5 seconds
EXT3-fs (sda1): using internal journal
EXT3-fs (sda1): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) on device 8:1.
Freeing unused kernel memory: 292K (806aa000 - 806f3000)
random: init urandom read with 14 bits of entropy available
Ubuntu 11.04 gem5sim ttySA0
9. login as root
Voila, you have run the gem5 in FS mode.
I have a simple rule to generate a file in Snakemake. Running snakemake results in an immediate error that it cannot find the generated file, even when --latency-wait is specified as a command line option.
However, this does seem to be a latency-related issue, as this Snakefile runs without problems on a local machine. The output below is on a system that has known latency problems.
Contents of Snakefile:
rule generate_file:
output:
"dummy.txt"
shell:
"head --bytes 1024 < /dev/zero | base64 > '{output}'; ls"
Commands:
$ snakemake --version
5.2.0
$ snakemake -p --latency-wait 10
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 generate_file
1
rule generate_file:
output: dummy.txt
jobid: 0
head --bytes 1024 < /dev/zero | base64 > 'dummy.txt'; ls
dummy.txt Snakefile
MissingOutputException in line 1 of /home/user/project/Snakefile:
[Errno 2] No such file or directory: ''
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Removing output files of failed job generate_file since they might be corrupted:
dummy.txt
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/user/project/.snakemake/log/2018-08-08T101648.774072.snakemake.log
Interestingly, the ls command shows the file is created and visible.
Your rule creates output file dummy.txt when used with snakemake version 5.2.2 and linux, and snakemake ends successfully. Perhaps it is a bug in version 5.2.0? I don't see anything about it in change logs though.
On related note, use of head in shell command used to result in non-zero exit status error. Apparently recent version behaves differently in this respect.