My DBT project is highly dependent on receiving variables at runtime. When I pass these variables using the below command I get an error from DBT. The command complies 100% with the documentation . I have no idea why DBT will reject this command
dbt parse --profiles-dir . --target dev --vars '{"client_name": "ecc6_test_client","fiscal_start_month_number": "3","fiscal_year": "2020"}'
Here is the error:
...
dbt: error: unrecognized arguments: ecc6_test_client,fiscal_start_month_number: 3,fiscal_year: 2020}'
These variables are also defined in my dbt_project.yml, but I need to override them at runtime
vars:
# Client name
client_name: "ecc6_test_client_1"
# Fiscal period start month as number
fiscal_start_month_number: "1"
# Fiscal year
fiscal_year: "2000"
# Source Schema
source_schema: "ecc6_test_client_1_2000"
When I run dbt without the --vars it executes without any issue
Solution mentioned in https://github.com/dbt-labs/docs.getdbt.com/issues/620
Change the quotes, add the dictionary string within double quotes
dbt parse --profiles-dir . --target dev --vars "{client_name: ecc6_test_client,fiscal_start_month_number: 3,fiscal_year: 2020}"
Related
dbt run
21:58:09 Running with dbt=1.1.0
21:58:09 Unable to do partial parsing because a project config has changed
21:58:10 Encountered an error:
string indices must be integers
running into above error when I try to run dbt. Appreciate your inputs !
I have the usual section
variables:
DESTINATION: "dev"
dev-SERVERNUMBERS: "0 1"
And, this had been working all long and suddenly. Gitlab CI is complaining and giving the following error message
/bin/bash: line 103: export: `dev-SERVERNUMBERS=0 1': not a valid identifier
Initially, I thought I was exporting something wrong. But, it looks like that is how gitlab runners work they export your variables to make them available to all jobs within your pipeline. I have NO export statement within any before_script.
I can understand why it is complaining because it expects the variable to have double quotes around it in order to export. But, it DID NOT complain now for atleast 2 months, so I cannot understand why now ?
I find gitlab-ci just so unpredictable and extremely frustrating to use. By now, if I had automated this build of mine via shell scripts I would have be on my way. And, I have been stuck on this error for 3 days now. It does not even print out any debug echos that I have set up and straight away gives this line number which has nothing to do with my .gitlab-ci.yaml
At the very least it should give the verbose command that is trying to execute.
Bash is complaining about the environment variable name (the identifier) not necessarily the value. - is not an allowed character in environment variable names. You can only use alphanumerics (a-Z, 0-9), and underscores (_).
This is a limitation of bash, not GitLab CI.
export foo="0 1" # OK
export foo-bar="0 1" # Not OK
bash: export: `foo-bar=0 1': not a valid identifier
I have pipeline on GitLab and there the variable - ENV_VAR. This variable is changing based on branch for pipeline.
In the same yml file I have script with newman, where I want to pass this variable like this -> newman run ... -e test/apis/$ENV_VAR_environment.json
But the issue I have right now is that it seems the variable is not being passed as i want. The pipeline shows error - cannot read the test/apis/here_should_be_the_variable_name.json
Is there a way to pass this variable into the file source?
It looks like you only need to enclose the variable name in braces:
-e test/apis/${ENV_VAR}_environment.json
because test/apis/$ENV_VAR_environment.json means that it looks for $ENV_VAR_environment variable which obviously does not exist.
I have a CMake script that runs some tests via add_test(), running under Windows (Server 2008, don't ask) in CMake 3.15. When these tests are called, the PYTHONPATH environment variable in the environment they run in seems to get reset to the environment default, and doesn't contain some paths that it needs to.
I therefore need to set PYTHONPATH when the tests are run to the value of the $ENV{PYTHONPATH} variable when CMake runs. This has a number of semicolon-separated paths, so CMake thinks it's a list and tries to expand it into a number of space-separated strings, which obviously ends badly.
I cannot work out how to stop CMake doing this. From everything I can see, you should be able to do just surround with quotes:
add_test(
NAME mytest
COMMAND cmake -E env PYTHONPATH="$ENV{PYTHONPATH}"
run_test_here)
...but it always does the expansion. I also tried setting with set_tests_properties:
set_tests_properties(mytest PROPERTIES
ENVIRONMENT PYTHONPATH="$ENV{PYTHONPATH}")
...but that didn't appear to do anything at all - PYTHONPATH at test time wasn't altered. I thought it was because it's an environment variable, but using a regular CMake variable via set() makes no difference, so I'm doing something wrong. Help please!
The following should work:
COMMAND cmake -E env "PYTHONPATH=$ENV{PYTHONPATH}"
You need to quote the full part of the command line, to make properly expanded message.
Tested with:
set(tmp "C:\\Python27\\Scripts;E:\\JenkinsMIDEBLD\\workspace\\...;...")
add_test(NAME MYtest1 COMMAND cmake -S . -E env "tmp=${tmp}")
add_test(NAME MYtest2 COMMAND cmake -S . -E env tmp="${tmp}")
After running ctest I get:
1: Test command: /bin/cmake "-S" "." "-E" "env" "tmp=C:\Python27\Scripts;E:\JenkinsMIDEBLD\workspace\...;..."
2: Test command: /bin/cmake "-S" "." "-E" "env" "tmp="C:\Python27\Scripts" "E:\JenkinsMIDEBLD\workspace\..." "...""
The first test has proper ; passed to var, while the second one passes space separated list.
This is how cmake parses quoted arguments. An argument is either fully quoted or not quoted at all - partial quotes are interpreted as a literal ". So assumnig that:
set(var a;b;c)
The following:
var="$var"
Is not a quoted argument and " are taken literally! It expands the $var list into space separated list and the " stay, there is one " between = and a, and there is additional " on the end. The var="$var" is equal to:
var=\"a b c\"
^^ ^^ - the quotes stay!
^^^^^^^ ^ ^^^ - these are 3 arguments, the last one is `c"`
Without quotes is:
var=$var
is equal to (notice the missing quotes):
var=a c c
To quotes argument you have to quote it all, with first and last character of the element beeing ":
"var=$var"
will expand to:
"var=a;b;c"
You can make this work with the ENVIRONMENT test property, but there's a catch:
Semicolon separates different environment variables to set; you need to escape semicolons in your environment variable. For example instead of
set_tests_properties(mytest PROPERTIES
ENVIRONMENT "PYTHONPATH=foo;bar")
you need to use
set_tests_properties(mytest PROPERTIES
ENVIRONMENT "PYTHONPATH=foo\\;bar")
The fact that an environment variable may contain semicolons makes some transformation necessary: since ; is used to separate list elements you can simply use list(JOIN) to replace those with "\\;".
The following example works with PATH, not PYTHONPATH, since I don't have python installed:
CMakeLists.txt
cmake_minimum_required(VERSION 3.12.4) # required for list(JOIN)
project(TestProject)
# get a version of the PATH environment var that can be used in the ENVIRONMENT test property
set(_PATH $ENV{PATH})
list(JOIN _PATH "\\;" _PATH_CLEAN)
# just use a cmake script so we don't need to require any program able to retrieve environment vars
add_test(NAME Test1 COMMAND "${CMAKE_COMMAND}" -P "${CMAKE_CURRENT_SOURCE_DIR}/test_script.cmake")
# we add another simpler var to test in the cmake script
set_tests_properties(Test1 PROPERTIES ENVIRONMENT "PATH=${_PATH_CLEAN};FOO=foo")
enable_testing()
test_script.cmake
message(STATUS "PATH=$ENV{PATH}")
if (NOT "$ENV{FOO}" STREQUAL "foo")
# the following command results in a non-0 exit code, if executed
message(FATAL_ERROR "FOO environment var should contain \"foo\" but contains \"$ENV{FOO}\"")
endif()
When defining a Bamboo plan variable, the page has this.
For task configuration fields, use the syntax
${bamboo.myvariablename}. For inline scripts, variables are exposed as
shell environment variables which can be accessed using the syntax
$BAMBOO_MY_VARIABLE_NAME (Linux/Mac OS X) or %BAMBOO_MY_VARIABLE_NAME%
(Windows).
However, that doesn't work in my Linux inline script. For example, I have the following defined a a plan variable
name: my_plan_var value: some_string
My inline script is simply...
PLAN_VAR=$BAMBOO_MY_PLAN_VAR
echo "Plan var: $PLAN_VAR"
and I just get a blank string.
I've tried this
PLAN_VAR=${bamboo.my_plan_var}
But I get
${bamboo.my_plan_var}: bad substitution
on the log viewer window.
Any pointers?
I tried the following and it works:
On the plan, I set my_plan_var to "it works" (w/o quotes)
In the inline script (don't forget the first line):
#/bin/sh
PLAN_VAR=$bamboo_my_plan_var
echo "testing: $PLAN_VAR"
And I got the expected result:
testing: it works
I also wanted to create a Bamboo variable and the only thing I've found to share it between scripts is with inject-variables like following:
Add to your bamboo-spec.yaml the following after your script that will create the variable:
Build:
tasks:
- script: create-bamboo-var.sh
- inject-variables:
file: bamboo-specs/vars.yaml
scope: RESULT
# namespace: plan
- script: echo ${bamboo.inject.GIT_VERSION} # just for testing
Note: Namespace defaults to inject.
In create-bamboo-var.sh create the file bamboo-specs/vars.yaml:
#!bin/bash
versionStr=$(git describe --tags --always --dirty --abbrev=4)
echo "GIT_VERSION: ${versionStr}" > ./bamboo-specs/vars.yaml
Or for multiple lines you can use:
SW_NUMBER_DIGITS=${1} # Passed as first parameter to build script
cat <<EOT > ./bamboo-specs/vars.yaml
GIT_VERSION: ${versionStr}
SW_NUMBER_APP: ${SW_NUMBER_DIGITS}
EOT
Scope can be local or result. Local means it's only available for current job and result means it can be used in subsequent stages of this plan and releases that are created from the result.
Namespace is just used to avoid naming collisions with other variables.
With the above you can use that variable in later scripts with ${bamboo.inject.GIT_VERSION}. The last script task is just to see that it is working in other scripts. You can also see the variables in the web app as build meta data.
I'm using the above script before the build (in my case compiling C-Code) takes place so I can also create a version.h file that can be used by the source code.
This is still a bit cumbersome but I'm happy with it and I hope it will help others to configure Bamboo. Bamboo documentation could be better. (Still a lot try and error)