How do I escape a "$" in bitbake/yocto? - variables

One of my recipes in Yocto need to create a file containing a very specific line, something like:
${libdir}/something
To do this, I have the recipe task:
do_install() {
echo '${libdir}/something' >/path/to/my/file
}
Keeping in mind that I want that string exactly as shown, I can't figure out how to escape it to prevent bitbake from substituting in its own value of libdir.
I originally thought the echo command with single quotes would do the trick (as it does in the bash shell) but bitbake must be interpreting the line before passing it to the shell. I've also tried escaping it both with $$ and \$ to no avail.
I can find nothing in the bitbake doco about preventing variable expansion, just stuff to do with immediate, deferred and Python expansions.
What do I need to do to get that string into the file as is?

Bitbake seems to have particular issues in preventing expansion from taking place. Regardless of whether you use single or double quotes, it appears that the variables will be expanded before being passed to the shell.
Hence, if you want them to not be expanded, you need to effectively hide them from BitBake, and this can be done with something like:
echo -e '\x24{libdir}/something' >/path/to/my/file
This uses the hexadecimal version of $ so that BitBake does not recognise it as a variable to be expanded.
You do need to ensure you're running the correct echo command however. Under some distros (like Ubuntu), it might run the sh-internal echo which does not recognise the -e option. In order to get around that, you may have to run the variant of echo that lives on the file system (and that does recognise that option):
/bin/echo -e '\x24{libdir}/something' >/path/to/my/file

By default this task will be executed as shell function via /bin/sh, but it depends on your system what it will be as you can have a symlink named /bin/sh pointing to bash. The BitBake's manual prevents from using bashism syntax though.
You can consider just adding this task in your recipe as python function:
python do_install () {
with open('/path/to/your/file', 'a') as file:
file.write('${libdir}/something')
}
'a' stands for append.
This should eliminate the problem with variable expansion.

There is no standard way to escape these sorts of expressions that I am aware of, other than to try to break up the expression - accordingly this should work:
do_install() {
echo '$''{libdir}/something' >/path/to/my/file
}

The best solution is simply this:
bitbake_function() {
command $libdir/whatever
}
Bitbake will only expand ${libdir}; $libdir is passed through verbatim.
We don't have to worry about dollar signs that are not followed by {, and in this case, there is no need for libdir to be wrapped in braces.
The only time we run into a problem with just $foo is if we have something like ${foo}bar where the braces are required as delimiters so that bar isn't included into the variable name. In that situation, there are other solutions, such as for instance generating the shell syntax "$foo"bar. This is less cryptic than resorting to \x24.

If you need to use $ in variable assignment, remember that bitbake won't evaluate $whatever so you have to escape it for the underlying shell.
For instance I set gcc/ld Rpath option to use $ORIGIN keyword this way:
TARGET_LDFLAGS_append = " -Wl,-rpath-link=\\$$ORIGIN"
https://lists.yoctoproject.org/pipermail/yocto/2017-September/037820.html

You can define a variable to be a literal dollar sign.
DOLLAR = "$"
do_install() {
echo '${DOLLAR}{libdir}/something' >/path/to/my/file
}
no extra quoting required.

Related

Jenkins variable not working with sed command in pipeline

The Sed command is giving me issues with incorporating the $tag variable witch is equal to "latest${GIT_COMMIT:0:7}". Here is the Sed command:
sh "sed -i 's/{BUILD_NUMBER}/$tag/' /var/lib/jenkins/workspace/${JOB_NAME}/em-api/dev-nics-emapi-svc-param.json"
I obviously want to put into my .json file the commit information but It doesnt pull the actual commit sha. When I take a look at the .json file it inserted the literal definition of the variable which is “latest${GIT_COMMIT:0:4}”. I am trying to do this on a declarative pipeline on my jenkins server running on linux.
I would like it to insert "latestxxxx". Any suggestions on how I can get around this?
GIT_COMMIT is an environment variable available to you; tag is a groovy variable, you have set to 'latest${GIT_COMMIT:0:4}'. So this gets replaced since you are using " for your sed command. But you are using ' for your sed expression, which then again will not replace environment variables. So you have basically two options:
Use " to quote the sed command, if you feel safe about the content, that gets replaced (you can use """ triple quotes for the whole command to don't have to quote the " for groovy)
Resolve the variable from the environment yourself in groovy (e.g. something like System.env['GIT_COMMIT].substring(0,4))

How do I include a literal double quote in a custom CMake command?

I'm trying to create a custom command that runs with some environment variables, such as LDFLAGS, whose value needs to be quoted if it contains spaces:
LDFLAGS="-Lmydir -Lmyotherdir"
I cannot find a way to include this argument in a CMake custom command, due to CMake's escaping rules. Here's what I've tried so far:
COMMAND LDFLAGS="-Ldir -Ldir2" echo blah VERBATIM)
yields "LDFLAGS=\"-Ldir -Ldir2\"" echo blah
COMMAND LDFLAGS=\"-Ldir -Ldir2\" echo blah VERBATIM)
yields LDFLAGS=\"-Ldir -Ldir2\" echo blah
It seems I either get the whole string quoted, or the escaped quotes don't resolve when used as part of the command.
I would appreciate either a way to include the literal double-quote or as an alternative a better way to set environment variables for a command. Please note that I'm still on CMake 2.8, so I don't have the new "env" command available in 3.2.
Note that this is not a duplicate of When to quote variables? as none of those quoting methods work for this particular case.
The obvious choice - often recommended when hitting the boundaries of COMMAND especially with older versions of CMake - is to use an external script.
I just wanted to add some simple COMMAND only variations that do work and won't need a shell, but are - I have to admit - still partly platform dependent.
One example would be to put only the quoted part into a variable:
set(vars_as_string "-Ldir -Ldir2")
add_custom_target(
QuotedEnvVar
COMMAND env LD_FLAGS=${vars_as_string} | grep LD_FLAGS
)
Which actually does escape the space and not the quotes.
Another example would be to add it with escaped quotes as a "launcher" rule:
add_custom_target(
LauncherEnvVar
COMMAND env | grep LD_FLAGS
)
set_target_properties(
LauncherEnvVar
PROPERTIES RULE_LAUNCH_CUSTOM "env LD_FLAGS=\"-Ldir -Ldir2\""
)
Edit: Added examples for multiple quoted arguments without the need of escaping quotes
Another example would be to "hide some of the complexity" in a function and - if you want to add this to all your custom command calls - use the global/directory RULE_LAUNCH_CUSTOM property:
function(set_env)
get_property(_env GLOBAL PROPERTY RULE_LAUNCH_CUSTOM)
if (NOT _env)
set_property(GLOBAL PROPERTY RULE_LAUNCH_CUSTOM "env")
endif()
foreach(_arg IN LISTS ARGN)
set_property(GLOBAL APPEND_STRING PROPERTY RULE_LAUNCH_CUSTOM " ${_arg}")
endforeach()
endfunction(set_env)
set_env(LDFLAGS="-Ldir1 -Ldir2" CFLAGS="-Idira -Idirb")
add_custom_target(
MultipleEnvVar
COMMAND env | grep -E 'LDFLAGS|CFLAGS'
)
Alternative (for CMake >= 3.0)
I think what we actually are looking for here (besides the cmake -E env ...) is named Bracket Argument and does allow any character without the need of adding backslashes:
set_property(
GLOBAL PROPERTY
RULE_LAUNCH_CUSTOM [=[env LDFLAGS="-Ldir1 -Ldir2" CFLAGS="-Idira -Idirb"]=]
)
add_custom_target(
MultipleEnvVarNew
COMMAND env | grep -E 'LDFLAGS|CFLAGS'
)
References
0005145: Set environment variables for ADD_CUSTOM_COMMAND/ADD_CUSTOM_TARGET
How to modify environment variables passed to custom CMake target?
[CMake] How to set environment variable for custom command
cmake: when to quote variables?
You need three backslashes. I needed this recently to get a preprocessor define from PkgConfig and apply it to my C++ flags:
pkg_get_variable(SHADERDIR movit shaderdir)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DSHADERDIR=\\\"${SHADERDIR}\\\"")
Florian's answer is wrong on several counts:
Putting the quoted part in a variable makes no difference.
You should definitely use VERBATIM. It fixes platform-specific quoting bugs.
You definitely shouldn't use RULE_LAUNCH_CUSTOM for this. It isn't intended for this and only works with some generators.
You shouldn't use env as the command. It isn't available on Windows.
It turns out the real reason OPs code doesn't work is that CMake always fully quotes the first word after COMMAND because it's supposed to be the name of an executable. You simply shouldn't put environment variables first.
For example:
add_custom_command(
OUTPUT q1.txt
COMMAND ENV_VAR="a b" echo "hello" > q1.txt
VERBATIM
)
add_custom_target(q1 ALL DEPENDS q1.txt)
$ VERBOSE=1 make
...
"ENV_VAR=\"a b\"" echo hello > q1.txt
/bin/sh: ENV_VAR="a b": command not found
So how do you pass an environment variable with spaces? Simple.
add_custom_command(
OUTPUT q1.txt
COMMAND ${CMAKE_COMMAND} -E env ENV_VAR="a b" echo "hello" > q1.txt
VERBATIM
)
Ok, I removed my original answer as the one proposed by #Florian is better. There is one additional tweak needed for multiple quoted args. Consider a list of environment variables as such:
set(my_env_vars LDFLAGS="-Ldir1 -Ldir2" CFLAGS="-Idira -Idirb")
In order to produce the desired expansion, convert to string and then replace ; with a space.
set(my_env_string "${my_env_vars}") #produces LDFLAGS="...";CFLAGS="..."
string(REPLACE ";" " " my_env_string "${my_env_string}")
Then you can proceed with #Florian's brilliant answer and add the custom launch rule. If you need semicolons in your string then you'll need to convert them to something else first.
Note that in this case I didn't need to launch with env:
set_target_properties(mytarget PROPERTIES RULE_LAUNCH_CUSTOM "${my_env_string}")
This of course depends on your shell.
On second thought, my original answer is below as I also have a case where I don't have access to the target name.
set(my_env LDFLAGS=\"-Ldir -Ldir2" CFLAGS=\"-Idira -Idirb\")
add_custom_command(COMMAND sh -c "${my_env} grep LDFLAGS" VERBATIM)
This technique still requires that the semicolons from the list->string conversion be replaced.
Some folks suggest to use ${CMAKE_COMMAND} and pass your executable as an argument, e.g:
COMMAND ${CMAKE_COMMAND} -E env "$(WindowsSdkDir)/bin/x64/makecert.exe" ...
That worked for me.

How to preserve single quotes in a CMake cached variable?

I have a variable
SET(CODE_COVERAGE_EXCLUSION_LIST
""
CACHE STRING "List of resources to exclude from code coverage analysis")
It must contain a list of expressions such as : 'tests/*' '/usr/*'
When trying to set the default value to the above expressions, the single quotes are removed.
How to preserve them ?
Moreover, when I try to pass the exclusion list like this
cmake -DCODE_COVERAGE_EXCLUSION_LIST="'tests/*' '/usr/*'" ..
The initial and final single quotes are lost. How to preserve them as well ?
Finally, the same question applies when using cmake-gui.
EDIT : I tried to use backslash to escape the quotes :
SET(CODE_COVERAGE_EXCLUSION_LIST
" \'tests/*\' \'/usr/*\'"
CACHE STRING "List of resources to exclude from code coverage analysis : ")
It gave me the following error :
Syntax error in cmake code at
xxx.cmake:106
when parsing string
\'tests/*\' \'/usr/*\'
Invalid escape sequence \'
EDIT2 : code of the add_custom_target (and not add_custom_command, my bad)
ADD_CUSTOM_TARGET(${_targetname}
# Cleanup lcov
${LCOV_PATH} --directory . --zerocounters
# Run tests
COMMAND ${_testrunner} ${ARGV3}
# Capturing lcov counters and generating report
COMMAND ${LCOV_PATH} --directory . --capture --output-file ${_outputname}.info
COMMAND ${LCOV_PATH} --remove ${_outputname}.info 'tests/*' '/usr/*' ${CODE_COVERAGE_EXCLUSION_LIST} --output-file ${_outputname}.info.cleaned
COMMAND ${GENHTML_PATH} -o ${_outputname} ${_outputname}.info.cleaned
COMMAND ${CMAKE_COMMAND} -E remove ${_outputname}.info ${_outputname}.info.cleaned
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "Resetting code coverage counters to zero.\nProcessing code coverage counters and generating report."
)
Turning my comments into an answer
First - taking the question why you need those quotes aside - I could reproduce your problem and found several possible solutions:
Adding spaces at the begin and end of your cached variable
cmake -DCODE_COVERAGE_EXCLUSION_LIST=" 'tests/*' '/usr/*' " ..
Using "escaped" double quotes instead of single quotes
cmake -DCODE_COVERAGE_EXCLUSION_LIST:STRING="\"tests/*\" \"/usr/\"" ..
Using your set(... CACHE ...) example by setting policy CMP0053 switch introduced with CMake 3.1:
cmake_policy(SET CMP0053 NEW)
set(CODE_COVERAGE_EXCLUSION_LIST
"\'tests/*\' \'/usr/*\'"
CACHE STRING "List of resources to exclude from code coverage analysis : ")
But when setting this in the code I could also just do
set(CODE_COVERAGE_EXCLUSION_LIST
"'tests/*' '/usr/*'"
CACHE STRING "List of resources to exclude from code coverage analysis : ")
The quoting issue seems only to be a problem when called from command line
Then - if I do assume you may not need the quotes - you could pass the paths as a list:
A semicolon separated CMake list is expanded to parameters again (with spaces as delimiter) when used in a COMMAND
cmake -DCODE_COVERAGE_EXCLUSION_LIST:STRING="tests/*;/usr/*" ..
with
add_custom_target(
...
COMMAND ${LCOV_PATH} --remove ${_outputname}.info ${CODE_COVERAGE_EXCLUSION_LIST} --output-file ${_outputname}.info.cleaned
)
would give something like
.../lcov --remove output.info tests/* /usr/* --output-file output.info.cleaned
I also tried to add the VERBATIM option, because "all arguments to the commands will be escaped properly for the build tool so that the invoked command receives each argument unchanged". But in this case, it didn't change anything.
References
add_custom_target()
CMake Language: Escape Sequences
0015200: Odd quoting issue when mixing single, double and escaped quotes to COMMAND

An alias that just prints the named directory?

I'm kind of just realizing how powerful the terminal can be. My question is essentially if I can create an alias that just prints the name of a directory. For example, I could easily make an alias such as "alias sitename="cd ~/sites/path/to/my/site/". But what I want is an alias that only prints the directory name so that I can use it for several things. So that, for example, if I wanted I could just say cd "alias", or mv from-dir "alias".
Is there a way to do this? I've tried and it seems to recognize the alias if I just type it in: it will report "alias" is a directory. But if I try to couple it with another command, it fails.
You don't want to use alias, what you are after is an environment variable
$ export SITENAME="~/sites/path/to/my/site/"
$ cd $SITENAME
Bash is quite picky over syntax - note the lack of spaces in the export and the $ when you use it.
Use a variable, simply
d=/path/to/some/directory
echo $d
cd $d
mv somedir $d/
You don't need to use an alias here, a variable is sufficient.
It sounds like you want to set a variable, not an alias. Such as,
sitename=/home/jimbo/. Then, cd $sitename would put you at /home/jimbo/.
If you want this variable to have permanence (i.e. you don't have to set it every time you open a new session), then you can make it an environmental variable using the export command or add it to your .bashrc file (typically located at $HOME/.bashrc) using the line: sitename=/home/jimbo/.
FYI, $HOME is another environmental variable that's equivalent to ~/.
Using a variable is the simplest solution. You could get fancy and use an array:
mydir() { echo "/my/directory"; }
To display the value
mydir
To use the value, you need some extra puncuation
cd $(mydir)
cd `mydir`

Zsh trouble when using echo with color/formatting characters

I'm just switch to zsh and now adapting the alias in which was printing some text (in color) along with a command.
I have been trying to use the $fg array var, but there is a side effect, all the command is printed before being executed.
The same occur if i'm just testing a echo with a color code in the terminal:
echo $fg_bold[blue] "test"
]2;echo "test" test #the test is in the right color
Why the command print itself before to do what it's supposed to do ? (I precise this doesn't happen when just printing whithout any wariable command)
Have I to set a specific option to zsh, use echo with a special parameter to get ride of that?
Execute the command first (keep its output somewhere), and then issue echo. The easiest way I can think of doing that would be:
echo $fg[red] `ls`
Edit: Ok, so your trouble is some trash before the actual output of echo. You have some funny configuration that is causing you trouble.
What to do (other than inspecting your configuration):
start a shell with zsh -f (it will skip any configuration), and then re-try the echo command: autoload colors; colors; echo $fg_bold[red] foo (this should show you that the problem is in your configuration).
Most likely your configuration defines a precmd function that gets executed before every command (which is failing in some way). Try which precmd. If that is not defined, try echo $precmd_functions (precmd_functions is an array of functions that get executed before every command). Knowing which is the code being executed would help you search for it in your configuration (which I assume you just took from someone else).
If I had to guess, I'd say you are using oh-my-zsh without knowing exactly what you turned on (which is an endless source of troubles like this).
I don't replicate your issue, which I think indicates that it's either an option (that I've set), or it's a zsh version issue:
$ echo $fg_bold[red] test
test
Because I can't replicate it, I'm sure there's an option to stop it happening for you. I do not know what that option is (I'm using heavily modified oh-my-zsh, and still haven't finished learning what all the zsh options do or are).
My suggestions:
You could try using print:
$ print $fg_bold[red] test
test
The print builtin has many more options than echo (see man zshbuiltins).
You should also:
Check what version zsh you're using.
Check what options (setopt) are enabled.
Check your ~/.zshrc (and other loaded files) to see what, if any, options and functions are being run.
This question may suggest checking what TERM you're using, but reading your question it sounds like you're only seeing this behaviour (echoing of the command after entry) when you're using aliases...?