Is it possible to use a Linux environment variable inside a .sql file? I'm using the copy/select query to write to an output file, and I'll like to put that directory in a variable. So I want to do something like:
COPY (SELECT * FROM a)
TO $outputdir/a.csv
Outputdir would be set in my environment. Is this possible?
You can store the result of a shell command inside a psql variable like this:
\set afile `echo "$outputdir/a.csv"`
COPY (SELECT * FROM a) TO :'afile';
Another (better in my opinion) solution is to use only psql variables, see this answer of mine about psql variables, which is similar to your example. A example for your case would be:
\set outputdir '/path/to/output'
\set afile :outputdir '/a.csv'
COPY (SELECT * FROM a) TO :'afile';
Note that, in the example, you need to set the variable inside the script file, but you can skip the first line if you set it when you call psql:
psql --set=outputdir="$outputdir" <conn parameters> -f /path/to/yourscript.sql
This appears to work for your use case, provided you single quote the output file name as I mentioned. It will escape any double quotes as well contained within the SQL.
psql -c "$(eval echo '"' $(<envvars.sql | sed 's/"/\\"/g') '"')"
Of course, note that if your file contains any dollar quoted variables, the shell is going to try to interpret as a variable, and your script will break, so you will need to escape any dollar signs you need preserved literally with a backslash.
See also the second snippet in the accepted answer to this question for a possibly more robust answer.
The accepted answer is correct for PostgreSQL running on Unix. Under Windows a different incantation is required for obtaining the value of the environment variable from the CMD shell and for avoiding the carriage return returned by the echo command.
\set afile `set /p=%outputdir%/a.csv`
COPY (SELECT * FROM a) TO :'afile';
Related
The Sed command is giving me issues with incorporating the $tag variable witch is equal to "latest${GIT_COMMIT:0:7}". Here is the Sed command:
sh "sed -i 's/{BUILD_NUMBER}/$tag/' /var/lib/jenkins/workspace/${JOB_NAME}/em-api/dev-nics-emapi-svc-param.json"
I obviously want to put into my .json file the commit information but It doesnt pull the actual commit sha. When I take a look at the .json file it inserted the literal definition of the variable which is “latest${GIT_COMMIT:0:4}”. I am trying to do this on a declarative pipeline on my jenkins server running on linux.
I would like it to insert "latestxxxx". Any suggestions on how I can get around this?
GIT_COMMIT is an environment variable available to you; tag is a groovy variable, you have set to 'latest${GIT_COMMIT:0:4}'. So this gets replaced since you are using " for your sed command. But you are using ' for your sed expression, which then again will not replace environment variables. So you have basically two options:
Use " to quote the sed command, if you feel safe about the content, that gets replaced (you can use """ triple quotes for the whole command to don't have to quote the " for groovy)
Resolve the variable from the environment yourself in groovy (e.g. something like System.env['GIT_COMMIT].substring(0,4))
I have the following code in my script. There are 50 more commands like this:
"SELECT date, count(*) FROM ttc_table....date_sub(current_date, **30**)" > **/path/location/file1.txt**
"SELECT date, count(*) FROM ascc_table....date_sub(current_date, **30**)" > **/path/location/file2.txt**
"SELECT date, count(*) FROM bmtc_table....date_sub(current_date, **30**)" > **/path/location/file3.txt**
I want to create constant for value 30 and have a variable for filepath.
What is the best approach?
The comment by #Arminius answers actually half of the question, the one about variables, and it is specific to bash.
Since you have tagged the question as shell, I assume that you are searching an answer which works on POSIX shells. This means:
(1) While many of the variable substitions explained in the document by Arminius work in POSIX shell too, be aware that not all do.
(2) You also asked about defining a constant. In POSIX shell, it is AFIK not possible to define a constant. If you really need this, you have to switch to a different shell - for instance Zsh or bash. Note that in these shells, constants are actually called readonly variables.
One of my recipes in Yocto need to create a file containing a very specific line, something like:
${libdir}/something
To do this, I have the recipe task:
do_install() {
echo '${libdir}/something' >/path/to/my/file
}
Keeping in mind that I want that string exactly as shown, I can't figure out how to escape it to prevent bitbake from substituting in its own value of libdir.
I originally thought the echo command with single quotes would do the trick (as it does in the bash shell) but bitbake must be interpreting the line before passing it to the shell. I've also tried escaping it both with $$ and \$ to no avail.
I can find nothing in the bitbake doco about preventing variable expansion, just stuff to do with immediate, deferred and Python expansions.
What do I need to do to get that string into the file as is?
Bitbake seems to have particular issues in preventing expansion from taking place. Regardless of whether you use single or double quotes, it appears that the variables will be expanded before being passed to the shell.
Hence, if you want them to not be expanded, you need to effectively hide them from BitBake, and this can be done with something like:
echo -e '\x24{libdir}/something' >/path/to/my/file
This uses the hexadecimal version of $ so that BitBake does not recognise it as a variable to be expanded.
You do need to ensure you're running the correct echo command however. Under some distros (like Ubuntu), it might run the sh-internal echo which does not recognise the -e option. In order to get around that, you may have to run the variant of echo that lives on the file system (and that does recognise that option):
/bin/echo -e '\x24{libdir}/something' >/path/to/my/file
By default this task will be executed as shell function via /bin/sh, but it depends on your system what it will be as you can have a symlink named /bin/sh pointing to bash. The BitBake's manual prevents from using bashism syntax though.
You can consider just adding this task in your recipe as python function:
python do_install () {
with open('/path/to/your/file', 'a') as file:
file.write('${libdir}/something')
}
'a' stands for append.
This should eliminate the problem with variable expansion.
There is no standard way to escape these sorts of expressions that I am aware of, other than to try to break up the expression - accordingly this should work:
do_install() {
echo '$''{libdir}/something' >/path/to/my/file
}
The best solution is simply this:
bitbake_function() {
command $libdir/whatever
}
Bitbake will only expand ${libdir}; $libdir is passed through verbatim.
We don't have to worry about dollar signs that are not followed by {, and in this case, there is no need for libdir to be wrapped in braces.
The only time we run into a problem with just $foo is if we have something like ${foo}bar where the braces are required as delimiters so that bar isn't included into the variable name. In that situation, there are other solutions, such as for instance generating the shell syntax "$foo"bar. This is less cryptic than resorting to \x24.
If you need to use $ in variable assignment, remember that bitbake won't evaluate $whatever so you have to escape it for the underlying shell.
For instance I set gcc/ld Rpath option to use $ORIGIN keyword this way:
TARGET_LDFLAGS_append = " -Wl,-rpath-link=\\$$ORIGIN"
https://lists.yoctoproject.org/pipermail/yocto/2017-September/037820.html
You can define a variable to be a literal dollar sign.
DOLLAR = "$"
do_install() {
echo '${DOLLAR}{libdir}/something' >/path/to/my/file
}
no extra quoting required.
I have the following sql script:
\COPY my_table (column_1, column_2) FROM :csv_file WITH (FORMAT CSV, DELIMITER ',', ESCAPE '"');
which I am calling from:
psql -d $DB_NAME -f $SQL_FILE -v csv_file="$CSV_FILE"
But it keeps looking for a file called ":csv_file". Am I doing something wrong?
In general, variable substitution does work with meta-commands (starting with backslash), but \copy is an exception, as documented in psql's manpage:
The syntax of this command is similar to that of the SQL COPY command.
All options other than the data source/destination are as specified
for COPY. Because of this, special parsing rules apply to the \copy
command. In particular, psql's variable substitution rules and
backslash escapes do not apply.
I disagree with #BaconBits comment to the question that plpgsql's EXECUTE could be the answer. A server-side statement, dynamic or otherwise, will not access the client-side file system, contrary to \copy. You may use COPY instead, but it requires to be superuser and that file to be accessible to the postgres user on the server.
I believe that to \copy from a variable filename, the variable must be injected into the script before psql reads it. You might integrate the SQL script into a shell script and feed it to psql as a patchable here-string, or filter it through sed or perl or any similar unix-ish method.
I am creating a batch file where I am restoring a database from an IP address and then executing a couple .sql files onto the database. In a couple of the .sql files there are variables declared and set. But this process has to be done on many machines with different values for each variable in each machine.
So I'm able to restore the database through user input of the IP, but I'm not sure how to use the batch script command to change the variable values.
For example, in one of the .sql files, a variable #store was declared and set to some random number. I want to change that number through the batch file.
I am using windows and sql server express 2008 r2
You can use "scripting variables" with SQLCMD.
Here's an example from that MSDN page:
You can also use the -v option to set a scripting variable that exists in a script. In the following script (the file name is testscript.sql), ColumnName is a scripting variable.
USE AdventureWorks2012;
SELECT x.$(ColumnName)
FROM Person.Person x
WHERE c.BusinessEntityID < 5;
You can then specify the name of the column that you want returned by using the -v option:
sqlcmd -v ColumnName ="FirstName" -i c:\testscript.sql
To return a different column by using the same script, change the value of the ColumnName scripting variable.
sqlcmd -v ColumnName ="LastName" -i c:\testscript.sql
If you are working on a Unix / Linux system, you can use sed to search a string.
Example: Assuming you need to replace 127.0.0.1 to 192.168.1.1, you can use the following instruction:
$ sed 's/127.0.0.1/192.168.1.1/g' script.sql > newScript.sql
This will replace the old ip in script.sql and will save a copy of this script in newScript.sql.
On windows, I don't know how to do it, but you can always download and install Cygwin to do exactly as above.
Hope this helps you.