I have the following sql script:
\COPY my_table (column_1, column_2) FROM :csv_file WITH (FORMAT CSV, DELIMITER ',', ESCAPE '"');
which I am calling from:
psql -d $DB_NAME -f $SQL_FILE -v csv_file="$CSV_FILE"
But it keeps looking for a file called ":csv_file". Am I doing something wrong?
In general, variable substitution does work with meta-commands (starting with backslash), but \copy is an exception, as documented in psql's manpage:
The syntax of this command is similar to that of the SQL COPY command.
All options other than the data source/destination are as specified
for COPY. Because of this, special parsing rules apply to the \copy
command. In particular, psql's variable substitution rules and
backslash escapes do not apply.
I disagree with #BaconBits comment to the question that plpgsql's EXECUTE could be the answer. A server-side statement, dynamic or otherwise, will not access the client-side file system, contrary to \copy. You may use COPY instead, but it requires to be superuser and that file to be accessible to the postgres user on the server.
I believe that to \copy from a variable filename, the variable must be injected into the script before psql reads it. You might integrate the SQL script into a shell script and feed it to psql as a patchable here-string, or filter it through sed or perl or any similar unix-ish method.
Related
The Sed command is giving me issues with incorporating the $tag variable witch is equal to "latest${GIT_COMMIT:0:7}". Here is the Sed command:
sh "sed -i 's/{BUILD_NUMBER}/$tag/' /var/lib/jenkins/workspace/${JOB_NAME}/em-api/dev-nics-emapi-svc-param.json"
I obviously want to put into my .json file the commit information but It doesnt pull the actual commit sha. When I take a look at the .json file it inserted the literal definition of the variable which is “latest${GIT_COMMIT:0:4}”. I am trying to do this on a declarative pipeline on my jenkins server running on linux.
I would like it to insert "latestxxxx". Any suggestions on how I can get around this?
GIT_COMMIT is an environment variable available to you; tag is a groovy variable, you have set to 'latest${GIT_COMMIT:0:4}'. So this gets replaced since you are using " for your sed command. But you are using ' for your sed expression, which then again will not replace environment variables. So you have basically two options:
Use " to quote the sed command, if you feel safe about the content, that gets replaced (you can use """ triple quotes for the whole command to don't have to quote the " for groovy)
Resolve the variable from the environment yourself in groovy (e.g. something like System.env['GIT_COMMIT].substring(0,4))
I need to insert both English and Arabic text into a PostgreSQL database.
I'm running the following command via a .bat script:
psql.exe --echo-all --username=postgres --dbname=dbname -f populate.sql
populate.sql has statements like this:
insert into table1 (column1, column2) VALUES (2, 'المستخدم ');
If I do this via pgadmin, it works. The thing is I need to do this via some .sql population scripts that are ran once the application is started.
In that case I get gibberish, like this:
العرض
I created the scripts in Notepad++ using the Encode in UTF-8 without BOM option, since the normal encoding in UTF-8 adds an extra character to the start of the file and some of the inserts are not made.
I'm assuming this is an encode problem, but I have yet to figure out exactly what is wrong.
The databse is in UTF-8.
Thanks in advance!
The Windows console doesn't speak Unicode of any form by default, it speaks a "native" codepage. Which codepage depends on your Windows install's language settings.
If you chcp 65001 then it'll switch into utf-8.
Overall, though, text encoding handling in batch/cmd files and the Windows command line is absolutely awful. I generally recommend that you instead put anything that isn't basically 7-bit ASCII into a separate .sql file and execute it via psql -f with a suitable client_encoding; the PGCLIENTENCODING environment variable is useful for this.
So try:
SET PGCLIENTENCODING=utf-8
psql.exe --username=postgres --dbname=dbname -f populate.sql
Is it possible to use a Linux environment variable inside a .sql file? I'm using the copy/select query to write to an output file, and I'll like to put that directory in a variable. So I want to do something like:
COPY (SELECT * FROM a)
TO $outputdir/a.csv
Outputdir would be set in my environment. Is this possible?
You can store the result of a shell command inside a psql variable like this:
\set afile `echo "$outputdir/a.csv"`
COPY (SELECT * FROM a) TO :'afile';
Another (better in my opinion) solution is to use only psql variables, see this answer of mine about psql variables, which is similar to your example. A example for your case would be:
\set outputdir '/path/to/output'
\set afile :outputdir '/a.csv'
COPY (SELECT * FROM a) TO :'afile';
Note that, in the example, you need to set the variable inside the script file, but you can skip the first line if you set it when you call psql:
psql --set=outputdir="$outputdir" <conn parameters> -f /path/to/yourscript.sql
This appears to work for your use case, provided you single quote the output file name as I mentioned. It will escape any double quotes as well contained within the SQL.
psql -c "$(eval echo '"' $(<envvars.sql | sed 's/"/\\"/g') '"')"
Of course, note that if your file contains any dollar quoted variables, the shell is going to try to interpret as a variable, and your script will break, so you will need to escape any dollar signs you need preserved literally with a backslash.
See also the second snippet in the accepted answer to this question for a possibly more robust answer.
The accepted answer is correct for PostgreSQL running on Unix. Under Windows a different incantation is required for obtaining the value of the environment variable from the CMD shell and for avoiding the carriage return returned by the echo command.
\set afile `set /p=%outputdir%/a.csv`
COPY (SELECT * FROM a) TO :'afile';
I am creating a batch file where I am restoring a database from an IP address and then executing a couple .sql files onto the database. In a couple of the .sql files there are variables declared and set. But this process has to be done on many machines with different values for each variable in each machine.
So I'm able to restore the database through user input of the IP, but I'm not sure how to use the batch script command to change the variable values.
For example, in one of the .sql files, a variable #store was declared and set to some random number. I want to change that number through the batch file.
I am using windows and sql server express 2008 r2
You can use "scripting variables" with SQLCMD.
Here's an example from that MSDN page:
You can also use the -v option to set a scripting variable that exists in a script. In the following script (the file name is testscript.sql), ColumnName is a scripting variable.
USE AdventureWorks2012;
SELECT x.$(ColumnName)
FROM Person.Person x
WHERE c.BusinessEntityID < 5;
You can then specify the name of the column that you want returned by using the -v option:
sqlcmd -v ColumnName ="FirstName" -i c:\testscript.sql
To return a different column by using the same script, change the value of the ColumnName scripting variable.
sqlcmd -v ColumnName ="LastName" -i c:\testscript.sql
If you are working on a Unix / Linux system, you can use sed to search a string.
Example: Assuming you need to replace 127.0.0.1 to 192.168.1.1, you can use the following instruction:
$ sed 's/127.0.0.1/192.168.1.1/g' script.sql > newScript.sql
This will replace the old ip in script.sql and will save a copy of this script in newScript.sql.
On windows, I don't know how to do it, but you can always download and install Cygwin to do exactly as above.
Hope this helps you.
I have a MS SQL 2005 stored proc which takes an out parameter. How can I call this from a dos batch file and get the value of the out param? I know I have to use sqlcmd, but cant find anyting in there by which I can pass an out param and access its value in dos batch file.
Thanks
vikram
I do this kind of thing all the time (like so) with standard T-SQL, but you might be able to do something like this with a stored procedure if you edit the stored procedure to show a one line result set.
sqlcmd -b -S %COMPUTERNAME% -E -d %DBNAME% -Q "exec getXMLLocation;" -h-1
-o SearchResult.txt
set /p URI=<SearchResult.txt
#echo The XML file URI is: %URI%
In DOS, you will get any information returned in the standard out, but you cannot easily manipulate this. Must this be DOS? Is PowerShell an option, as you have more capabilities with PowerShell (heck even WSH is a better option for DOS if you need to store this value and not just show it in the command prompt).
Adding this based on comment this must be DOS. Here are my thoughts:
First, I would use the out macro statement to direct to stdout:
: out stdout
Once you have output in stdout, you can use DOS commands to direct it to variables you have set up in DOS. stdout is handle 1 in DOS.
The one issue I can think that might make this fail is if other items are cluttering up stdout. I would not want to parse through a lot of junk.