How do i write a shell script that runs postgres sql scripts stored in one folder sequentially.Example
1)dump.sql,
2)store.sql,
3)merge.sql,
4)import.sql
are to run sequentially in the same order and all the scripts are in the same folder. How do i write a shell script that performs the tasks sequentially.
Inside psql you can use "\i " to load a script. But also "\ir ".
So in your case create a load-all.sql file in the same directory as your other files..
contents load-all.sql:
\ir dump.sql
\ir store.sql
\ir merge.sql
\ir import.sql
inside psql:
psql# \i /path/to/file/load-all.sql
You can create a simple script e.g run.sh and the define variables corresponding to your environment. Here is a simple script that is running 2 sql files which are present on Desktop
PG_HOME=/usr/pgsql-12
PGUSER=postgres
DATABASE=postgres
PORT=5432
FILES_HOME=/home/edb/Desktop
$PG_HOME/bin/psql -U $PGUSER -d $DATABASE -p $PORT -f $FILES_HOME/dump.sql
$PG_HOME/bin/psql -U $PGUSER -d $DATABASE -p $PORT -f $FILES_HOME/store.sql
There are database schema migration tools available to help manage this process.
Examples include:
Flyway
Liquibase
I use Flyway. It does exactly what you want, connects to the database and runs a bunch of SQL scripts found in a folder. The order of execution is determined by the file names named using a certain convention.
On the first run, Flyway adds its own table to your database to store the history of what SQL scripts have been run. On subsequent runs, Flyway knows what SQL scripts have already been applied (and should be skipped) and which scripts are fresh (and should be applied).
Try this:
param([string]$directory = "")
$files = Get-ChildItem $directory
for ($i=0; $i -lt $files.Count; $i++) {
$scriptName = $files[$i].Name
psql -U <username> -d <database> -h <host> -p <port> -f $directory\$scriptName
}
directory is an argument you should pass to the script.
So executing it with:
./sqlScript.ps1 C:\Users\<username>\Desktop\<pathToDict>
If you are troubling with executing psql on windows environment you can get help with
this:
How to start psql.exe
Related
I have a folder in a directory in my PC which contains multiple SQL files. Each of the file is a Postgres function. I want to execute every SQL file situated in the folder at a time in PostgreSQL server using PgAdmin or in other way. How can I accomplish this?
I apologize if I'm oversimplifying your question, but if the main issue is how to execute all SQL files without having to call them one by one, you just need to put them in a loop, e.g. in bash calling psql
#!/bin/bash
for f in *.sql
do
psql -h dbhost -d db -U dbuser -f $f
done
Or cat them and pipe the result to psql stdin:
$ cat /path/to/files/*.sql | psql -h dbhost -d db -U dbuser
And if you need them to run in a single transaction, consider merging the SQL files, e.g. using cat - this assumes all statements in your sql file are properly terminated:
$ cat /path/to/files/*.sql > merged.sql
I am trying to execute the .sql file from batch script, but not able to execute.
Script has multiple scripts such as Create Table , Stored Procedure separated by GO statement.
I tried with the below script, however script was not executed.
DB Servername : localhost\SQL2017
Authentication : Windows
sqlcmd -E -S localhost\SQL2017\MyDatabase -i C:\SQLScripts\TestScript.sql
Try this instead:
sqlcmd -E -S localhost\SQL2017 -d MyDatabase -i C:\SQLScripts\TestScript.sql
You need to use the -d parameter to specify the database name. The -S parameter should contain only the server name and the instance.
I have a file that contains several SQL queries.
Can I somehow run them via isql (I'm doing the calls from Bash script, so no access to Perl DBI or JDBC)
I tried piping them into isql command via echo /my/file | isql -my-other-parameters but that didn't work.
Yes.
If you're running ISQL in interactive mode, you can load an entire contents of the file using :r my-filename command from > prompt.
From Bash script, it's also possible to do - but you need to carefully make sure that
The SQL file you are piping in has a go statement at the end. That is a VERY common cause of issues like the one you mentioned.
That statement has a newline after it.
From a script, you can do it 2 ways: Pass on STDIN via a pipe/redirect; OR, pass in the file name via isql's -i parameter
In my case it was isq -n for a piped query to work.
isql -U $DB_USR -P $DB_PWD -S $DB_PATH -D $DB_NAME -w 500 < $FILE
I am learning the shell language. I have creating a shell script whose function is to login into the DB and run a .sql file. Following are the contents of the script -
#!/bin/bash
set -x
echo "Login to postgres user for autoqa_rpt_production"
$DB_PATH -U $POSTGRESS_USER $Auto_rpt_production$TARGET_DB -p $TARGET_PORT
echo "Running SQL Dump - auto_qa_db_sync"
\\i auto_qa_db_sync.sql
After running the above script, I get the following error
./autoqa_script.sh: 39: ./autoqa_script.sh: /i: not found
Following one article, I tried reversing the slash but it didn't worked.
I don't understand why this is happening. Because when I try manually running the sql file, it works properly. Can anyone help?
#!/bin/bash
set -x
echo "Login to postgres user for autoqa_rpt_production and run script"
$DB_PATH -U $POSTGRESS_USER $Auto_rpt_production$TARGET_DB -p $TARGET_PORT -f auto_qa_db_sync.sql
The lines you put in a shell script are (moreless, let's say so for now) equivalent to what you would put right to the Bash prompt (the one ending with '$' or '#' if you're a root). When you execute a script (a list of commands), one command will be run after the previous terminates.
What you wanted to do is to run the client and issue a "\i ./autoqa_script.sh" comand in it.
What you did was to run the client, and after the client terminated, issue that command in Bash.
You should read about Bash pipelines - these are the way to run programs and input text inside them. Following your original idea to solving the problem, you'd write something like:
echo '\i auto_qa_db_sync.sql' | $DB_PATH -U $POSTGRESS_USER $Auto_rpt_production$TARGET_DB -p $TARGET_PORT
Hope that helps to understand.
I have some SQL scripts that I'm trying to automate. In the past I have used SQL*Plus, and called the sqlplus binary manually, from a bash script.
However, I'm trying to figure out if there's a way to connect to the DB, and call the script from inside of the bash script... so that I can insert date and make the queries run relative to a certain number of days in the past.
I'm slightly confused. You should be able to call sqlplus from within the bash script. This may be what you were doing with your first statement
Try Executing the following within your bash script:
#!/bin/bash
echo Start Executing SQL commands
sqlplus <user>/<password> #file-with-sql-1.sql
sqlplus <user>/<password> #file-with-sql-2.sql
If you want to be able to pass data into your scripts you can do it via SQLPlus by passing arguments into the script:
Contents of file-with-sql-1.sql
select * from users where username='&1';
Then change the bash script to call sqlplus passing in the value
#!/bin/bash
MY_USER=bob
sqlplus <user>/<password> #file-with-sql-1.sql $MY_USER
You can also use a "here document" to do the same thing:
VARIABLE=SOMEVALUE
sqlplus connectioninfo << HERE
start file1.sql
start file2.sql $VARIABLE
quit
HERE
Here is a simple way of running MySQL queries in the bash shell
mysql -u [database_username] -p [database_password] -D [database_name] -e "SELECT * FROM [table_name]"
Maybe you can pipe SQL query to sqlplus. It works for mysql:
echo "SELECT * FROM table" | mysql --user=username database
I've used the jdbcsql project on Sourceforge.
On *nix systems, this will create a csv stream of results to standard out:
java -Djava.security.egd=file///dev/urandom -jar jdbcsql.jar -d oracledb_SID -h $host -p 1521 -U some_username -m oracle -P "$PW" -f excel -s "," "$1"
Note that adding the -Djava.security.egd=file///dev/urandom increases performance greatly
Windows commands are similar: see http://jdbcsql.sourceforge.net/
If you do not want to install sqlplus on your server/machine then the following command-line tool can be your friend. It is a simple Java application, only Java 8 that you need in order to you can execute this tool.
The tool can be used to run any SQL from the Linux bash or Windows command line.
Example:
java -jar sql-runner-0.2.0-with-dependencies.jar \
-j jdbc:oracle:thin:#//oracle-db:1521/ORCLPDB1.localdomain \
-U "SYS as SYSDBA" \
-P Oradoc_db1 \
"select 1 from dual"
Documentation is here.
You can download the binary file from here.
As Bash doesn't have built in sql database connectivity... you will need to use some sort of third party tool.