I want to perform a full vacuum and reindex on my database for my app hosted on Heroku.
I can't work out how to do it via the heroku command line remotely.
I can do it on my local Mac osx machine via the below commands in terminal...
psql database_name
>> vaccuum full;
>> \q
reindex database database_name
How can i perform a full vaccuum and reindex all my tables for my app on Heroku?
If possible I would like to do it without exporting the database.
Okay so it seems Heroku doesn't support this functionality unless you pay up. Looks like i'll have to pull the database, perform the actions and push it back upstream! Fun times.
You can use the psql interactive terminal with Heroku. From Heroku PostgreSQL:
If you have PostgreSQL installed on your system, you can open a direct psql console to your remote db:
$ heroku pg:psql
Connecting to HEROKU_POSTGRESQL_RED... done
psql (9.1.3, server 9.1.3)
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.
rd2lk8ev3jt5j50=>
You can also pass-in the parameters at the psql command-line, or from a batch file. The first statements gather necessary details for connecting to your database.
The final prompt asks for the constraint values, which will be used in the WHERE column IN() clause. Remember to single-quote if strings, and separate by comma:
#echo off
echo "Test for Passing Params to PGSQL"
SET server=localhost
SET /P server="Server [%server%]: "
SET database=amedatamodel
SET /P database="Database [%database%]: "
SET port=5432
SET /P port="Port [%port%]: "
SET username=postgres
SET /P username="Username [%username%]: "
"C:\Program Files\PostgreSQL\9.0\bin\psql.exe" -h %server% -U %username% -d %database% -p %port% -e -v -f cleanUp.sql
Now in your SQL code file, add the clean-up SQL, vacuum full (note the spelling). Save this as cleanUp.sql:
VACUUM FULL;
In Windows, save the whole file as a DOS BATch file (.bat), save the cleanUp.sql in the same directory, and launch the batch file. Thanks for Dave Page, of EnterpriseDB, for the original prompted script.
Also Norto, check out my other posting if you want to add parameters to your script, that can be evaluated in the SQL. Please vote it up.
Related
Files are being written to a directory using the COPY query:
Copy (SELECT * FROM animals) To '/var/lib/postgresql/data/backups/2020-01-01/animals.sql' With CSV DELIMITER ',';
However if the directory 2020-01-01 does not exist, we get the error
could not open file "/var/lib/postgresql/data/backups/2020-01-01/animals.sql" for writing: No such file or directory
PostgeSQL server is running inside a Docker container with the volume mapping /mnt/backups:/var/lib/postgresql/data/backups
The Copy query is being sent from a Node.js app outside of the Docker container.
The mapped host directory /mnt/backups was created by Docker Compose and is owned by root, so the Node.js app sending the COPY query is unable to create the missing directories due to insufficient permissions.
The backup file is meant to be transferred out of the Docker container to the Docker host.
Question: Is it possible to use an SQL query to ask PostgreSQL 11.2 to create a directory if it does not exist? If not, how will you recommend the directory creation be done?
Using Node.js 12.14.1 on Ubuntu 18.04 host. Using PostgreSQL 11.2 inside container, Docker 19.03.5
An easy way to solve it is to create the file directly into the client machine. Using STDOUT from COPY you can let the query output be redirected to the client standard output, which you can catch and save in a file. For instance, using psql in the client machine:
$ psql -U your_user -d your_db -c "COPY (SELECT * FROM animals) TO STDOUT WITH CSV DELIMITER ','" > file.csv
Creating an output directoy in case it does not exist:
$ mkdir -p /mnt/backups/2020-01/ && psql -U your_user -d your_db -c "COPY (SELECT * FROM animals) TO STDOUT WITH CSV DELIMITER ','" > /mnt/backups/2020-01/file.csv
On a side note: try to avoid exporting files into the database server. Although it is possible, I consider it a bad practice. Doing so you will either write a file into the postgres system directories or give the postgres user permission to write somewhere else, and it is something you shouldn't be comfortable with. Export data directly to the client either using COPY as I mentioned or follow the advice from #Schwern. Good luck!
Postgres has its own backup and restore utilities which are likely to be a better choice than rolling your own.
When used with one of the archive file formats and combined with pg_restore, pg_dump provides a flexible archival and transfer mechanism. pg_dump can be used to backup an entire database, then pg_restore can be used to examine the archive and/or select which parts of the database are to be restored. The most flexible output file formats are the “custom” format (-Fc) and the “directory” format (-Fd). They allow for selection and reordering of all archived items, support parallel restoration, and are compressed by default. The “directory” format is the only format that supports parallel dumps.
A simple backup rotation script might look like this:
#!/bin/sh
table='animals'
url='postgres://username#host:port/database_name'
date=`date -Idate`
file="/path/to/your/backups/$date/$table.sql"
mkdir -p `dirname $file`
pg_dump $url -w -Fc --table=$table -f $file
To avoid hard coding the database password, -w means it will not prompt for a password and instead look for a password file. Or you can use any of many Postgres authentication options.
I have been searching for a resolution for days and I am stuck. I've exhausted all resource and need to resolve this. If any one out there with ideas or know how to do this, please help.
I have an SQL Server Database, on Server1. I have an application, which is installed and runs on Server2 (ServerName2) and a procedure that needs to run within an application that is installed on this server.
What I need is to execute the batch file when ever a row get inserted into a table. The SQL Server database is on Server1. The batch file starts an application and calls a script, which executes a procedure. This script needs to run inside an application which is installed on Server2. I have a trigger after insert created on the table. Now I need to some how execute this batch file. Please note, the batch file calls a program which is installed on a different server.
I thought of 2 solutions:
To use SQL Server to execute the batch file which sits on Server2. I tried using "EXEC xp_cmdshell '\\ServerName2\C:\MSTR SCRIPT FILES\Batch_File_Execute_OTB_Script.bat".
It returned 2 records, the first one "The network name cannot be found." and NULL for second one.
Create a batch file with a connection string to ServerName2, that Start the .exe program. I am able to get the batch file to start this program and run the script if I run this on ServerName2.
In summary, I need to connect from Server1 to Server2, either through SQL Server or through batch? Is this possible? If yes, please provide me with some guidance.
Once I establish a connection, do I need to call the .exe program and run my script? What is the command to trigger this from remote connection?
Here's a little code to help you connect to the SQL server and interrogate the SQL version.
#ECHO OFF
SQLCMD /? > nul 2> nul
IF ERRORLEVEL 1 ECHO.SQLCMD not found on this machine & PAUSE & GOTO :eof
SET SQLCmdLine=SQLCMD -b -w400
REM use something like the next line for NT authentication; second line for SQL authentication
REM for NT SET SQLCmdLine=%SQLCmdLine% -E -S "YourServerName"
REM for SA SET SQLCmdLine=%SQLCmdLine% -U "YourSQLUserName" -P "YourSQLPassword" -S "YourServerName"
SET Database=YourDatabaseName
REM I will use NT Authentication for this example
SET SQLCmdLine=%SQLCmdLine% -E -S "(local)"
SET TestSQLCmdLine=%SQLCmdLine% -h-1 -d "%Database%"
REM this will get the SQL version
%TestSQLCmdLine% -Q "SELECT ##VERSION" | FINDSTR /v "rows affected"
REM this will run a SQL script
%TestSQLCmdLine% -i "YourSQLScript.SQL"
PAUSE
I deploy some .bteq and .sql scripts on a TERADATA database. For doing this, I use a client on my desktop called BTEQWin version 13.10.0.03.
I get the .bteq/.sql from a version control like pvcs/svn etc and all I do once the files are in my workspace folder (from Version control tool), to just drag and drop the files from Windows browser to BTEQWin client (which I connect to a database prior to drag/drop for running those scripts).
Now, I have to automate this whole process in UNIX.
I have written a SHELL KSH/BASH script which is getting all the .bteq/.sql from a TAG/LABEL in the version control tool to a given UNIX folder. Now, all I need to do is the pass these files one by one (i'll take care of the order) to Teradata client.
My ?
- what client do I need to tell Unix admin team to install on Unix server - so that I can run something like below:
someTeraDataCommand -u username -p password -h hostname -d database -f filenametoexectue | tee output_filename.log
Where, someTeraDataCommand is the client / executable - which will let me run Teradata scripts (like I was doing using BTEQWin on my desktop - GUI session). Other parameters can be username, password, which database to connect on what server and which file to run or make that file passed to the command using "<" operator at command line.
Any idea?
- What client ?
Assuming the complete Teradata Tools and Utilities package is installed on your UNIX server (which will have the connectivity tools to talk to Teradata), you should have access to bteq from the command line. Something like this:
bteq < script_file > output_file
Your script file should contain a .LOGON statement to establish the connection:
.LOGON yourTDPID/your_account,your_pw
You might also need to use other commands to set your default database or non-default session values.
Another option would be to combine the SQL and call to BTEQ in a Korn shell script:
#!/usr/bin/ksh
##############
SHELL_NAME=`basename $0`
PRG_NAME=`basename $(SHELL_NAME} .ksh`
LOG_FILE=${PRG_NAME}.log
OUT_FILE=${PRG_NAME}.out
#
bteq <<EOBTQ > ${LOG_FILE} 2>$1
.LOGON {TDPID}/{USERID},{PWD};
--.RUN file=${LOGON}
/* Add your SQL/BTEQ commands here */
.QUIT 0;
EOBTQ
Edit
The double hyphen indicates a single line comment. Typically in a UNIX script you do not leave your password in plain text of a KSH script. The .RUN command would reference a text file in a barely sufficient secure location containing the .LOGON {TPDID}/{USERID},{PWD}; command.
The .RUN command in BTEQ allows you to reference another text file containing a series of valid BTEQ commands that you want to run in the current BTEQ script.
Easiest way is to setup the Solaris TTU, is to request root sudo, and run an interactive installation into defaults as a root. That would cure all client issues.
I'm new to T-SQL and I'm trying to backup my databases (using SQL Server 2008).
When I try to run the script via sqlcmd -i inputfile I got this error messages:
'DATE' Scripting variable not defined.
The problem is I have a line like this:
...TO DISK = "FileName_$(ESCAPE_NONE(DATE)).BAK" ...
With a date in a filename, it will prevent it from replacing my old backups.
If I run it in management studio, it works, but if I run it in command line with the sqlcmd -i command, then it doesn't work.
EDIT:
I looked at the job history and I saw this error message:
"For SQL Server 2005 SP1 or later, you must use the appropriate ESCAPE_xxx
macro to update job steps containing tokens before the job can run"
I don't quite understand what that means. I've already used $ESCAPE_NONE(DATE), what's wrong?
Old question I know but this is one of the first results and if anyone else has the same problem the answer isn't particularly easy to find.
Including the -x switch to disable environment variables fixed the problem for me;
sqlcmd -x -i inputfile
If you're trying to backup your sql server databases and append the date to them using sqlcmd there's an easy thing you can try.
First, create the sp called sp_BackupDabases which you can find here:
http://support.microsoft.com/kb/2019698
You can invoke it from sql cmd using some command like this:
sqlcmd -U Damieh -P ilovechocolate -S (local) -Q "EXEC sp_BackupDatabases #backupLocation ='C:\MyBackups\', #BackupType='F'"
I'm sure you know this already, but just in case: -U is the user, -P is password, -S is server, and -Q is query. You can either backup all of your databases or some of them, there are parameters for that. You can find the stored proc parameters details on the same link I gave you.
The date will be automatically appended and you can play with the sp's code if you want it in a different place/way/format. I use this regularly on servers which don't have a non-express sqlserver (meaning that I can't schedule backups without using a .bat and task scheduler) with great success.
I apologize if this wasn't the answer you were looking for =). Have a nice day!
I know I'm coming along late on this thread, but you can use the following:
SQLCMD -S YourServer -E -d YourDatabase -i YourScript 2> nul
That will send the StdErrorOut to the bit bucket.
Can someone link me to a tutorial or explain if there is a way to create some sort of batch file of mysql scripts / stored procs and run them all at the same time? I can not seem to find any documentation on this online but I feel that I might be searching using the wrong terms.
You can chain mysql scripts by calling them from within a script using the source command (details of command line options)
# my_textfile.sql
# ---------------
USE my_database;
\. subscript1.sql
\. subdir/subscript2.sql
\. /full/path/to/subscript3.sql
Command Line:
mysql < my_textfile.sql
Don't forget the command line options, if you are going to script the files you might need the password/ user account.
mysql -uyouraccount -pyourpassword YourDatabase < mytextfile.sql
This isn't the most secure way to do it because it puts your username/ password on the command line but it works. If you are doing much scripting I suggest you look into .my.cnf and the various options for saving your account/ password in there (and securing that file).
You can simply create a text file with SQL statements separated with ; and then execute all statements with the MySQL command line client:
# my_textfile.sql
# ---------------
USE my_database;
SELECT * FROM table1;
UPDATE table2 SET foo='bar';
Command Line:
mysql < my_textfile.sql
For peeps running MAMP PRO on OS X Yosemite, I was able to get all my *.sql scripts executed (import) by running from terminal:
/Applications/MAMP/Library/bin/mysql -h localhost -u root -p < /Applications/MAMP/myDBRestore.sql
myDBRestore.sql contained a reference to all the MySQL DB scripts as thus:
\. /full/path/to/sql/file1.sql
\. /full/path/to/sql/file2.sql
\. /full/path/to/sql/file3.sql
...
\. /full/path/to/sql/file(n).sql
where n is the last .sql file in the directory.