When run with this command, pgbadger finds no queries, even though there are slow queries logged in the database log.
pgbadger --dbname foo /path/to/logfile --format stderr --prefix '%m [%p] '
Example log entry
2020-04-08 07:58:42.014 BST [1505890] LOG: duration: 119.079 ms statement: SELECT "bar".id FROM "bar" WHERE (("bar"."baz" = 0.0) AND ("bar"."quux" = 0.0)) ORDER BY "bar"."id"
The problem was passing the --dbname flag and its argument. The log prefix '%m [%p] ' does not include the database name, so pgbadger, presumably, is unable to find any statements logged against the provided database name and reports accordingly.
The solution is to either not pass --dbname or modify the log prefix in postgresql.conf to include the database name (for example '%m [%p] %d '), reload the server config and wait for new entries in the log.
I found this on an Openstack Fedora vm, where '%m [%p] ' was the default log prefix.
Related
I have a Bash Shell Script that runs a DB2 sql file. The job of this sql file is to completely replace the contents of a database table with whatever are the contents of this sql file.
However, I also need that database table to have its contents preserved if errors are discovered in the ingested file. For example, supposing my table currently looks like this:
MY_TABLE
C1
C2
row0
15
27
row1
19
20
And supposing I have an input file that looks like this:
15,28
34,90
"a string that's obviously not supposed to be here"
54,23
If I run the script with this input file, the table should stay exactly the same as it was before, not using the contents of the file at all.
However, when I run my script, this isn't the behavior I observe: instead, the contents of MY_TABLE do get replaced with all of the valid rows of the input file so the new contents of the table become:
MY_TABLE
C1
C2
row0
15
28
row1
34
90
row2
54
23
In my script logic, I explicitly disable autocommit for the part of the script that ingests the file, and I only call commit after I've checked that the sql execution returned no errors; if it did cause errors, I call rollback instead. Nonetheless, the contents of the table get replaced when errors occur, as though the rollback command wasn't called at all, and a commit was called instead.
Where is the problem in my script?
script.ksh
SQL_FILE=/app/scripts/script.db2
LOG=/app/logs/script.log
# ...
# Boilerplate to setup the connection to the database server
# ...
# +c: autocommit off
# -v: echo commands
# -s: Stop if errors occur
# -p: Show prompt for interactivity (for debugging)
# -td#: use '#' as the statement delimiter in the file
db2 +c -s -v -td# -p < $SQL_FILE >> $LOG
if [ $? -gt 2 ];
then echo "An Error occurred; rolling back the data" >> $LOG
db2 "ROLLBACK" >> $LOG
exit 1
fi
# No errors, commit the changes
db2 "COMMIT" >> $LOG
script.db2
ingest from file '/app/temp/values.csv'
format delimited by ','
(
$C1 INTEGER EXTERNAL,
$C2 INTEGER EXTERNAL
)
restart new 'SCRIPT_JOB'
replace into DATA.MY_TABLE
(
C1,
C2
)
values
(
$C1,
$C2
)#
Adding as answer per OP's suggestion:
Per the db2 documentation for the ingest command It appears that the +c: autocommit off will not function:
Updates from the INGEST command are committed at the end of an ingest
operation. The INGEST command issues commits based on the commit_period
and commit_count configuration parameters. As a result of this, the
following do not affect the INGEST command: the CLP -c or +c options, which
normally affect whether the CLP automatically commits the NOT LOGGED
INITIALLY option on the CREATE TABLE statement
You probably want to set the warningcount 1 option, which will cause the command to terminate after the first error or warning. The default behaviour is to continue processing while ignoring all errors (warningcount 0).
I am new to programming world. I have a SQL script which needs to be automated. The automation required is as follows :
1) Script should run every sunday
2) Automatically dump the results in to DUMP_YYYYMMDDHH24MISS.txt
3) Result set is tar gziped
4) upload to SFTP URL with provided username and password.
I am using :
UNIX,
Vertica DB
Can the Gurus here please help ?
This is really 4 questions and should probably asked as such. To answer in the current format though:
1) Schedule a Task Automatically - Crontab
In the terminal, type crontab -e.
If you want something every Sunday at 1am, add the following line:
0 1 * * * 0 /path/to/script/script.sh
This will execute the script every Sunday.
2) Setting the output of the command
I'm only familiar with oracle. The format is probably similar. In order to get the filename as you want it, you'd use the date function as follows. (This is how I would do it in with Oracle):
d=$(date +%Y%M%D%H%M)
var=$(sqlplus -s / as blahblahblah
select * from stuff;
exit
EOF
)
file_name=DUMP_${d}MISS.txt
echo "${var}" >> ${file_name}
Note that your date command is probably different, if you do a man page on date it will tell you which parameters you'd need to get the date formatted as you like.
3) Taring the output
tar -xvf ${file_name}
4) Send over SFTP
You'd have to authenticate the sftp, that is beyond the scope of what anyone can answer without more details. Once you have the machines setup to authenticate, you would do:
sftp username#server<<EOF
put ${file_name}
EOF
I have a directory with a file named file1.txt
And I run the command:
bq query "SELECT * FROM [publicdata:samples.shakespeare] LIMIT 5"
In my local machine it works fine but in Compute Engine I receive this error:
Waiting on bqjob_r2aaecf624e10b8c5_0000014d0537316e_1 ... (0s) Current status: DONE
BigQuery error in query operation: Error processing job 'my-project-id:bqjob_r2aaecf624e10b8c5_0000014d0537316e_1': Field 'file1.txt' not found.
If the directory is empty it works fine. I'm guessing the asterisk is expanding the file(s) into the query but I don't know why.
Apparently the bq command which is located at /usr/bin/bq has the following script:
#!/bin/sh
exec /usr/lib/google-cloud-sdk/bin/bq ${#}
which expands the asterisk.
As a current workaround I'm calling /usr/lib/google-cloud-sdk/bin/bq directly.
I have made recently a huge MSSQL script which was working until the server environment changed and some queries are not anymore allowed. So I had to make a .bat file which executes this query from command line.
I get no error or something else. I just get a file with no entry. But I receive a lot of entries if I use the code of the query in the Management Studio.
Does somebody see where is the mistake in my command line?
I inserted some new lines for reading the code better. Everything except command PAUSE is on one line in the batch file.
EDIT: I figured out that the problem is in the last WHERE clause in the LIKE operater. If I take out the LIKE operater it works. It has nothing todo with the % caracter. it is effective the LIKE operater. Does anybody know how to fix that?
sqlcmd -S connection\string -U user -P password -d dbName -s";" -Q "SET NOCOUNT ON;
SELECT [per_nummer] as EmployeeID,
[per_id] as System_nr,
[per_pid] as PID,
[per_anrede] as Gender,
[per_vname] as Vorname,
[per_name] as Name,
[per_telEx] as Telefon,
[per_email] as Email,
[per_instradierungHauptort] as Instradierung,
[per_gebnr] as Gebaeudenummer,
(SELECT mobileTelephoneNumber FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Mobile,
[per_business_area] as Business_Area,
(SELECT csgdivision FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Division,
NULL as Bereich,
(SELECT csgCompany FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Firma,
[per_sprache] as Korespondenzsprache,
(SELECT roomnumber FROM [dbName].[dbo].[import_zuko_GLDAP_DUMP] WHERE UserID = per_pid ) as Bueronummer,
[per_floor] as Etage,
(SELECT convert(varchar, convert(date,[per_eintrittsdatum]), 104)) as Eintritt_per,
(SELECT convert(varchar, convert(date,[per_austrittsdatum]), 104)) as Austritt_per,
CONVERT(VARCHAR(10), GETDATE(), 104) AS letzte_mutation,
per_lm as Linemanager,
(SELECT TOP 1 per_pid FROM [dbName].[dbo].[person] WHERE per_nummer = master_table.per_lm) AS lm_pid,
(SELECT convert(varchar, convert(date,[per_lm_von]), 104)) as lm_von,
(SELECT convert(varchar, convert(date,[per_lm_bis]), 104)) as lm_bis,
NULL FROM [dbName].[dbo].[person] as master_table
WHERE
[per_nummer] IS NOT NULL AND per_pidStatus = 'A' AND
([per_pid] LIKE('A%')OR [per_pid] LIKE('F%')OR [per_pid] LIKE('W%'))"
-w 1000 -W -o "\\servername\G$\path\to\file.csv"
PAUSE
Different default options
Do you have read carefully sqlcmd Utility documentation by Microsoft?
There is at top the important information:
Because different default options may apply, you might see different behavior when you execute the same query in SQL Server Management Studio in SQLCMD Mode and in the sqlcmd utility.
Therefore the empty file could be caused by different default options.
Space between option and value of option
Further the documentation contains at top:
Currently, sqlcmd does not require a space between the command line option and the value. However, in a future release, a space may be required between the command line option and the value.
Double quotes inside an argument string are often problematic, see answer on Why double quotes should be always only at beginning and end of an argument string?
Therefore I suggest to follow the advice of Microsoft and change -s";" to -s ";" with a space charcter after -s as done for all other options, too.
Escaping percentage sign
Do you have tried to escape each % in the query string with an additional % and using therefore 3 times %% in the query string?
Command line interpreter cmd.exe could interpret the string between two % as a reference to an environment variable and as there is no such environment variable, removes everything between two % from the command line.
Update: It turned out that indeed the not escaped percentage signs resulted in a wrong query string and therefore in an empty results file.
Command with path and extension
General hint:
It is always advisable to specify in batch files commands like sqlcmd with full name which means with path and file extension as this makes the execution of the application independent on the values of the environment variables PATH and PATHEXT.
I assume that all I need to do is to:
Create an sql file e.g. nameofsqlfile.sql contents:
perform proc_my_sql_funtion();
Execute this as a cron job.
However, I don't know the commands that I'd need to write to get this cron job executed as a postgres function for a specified host,port,database, user & his password...?
You just need to think of cronjob as running a shell command at a specified time or day.
So your first job is to work out how to run your shell command.
psql --host host.example.com --port 12345 --dbname nameofdatabase --username postgres < my.sql
You can then just add this to your crontab (I recommend you use crontab -e to avoid breaking things)
# runs your command at 00:00 every day
#
# min hour wday month mday command-to-run
0 0 * * * psql --host host.example.com --port 12345 --dbname nameofdatabase < my.sql
In most cases you can put all of the sql source in a shell 'here document'. The nice thing about here documents is that the shell's ${MY_VAR} are expanded even within single quotes, e.g:
#!/bin/sh
THE_DATABASE=personnel
MY_TABLE=employee
THE_DATE_VARIABLE_NAME=hire_date
THE_MONTH=10
THE_DAY=01
psql ${THE_DATABASE} <<THE_END
SELECT COUNT(*) FROM ${MY_TABLE}
WHERE ${THE_DATE_VARIABLE_NAME} >= '2011-${THE_MONTH}-${THE_DAY}'::DATE
THE_END
YMMV
Check this
http://archives.postgresql.org/pgsql-admin/2000-10/msg00026.php
and
http://www.dbforums.com/postgresql/340741-cron-jobs-postgresql.html
or you can just create a bash script to include your coding and call it from crontab
For Postgresql 10 and above you can use pg_cron. As stated in its README.md,
pg_cron is a simple cron-based job scheduler for PostgreSQL (10 or higher) that runs inside the database as an extension. It uses the same syntax as regular cron, but it allows you to schedule PostgreSQL commands directly from the database: