I'm trying to set up a monitoring script that would take all the databases we have, showed tables and done some arithmetics on it.
I have this command:
impala-shell -i impalad -q " show databases;" -B | while read a; do impala-shell -q "show tables in ${a}" -B -i impalad; done
That produces following output:
Query: show tables in database1
table1
table2
How should I format the output to display the database name($a) with each table? I tried echoing it or || but this only prints the database name after displaying all the tables. Or is there a way how to pass the variable to awk?
Desired output would look like this:
database1.table1
database1.table2
It looks like the output of the show tables ... command will have a 1-line header, followed by the list of table names.
You could skip the first line by piping to tail -n +2,
and then use another while loop to echo the database name and table name pairs in the desired format:
impala-shell -i impalad -q " show databases;" -B | while read a; do
impala-shell -q "show tables in ${a}" -B -i impalad | tail -n +2 | while read table; do
echo $a.$table
done
done
You could also do
impala-shell -q ... | awk -v db="$a" 'NR > 1 {print db "." $0}'
Related
I have a file whose filename I am storing in a shell variable and I wish to pass that variable in the WHERE condition of my SQL select query. How can I achieve this ?
my code
cd /path/to/folder
var =$(ls tail)
id_var=$(echo "$var" | cut -f 1 -d '.')
...
...
sqlplus -s user/pwd#db < mysql.sql > output.txt
cat mysql.sql
select * from Records where "GlobalId"='$id_var'
From this answer:
cd /path/to/folder
var =$(ls tail)
id_var=$(echo "$var" | cut -f 1 -d '.')
sqlplus -s user/pwd#db #mysql.sql "${id_var}" > output.txt
Then in mysql.sql use &1 to substitute the first start argument:
select * from Records where "GlobalId"='&1'
Note: &1 is a substitution variable (and not a bind variable) so you will need to make sure that the value passed in does not perform any SQL injection attacks.
You can export the variable
export id_var
Then use envsubst command
envsubst < mysql.sql
This will substitute your variable.
We have used policy tag in bigquery for column level security. ( https://cloud.google.com/bigquery/docs/best-practices-policy-tags) . Now we want to check the list of tables and attributes which have the policy tag. Is there any way in bigquery to get it using INFORMATION_SCHEMA? Or any other approach pragmatically to get the attribute tagged with some policy tag?
You can try this solution, which will give you list of tables and columns where PolicyTag is used:
Write Table-List into a File:
DATASET="dataset-name"
bq ls --max_results=10000 ${DATASET} | awk '{ print $1 }' | sed '1,2d' > table_list.txt
Shell Script:
#!/bin/bash
DATASET="dataset-name"
echo "------------------------------"
echo "TableName ColumnName"
echo "------------------------------"
while IFS= read -r TABLE; do
TAG_COUNT="`bq show --schema ${DATASET}.${TABLE} | grep "policyTags" | wc -l`"
if [ "${TAG_COUNT}" -ge 1 ]
then
COLUMN="`bq show --format=prettyjson ${DATASET}.${TABLE} | jq '.schema.fields[] | select(.policyTags | length>=1)' | jq '.name'`"
echo "${TABLE} ${COLUMN}"
fi
done < table_list.txt
I have created a a script that execute sql (Sybase)
#!/bin/bash
command=$(
isql -U databasename_dba -P password -b <<EOF!
select label1, label2 from TABLE
go
EOF!
)
echo "$command" >> output_file.csv):
All good so far, the file is produced:
But as you can see, the output is represented in 1 column.
Is possible to add "Headers" and divide the column is 2 columns, my desired output would be:
Try to remove -b.
#!/bin/bash
command=$(
isql -U databasename_dba -P password <<EOF!
select label1, label2 from TABLE
go
EOF!
)
echo "$command" >> output_file.csv):
There is probably a very simple solution here, but I am probably not using the right search terms. I have a sql query running in a shell script. I get the results I am looking for, however, I am also getting the sql query as part of of the result. How can I suppress this and just show the result?
My script:
#!/usr/bin/sh
db2 connect to MYDB >/dev/null 2>&1;
db2 -x -v "select A, B, C from MYTABLE";
db2 connect reset >/dev/null 2>&1;
And my output looks like this:
select A, B, C from MYTABLE
AAA BBB CCC
AAA BBB CCC
I would like to get rid of the first row and just show the result. What am I missing?
Thanks in advance for your help!
The -v option for the DB2 command line processor causes the current statement being executed to be printed in the output.
Remove the -v from your command and you'll get only the results of the query.
if you just want to skip the 1st row from your output you could:
yourscript.sh | tail -n +2
test with seq:
kent$ seq 5|tail -n +2
2
3
4
5
Try this
db2 -o query
for more info. http://www.ibm.com/developerworks/data/library/techarticle/adamache/0109adamache.html
I need to process the output of a single record psql query through awk before assigning it to a value in my expect script.
The relevant code:
spawn $env(SHELL)
send "psql -U safeuser -h db test -c \"SELECT foo((SELECT id FROM table where ((table.col1 = \'$user\' AND table.col2 IS NULL) OR table.col2 = \'$user\') AND is_active LIMIT 1));\" | /bin/awk {{NR=3}} {{ print $1 }}; \r"
expect "assword for user safeuser:"
send "$safeuserpw\r"
expect -re '*'
set userpass $expect_out(0, string)
When I run the script, I get:
spawn /bin/bash
can't read "1": no such variable
"send "psql -U safeuser -h db test -c \"SELECT foo((SELECT id FROM table where ((table.col1 = \'$user\' AND table.col2..."
Is there something glaring that I'm missing here? I was under the impression that the double curly-brackets protected the awk code block.
The awk script will show all lines because you're using '=' instead of '==' in the conditional expression. Try the following:
spawn $env(SHELL)
send "psql -U safeuser -h db test -c \"SELECT foo((SELECT id FROM table where ((table.col1 = \'$user\' AND table.col2 IS NULL) OR table.col2 = \'$user\') AND is_active LIMIT 1));\" | /bin/awk \'NR==3 { print $1 }\'; \r"
expect "assword for user safeuser:"
send "$safeuserpw\r"
expect -re '*'
set userpass $expect_out(0, string)
Your send line is being evaluated by tcl because it is in quotes "". if you want to pass it as it should be you should change your awk portion to escape the $ :
...| /bin/awk \'NR==3 { print \$1 }\'; \r"