I'm doing a hive performance test, and I want to output the query result
like this:
Time taken is a STDERR output
if you use redirection result to a file " > x.log " to save result, add " 2>&1 " to redirect STDERR to STDOUT .
Related
I have a beeline query where I'm passing (-f) a file named as "some.sql" which is having multiple queries to be executed. But one of them failed then does it return 0 or some non zero value? please help me with this. I would like to capture and handle this situation.
The return code will be a non-zero value if atleast one of the queries in the file fails. Beeline will not execute other queries in the script after the failed one, if there are any. It is better to have one query per file.
A sample bash script.
#!/bin/bash
beeline -u $url -f queries.sql
rc=$?
if [ $rc -ne 0 ]
then
echo "return code is $rc. One or more queries in the file failed"
else
echo "return code is $rc. All queries executed successfully"
fi
You can also add printf statements after each query in the queries file to know the queries that executed successfully.
I'm trying to run a big query query from the command line, but because my query is very long I've written it in a text file. The query works from the GUI and I'm overwriting a table that already exsists
bq query --allow_large_results --replace --destination_table=me.Tbl_MyTable '`cat query.txt`'
However, I'm getting error results:
Error in query string: Error processing job
'dev:bqjob_r_00000123456789456123_1': Encountered "
"\'cat query.txt\' "" at line 1, column 1.
Was expecting: EOF
Do I need to put the entire file path in the .txt filename? (this doesn't seem to make a difference)
Are there any characters I need to be careful with in the text file (e.g. "\" or quotation marks) ?
I'm using where clauses and group by clauses - is that an issue?
Instead of cat, just pipe the input from the file. The command would be:
bq query --allow_large_results --replace --destination_table=me.Tbl_MyTable < query.txt
This will send the contents of query.txt to the bq tool.
Elliot is right, now if you want to cat, sed or anything, pipe it:
cat query.txt | bq query
I have a table called query_master table which has 4 columns and the 4th column has SQL query as values. In total there are 5 entries in the query table.
Table Structure:
S.No --> Key --> Title --> Query
1 100 EG select * from dual
Now my objective is, I have to fetch the SQL queries using shell script from the query_master and execute it. The output of that each SQL query should be written on a separate log file, and the log filename should be equal to the name of the title.
Can you please help in achieving this scenario using stored procedures or stored functions which will be more helpful for me.
I need to achieve this using shell scripting.
Try this, assuming you're using mysql:
awk -F'\t' 'NR!=1 {system("mysql -u user -p -e " $4 " database")}' file
Where file is the file containing the table, user is the user and database is the database. Alternatively set these as variables instead of hard coding them like this:
awk -F'\t' -v db="database" -v user="user" 'NR!=1 {system(""mysql -u " user " -p -e " $4 " " db)}' file
Make a shell script that accepts a SQL statement from commandline (or inputfile or stdin) and does all things for you like exporting ORACLE_HOME, tnsnames, username, password, redirecting output, calling sqlplus, output formatting, deleting column headers and other sqlplus settings.
With your magicsql.sh (after testing), aim for a solution like
magicsql.sh "select key, query from query_master order by key" | while read key query; do
magicsql.sh "${query}" > /tmp/${key}.out
done
I have a pig script where in the beginning I would like to generate a string of the dates of the past 7 days from a certain date (later used to retrieve log files for those days).
I attempt to do this with this line:
%declare CMD7 input= ; for i in {1..6}; do d=$(date -d "$DATE -i days" "+%Y-%m-%d"); input="\$input\$d,"; done; echo \$input
I get an error :
" ERROR 2999: Unexpected internal error. Error executing shell command: input= ; for i in {1..6}; do d=$(date -d "2012-07-10 -i days" "+%Y-%m-%d"); input="$input$d,"; done;. Command exit with exit code of 127"
however the shell command runs perfectly fine outside of pig. I am really not sure what is going wrong here.
Thank you!
I have got a working solution but not as streamlined as you want, essentially I don't manage to get Pig to execute a complex shell statement in the declare.
I first wrote a shell script (let's call it 6-days-back-from.sh):
#!/bin/bash
DATE=$1
for i in {1..6}; do d=$( date -d "$DATE -$i days" +%F ) ; echo -n "$d "; done
Then a pig script as follow (let's call it days.pig):
%declare my_date `./6-days-back-from.sh $DATE`
A = LOAD 'dual' USING PigStorage();
B = FOREACH A GENERATE '$my_date';
DUMP B
note that dual is a directory containing a text file with a single line of text, for the purpose of displaying our variable
I called the script as follow:
pig -x local -param DATE="2012-08-03" days.pig
and got the following output:
({(2012-08-02),(2012-08-01),(2012-07-31),(2012-07-30),(2012-07-29),(2012-07-28)})
i have a problem which i cant figure out myself, i have a SQLite database which contains data
to retrieve the data i use a bash script and the command sqlite3 db "SELECT prkey FROM printers"
the output is something like this:
1
3
4
5
6
7
i need to parse each line and use it for my next command. and with out the use of a > file
regards Marco
The for loops work, but they are not very robust: they break if a line in the output has a space, and the shell must store the whole output of the query in memory before it starts running the loop. These two issues are easily dealt with:
sqlite3 db "SELECT ..." | while read prkey; do
echo "do stuff with $prkey"
done
#!/bin/bash
for line in $(sqlite3 db "SELECT prkey FROM printers"); do
echo "line is --> $line" # do stuff with $line
done
You can use for loop on the return value of the sqlite command.
for prkey in `sqlite3 db "SELECT prkey FROM printers"`; do
echo $prkey; #replace with your command
done;