I'm new to Apache Drill.
For performance testing purpose, I'm trying to measure the time to execute a query. And also I do not need to print the executed result.
In Oracle SQL Plus, there is set autotrace traceonly. This setting feature is the following(quoting from the oracle web site):
Similar to SET AUTOTRACE ON, but suppresses the printing of the user's query output, if any. If STATISTICS is enabled, query data is still fetched, but not printed.
In Apache Drill's sqlline, I got the error like the following: Error: PARSE ERROR: Encountered "traceonly" at line 1, column 15...
Do you have any ideas for alternatives?
Thanks,
p.s.
I also read this answered question. Any command in mysql equivalent to Oracle's autotrace for performance turning
Unfortunately, it doesn't work on Apache Drill.
You could put your query(s) in a text file (e.g. query.sql), then run sqlline and forward the output to /dev/null:
bin/sqlline -u jdbc:drill:zk=localhost:2181 -f query.sql > /dev/null
It will still display some data, but only minimal:
1/1 select * from cp.`employee.json`;
1,155 rows selected (0.65 seconds)
Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
apache drill 1.4.0
"say hello to my little drill"
Related
I need to measure the execution time in a query in the ALTIBASE DBMS.
For example, I need to now in ms the time it took a SELECT * FROM example;
I tried to use the methods that have some traditional DBMS like MySQL or Oracle and doesn't work.
There are some method. The first one you can use the log from altibase to analyze. And the other method is to coding some program that recording time from using the SELECT to get result. Now, I also learn to use altibase - We can together.
I found the solution
After entering to the client with next command isql -u sys -p manager -sysdba you need to execute the next query
SET TIMING ON;
Next the following queries will have the execution time.
I am attempting to tune our oracle database that has been running a little slowly lately.
I have generated a SQL trace file, and can run the basic TKPROF from the command prompt, and generate the appropriate output file.
tkprof.exe source.trc output.txt
I would very much like to see the execution plan as well since there are a good number of indexes that should be used with this database. To do this, I am trying to run this:
tkprof.exe source.trc output.txt EXPLAIN=mbw/password
The problem is, the username which every application uses to connect with is lowercase (mbw for this example, and I have to leave it this way). So whenever I wish to look at data, I have to put quotes around the user like this:
SELECT * FROM "mbw".TABLE1
Unfortunately, I can never seem to get TKPROF to connect as "mbw"/password, it is always as mbw/password, which will never work. (I can see what TKPROF is attempting to connect with in the output.txt file)
I have tried a bunch of permutations on the command line, and just can't seem to make it happen. I've tried things like:
... EXPLAIN="mbw"/password
... EXPLAIN=""mbw""/password
... EXPLAIN="""mbw"""/password
... EXPLAIN=^"mbw^"/password
Does anyone have any ideas on how to properly structure the TKPROF command so I can connect as a user with lowercase letters?
I fully apologize for my lack of good oracle and sql skills, I have been rather unexpectedly thrown into this particular job and am trying to learn as fast as I can.
I'm totally new in running sql queries in linux and I'm having a hard time dealing with it's output.
So I managed to access my database in oracle in linux and trying to run a simple query right now:
SELECT IN_01, OUT_BD_01 FROM TRANSLATION_ROW WHERE IN_01 = 'LS3K5GB';
I'm expecting it to be in a table-like output but instead i got this:
Any Help would be much appreciated. By the way, I'm accessing my Oracle server through putty. I don't know if that helps in anything.
--forgot to mention that I also use sqlplus. Don't know if that would make any difference
Thanks in advance.
Welcome to the weird and wonderful world of Oracle.
Viewing large amounts of data (especially "wide" data) through sqlplus has always been less than pretty. Even back in the 1990s Oracle rival Ingres had a rather nice isql which made a much better fist of this, although the flipside of that was using isql to spool to a data file (no headers and trimmings, etc) was slightly harder. I think the rather primitive nature of SQLPLus is why TOAD/SQL*Developer etc have become popular.
To make the output easier to read, you need to learn the basics of sqlplus formatting, in particular SET LINES, PAGES, TRIMSPOOL, TAB, and the COLUMN formatting command.
Use COLUMN to control the formatting of each column.
One possible option is to use SET MARKUP and spool to a file, which formats the output as HTML table, but then you need a HTML viewer/browser to view the results.
On PuTTY your options are limited, but if you have xterm and can invoke the browser on Linux, you might find something like a shell script:
#!/bin/bash
sqlplus un/pw #the_file
firefox the_output.html
Contents of the_file.sql:
SET MARKUP ON
spool the_output.html
SELECT * FROM user_objects;
spool off
quit
If you have a share between the Linux system where the the_output.html resides and can mount that on WIndows, you could run the query on Linux with MARKUP oN, spool to the share, then click refresh on the Browser.
Clunky, and not really what you want, but try it and see what you get.
It displays the entire column that's it.
You can format your column before running the query with the below:
e.g.: format my column to display 10 characters only
column IN_01 format a10
There are some basic configuration tricks that you should apply when using SQLplus. A basic set of parameters would be something like this:
set pagesize 50000
set linesize 135
set long 50000
set trimspool on
set tab off
All these should be placed in a login.sql file which should be in the directory you are launching sqlplus from.
This will solve your current problem, but for further reading I suggest checking out this page: Configuring sqlplus.
Is there a programmatic way to validate HiveQL statements for errors like basic syntax mistakes? I'd like to check statements before sending them off to Elastic Map Reduce in order to save debugging time.
Yes there is!
It's pretty easy actually.
Steps:
1. Get a hive thrift client in your language.
I'm in ruby so I use this wrapper - https://github.com/forward/rbhive (gem install rbhive)
If you're not in ruby, you can download the hive source and run thrift on the included thrift configuration files to generate client code in most languages.
2. Connect to hive on port 10001 and run a describe query
In ruby this looks like this:
RBHive.connect(host, port) do |connection|
connection.fetch("describe select * from categories limit 10")
end
If the query is invalid the client will throw an exception with details of why the syntax is invalid. Describe will return you a query tree if the syntax IS valid (which you can ignore in this case)
Hope that helps.
"describe select * from categories limit 10" didn't work for me.
Maybe this is related to the Hive version one is using.
I'm using Hive 0.8.1.4
After doing some research I found a similar solution to the one Matthew Rathbone provided:
Hive provides an EXPLAIN command that shows the execution plan for a query. The syntax for this statement is as follows:
EXPLAIN [EXTENDED] query
So for everyone who's also using rbhive:
RBHive.connect(host, port) do |c|
c.execute("explain select * from categories limit 10")
end
Note that you have to substitute c.fetch with c.execute, since explain won't return any results if it succeeds => rbhive will throw an exception no matter if your syntax is correct or not.
execute will throw an exception if you've got an syntax error or if the table / column you are querying doesn't exist. If everything is fine, no exception is thrown but also you'll receive no results, which is not an evil thing
In the latest version hive 2.0 comes with hplsql tool which allows us to validate hive commands without actually running them.
Configuration:
add the below XML in hive/conf folder and restart hive
https://github.com/apache/hive/blob/master/hplsql/src/main/resources/hplsql-site.xml
To Run the hplsql and validate the query , please use the below command:
To validate Singe Query
hplsql -offline -trace -e 'select * from sample'
(or)
To Validate Entire File
hplsql -offline -trace -f samplehql.sql
If the query syntax is correct , the response from hplsql would be something like this:
Ln:1 SELECT // type
Ln:1 select * from sample // command
Ln:1 Not executed - offline mode set // execution status
if the query Syntax is wrong , the syntax issue in the query will be reported
If the hive version is older, we need to manually place the hplsql jars inside the hive/lib and proceed.
I AM NOT RUNNING THE COMMANDS FROM PHP!
I have MySQL log_error value set to /var/log/mysql/error.log
However when I connect to a database and run an SQL command, the error does not appear in the log.
SELECT *
FROM some_table
where this_is_invalid_and_will_fail="Doesn't matter because column doesn't exist!";
There are commands running from some sort of windows application. All I want to know is what invalid commands its sending to the MySQL server so that I can attempt to resolve them.
Error log doesn't do that:
https://dev.mysql.com/doc/refman/8.0/en/error-log.html
The error log contains information
indicating when mysqld was started and
stopped and also any critical errors
that occur while the server is
running.
MySQL doesn't log invalid/failed queries anywhere.
If it's for debugging purposes, you might try setting up a MySQL Proxy, which could log this I think:
http://dev.mysql.com/downloads/mysql-proxy/
Basically, there are 2 ways.
1) setup some kind of proxy, which can log error queries. There are a lot of forks of the original mysql proxy (which was mentioned by #Mchl) - https://github.com/search?utf8=%E2%9C%93&q=MySQL+Proxy Also you can find latest version of the original mysql proxy here https://github.com/mysql/mysql-proxy
I dont like this way, because its kind of too complicated for quick debug
2) you can enable general log in mysql (which will log ALL queries).
It will create big overhead and doesnt fit production requirements, but its fast and easy way to fix bugs in development environment.
Just login to your mysql and execute
SET GLOBAL general_log = 'ON';
SHOW GLOBAL VARIABLES LIKE 'general_log%';
You will see logs location, like
+------------------+------------------------+
| Variable_name | Value |
+------------------+------------------------+
| general_log | OFF |
| general_log_file | /mnt/ssd/mysql/s.log |
+------------------+------------------------+
After that execute
mysqladmin flush-logs -u root -p
When you will need to stop logs - just execute
SET GLOBAL general_log = 'OFF';
As of mysql version 5.6.3 (released 2011-10-03), there is a variable called log-raw which allows you to include invalid queries in the general query log:
https://dev.mysql.com/doc/refman/5.6/en/server-options.html#option_mysqld_log-raw
You will need to turn on general query log using general-log and general-log-file, e.g.:
[mysqld]
general-log=1
log-raw=1
general-log-file=/var/log/mysql/general.log
Could you enable the General Query Log? That should tell you everything you need to know.
This is not so trivial. The best way to do this, is to log bad queries in your application. There is no built-in way.