I have tables with a comment. I added the comment using:
ALTER TABLE table1 SET TBLPROPERTIES ('comment' = 'Hello World!');
ALTER TABLE table2 SET TBLPROPERTIES ('comment' = 'Hello World!');
...
Now my question is, is there a table storing the table properties ?
I want to write a query returning the following data :
+------------+--------------+
| Table | Comment |
+------------+--------------+
| table1 | Hello World! |
| table2 | Hello World! |
+------------+--------------+
Thanks !
Unfortunately, I couldn't find any easier way for a query to return comments than what #Rijul suggested.
But if you're on Cloudera and you just wanna be able to see the comments, this could help:
On Hue query editor, right click on the table (or view) name and choose "show details". Under the tab details, you can see the comment for that table.
Yes their is a embedded data base which stores all the metadata of hive tables schema and other properties.
by default when you setup your hadoop cluster and hive , apache Derby is used for storing hive metadata. although you can change your meta database to postgres or mysql etc while creating your cluster.
so answer to your question is you have to manually install apache derby drivers and through commandline you can query apache derby data base for your desired output, assuming your cluster is using derby. either way you have to find out what is used in your case.
more information on hive metastore:
http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive_metastore_configure.html
information about derby:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreDatabase(Derby)
DESCRIBE FORMATTED tablename;
This command can help you get the comments along with many more information.
Related
I am troubleshooting an application issue on an External (unmanaged) Table that was created using the CREATE TABLE X LIKE PARQUET syntax via Cloudera Impala. I am trying to determine the Location of the files comprising the partitions of the External table but having difficulty determining how to do this, or finding documentation describing this.
If I do a:
show create table T1;
I see the hive-managed location such as:
LOCATION 'hdfs://nameservice1/user/hive/warehouse/databaseName'
If I do a:
describe formatted T1;
I see that the table is in fact external but it doesnt give any insight on the unmanaged Location.
| Table Type: | EXTERNAL_TABLE
| Location: | hdfs://nameservice1/user/hive/warehouse/databaseName/T1
Question:
How do I determine the Location/URI/Parent Directory of the actual external files that comprise this External Table?
When you create a external table with impala or hive and you want know the location you must put the HDFS location, for example :
CREATE EXTERNAL TABLE my_db.table_name
(column string ) LOCATION 'hdfs_path'
The probably location o theses files if dont provite this, is under user directory that execute the comand create table.
For more detail you can see this link:
https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_create_table.html
I hope to help!
I am creating and insert tables in HIVE,and the files are created on HDFS and some on external storage S3
Assuming if I created a 10 tables,is there any system table in Hive where I can find the table info created by the user??? (for example like in Teradata we have DBC.tablesv which hold information of all the user defined tables)
You can find where you metastore is configured to be in the hive-site.xml file.
Its usual location is under /etc/hive/{$hadoop_version}/ or /etc/hive/conf/.
grep for "hive.metastore.uris" or "javax.jdo.option.ConnectionURL" to see which db you are using for the metastore. The credentials should also be there.
If, for example, your metastore is on a MySQL server, you can run queries like
SELECT * FROM TBLS;
SELECT * FROM PARTITIONS;
etc
You can't query (as in SELECT ... FROM...) the metadata from within Hive.
You do however have comnands that display that information, e.g. show databases, show tables, desc MyTable etc.
I'm not sure I understood 100% your question, if you mean the informations about the creation of the table, like the query itself, with the location on HDFS, table properties, etc, you can try with:
SHOW CREATE TABLE <table>;
If you need to retrieve a list of the columns names and datatypes try with:
DESCRIBE <table>;
I have created a table couple months ago. Is there any way in HIVE that I can see when was the table created?
show table doesn't give the date creation of the table.
Execute the command desc formatted <database>.<table_name> on the hive cli. It will show detailed table information similar to
Detailed Table Information
Database:
Owner:
CreateTime:
LastAccessTime:
You need to run the following command:
describe formatted <your_table_name>;
Or if you need this information about a particular partition:
describe formatted <your_table_name> partition (<partition_field>=<value>);
First of all, enable hive support when you create your spark session:
spark = SparkSession.builder.appName('AppName').enableHiveSupport().getOrCreate()
And then:
df_desc = spark.sql('describe formatted <your_table_name>')
df_desc.show()
Is there a Hive query to list only the views available in a particular database.
In MySql the query I think is below:
SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_TYPE LIKE 'VIEW' AND TABLE_SCHEMA LIKE 'database_name';
I want something similar for HiveQL.
There is no INFORMATION_SCHEMA implementation in Hive currently.
There is an Open JIRA which you can view at the following link:
https://issues.apache.org/jira/browse/HIVE-1010
However, if the Hive Metastore is configured using a Derby MySQL server then you can access the information you require.
The different ways of configuring a Hive Metastore can be found at:
http://www.thecloudavenue.com/2013/11/differentWaysOfConfiguringHiveMetastore.html
Here is a detailed E/R Diagram of the Metastore:
https://issues.apache.org/jira/secure/attachment/12471108/HiveMetaStore.pdf
After configuring this Metastore, you can obtain the information you want by a query like:
SELECT * from TBLS where TBLS_TYPE = "VIEW"
If you are stuck like me with older version of Hive and cannot use SHOW VIEWS then use the following script
export db=$1
tables=`hive -e "use $db; show tables;"`
show_table_command=''
for table in $tables; do
show_table_command="$show_table_command SHOW CREATE TABLE ${db}.${table};"
done
hive -e "$show_table_command" > show_table
sed_command="s/$db\.//g"
cat show_table | grep "CREATE VIEW" | cut -d" " -f3 | sed 's/`//g' | sed ${sed_command} > all_views
Run this code from as sh find_views.sh dbname. Now the table all_views has the list of views.
You can also use this same technique to find only tables by replacing "CREATE VIEW" to "CREATE TABLE" or "CREATE EXTERNAL TABLE" and adjusting the sed and cut statements accordingly.
SHOW VIEWS [in/from <dbName>] [<pattern>]
(much like SHOW TABLE command) can be tracked through https://issues.apache.org/jira/browse/HIVE-14558 patch is available and there is a chance it will land in Hive 2.0.
I have created a table in hive, I would like to know which directory my table is created in? I would like to know the path...
DESCRIBE FORMATTED my_table;
or
DESCRIBE FORMATTED my_table PARTITION (my_column='my_value');
There are three ways to describe a table in Hive.
1) To see table primary info of Hive table, use describe table_name; command
2) To see more detailed information about the table, use describe extended table_name; command
3) To see code in a clean manner use describe formatted table_name; command to see all information. also describe all details in a clean manner.
Resource: Hive interview tips
You can use below commands for the same.
show create table <table>;
desc formatted <table>;
describe formatted <table>;
DESCRIBE FORMATTED <tablename>
or
DESCRIBE EXTENDED <tablename>
I prefer formatted because it is more human readable format
To see both of the structure and location (directory) of an any (internal or external)table, we can use table's create statment-
show create table table_name;
in hive 0.1 you can use SHOW CREATE TABLE to find the path where hive store data.
in other versions, there is no good way to do this.
upadted:
thanks Joe K
use DESCRIBE FORMATTED <table> to show table information.
ps: database.tablename is not supported here.
Further to pensz answer you can get more info using:
DESCRIBE EXTENDED my_table;
or
DESCRIBE EXTENDED my_table PARTITION (my_column='my_value');
All HIVE managed tables are stored in the below HDFS location.
hadoop fs -ls /user/hive/warehouse/databasename.db/tablename
If you use Hue, you can browse the table in the Metastore App and then click on 'View file location': that will open the HDFS File Browser in its directory.
in the 'default' directory if you have not specifically mentioned your location.
you can use describe and describe extended to know about the table structure.