How to apply Partition on hive table which is already partitioned. I am not able to fetch the partitioned data into the folder after the data is loaded.
1st rule of partitioning in hive is that the partitionioning column should be the last column in the data. since the data is already partitioned lets say we are partitioning data on gender M/F there will be two directories gender=M and gender=F be created inside each of the directories respective gender data will be available and last column again in this data will be gender.
If you want to partiton data again on partitioned table use insert into select and make sure last column you use is the partition column you want to on the partitioned data.
Did you add a partition manually with the Hdfs command ? In that case metastore will not keep track of partitions being added unless you specify " alter table add partition "...
try this
MSCK REPAIR TABLE table_name;
If that is not the case , then try to drop partitions and create the partitions again . Use alter table command to do this. but you will lose the data . and your partitioning column value should be mentioned as last column in case if you are doing a dynamic partition insert.
Related
This is an extension of a previous question I asked: How to compare two columns with different data type groups
We are exploring the idea of changing the metadata on the table as opposed to performing a CAST operation on the data in SELECT statements. Changing the metadata in the MySQL metastore is easy enough. But, is it possible to have that metadata change applied to partitions (they are daily)? Otherwise, we might be stuck with current and future data being of type BIGINT while the historical is STRING.
Question: Is it possible to change partition meta data in HIVE? If yes, how?
You can change partition column type using this statement:
alter table {table_name} partition column ({column_name} {column_type});
Also you can re-create table definition and change all columns types using these steps:
Make your table external, so it can be dropped without dropping the data
ALTER TABLE abc SET TBLPROPERTIES('EXTERNAL'='TRUE');
Drop table (only metadata will be removed).
Create EXTERNAL table using updated DDL with types changed and with the same LOCATION.
recover partitions:
MSCK [REPAIR] TABLE tablename;
The equivalent command on Amazon Elastic MapReduce (EMR)'s version of Hive is:
ALTER TABLE tablename RECOVER PARTITIONS;
This will add Hive partitions metadata. See manual here: RECOVER PARTITIONS
And finally you can make you table MANAGED again if necessary:
ALTER TABLE tablename SET TBLPROPERTIES('EXTERNAL'='FALSE');
Note: All commands above should be ran in HUE, not MySQL.
You can not change the partition column in hive infact Hive does not support alterting of partitioning columns
Refer : altering partition column type in Hive
You can think of it this way
- Hive stores the data by creating a folder in hdfs with partition column values
- Since if you trying to alter the hive partition it means you are trying to change the whole directory structure and data of hive table which is not possible
exp if you have partitioned on year this is how directory structure looks like
tab1/clientdata/2009/file2
tab1/clientdata/2010/file3
If you want to change the partition column you can perform below steps
Create another hive table with required changes in partition column
Create table new_table ( A int, B String.....)
Load data from previous table
Insert into new_table partition ( B ) select A,B from table Prev_table
I have a partitioned table Student which already has one partition column dept. I need to add new partition column gender
Will it be possible to add this new partition column in already partitioned hive table.
The table data does not have gender column. It is a new constant column to be added in hive table.
Partitions are hierarchical folders like table_location/dept=Accounting/gender=male/
Folder structure should exist. You can easily add non-partition column as the last one and it will return NULLs if the data does not contain that column, but to add a partition column the easiest way is to create new table partitioned as you want, insert overwrite that table from the old one (selecting partitions columns as the last ones), drop old table, rename new one.
See this answer about dynamic partitions load: https://stackoverflow.com/a/48901871/2700344
Hi my table has 150 columns and i dont want to mention the column name while doing partition. Is there any way using temp table to create partitioned table from non partitioned one.
Temporary table in Hive doesn't support Partitioning.
You cannot create any permanent partitioned table without mentioning Partition
Column.
We can create a partition on BigQuery table while creating a BigQuery table.
I have some questions on the partition.
How to append data to an existing partition in the BigQuery table.
How to create a new Partition in an existing BigQuery table if there is already partition present in that BiQuery table.
How to do truncate and load data to a partition in the BigQuery table(overwrite data in a partition in the BigQuery table).
How to append data to an existing partition in the BigQuery table.
Either you do this from Web UI or with API or with any client of your choice - the approach is the same - you just set your Destination Table with respective partition decorator, like below as an example
yourProject.yourDataset.youTable$20171010
Please note: to append your data - you need to use Append to table for Write Preference
How to create a new Partition in an existing BigQuery table if there is already partition present in that BiQuery table.
If the partition you set in decorator of destination table does not exist yet - it will be added for you
How to do truncate and load data to a partition in the BigQuery table(overwrite data in a partition in the BigQuery table).
To truncate and load to a specific partition - you should use Overwrite table for Write Preference
I want to use the hive dynamic partitioning to overwrite a partitioned table "page_view":
INSERT OVERWRITE TABLE page_view PARTITION(date)
SELECT pvs.viewTime FROM page_view_stg pvs
My question is : If the table "page_view_stg" only has the data of "date=2017-01-01", but the dest table "page_view" has a partition "date=2017-01-02". So after running this query, will the partition "date=2017-01-02" get dropped or not? If not, how should I handle this case using dynamic partitioning?
Thanks
Query with dynamic partitioning will overwrite only partitions existing in the source dataset. In your case partition "date=2017-01-02" will remain unchanged if the the source table does not contain such date. If you want to drop it, the fastest method is to execute alter table drop partition statement because this is the metadata operation. You select partitions from target table which do not exist in the source and generate drop statements using shell. Or insert into new table, drop old target, then rename.