I have a Hive (ver 0.11.0) table partitioned by column date, of type string. I want to know if there exists a way in Hive by which I can drop partitions for a range of dates (say from 'date1' to 'date2'). I have tried the following (SQL type) queries, but they don't seem to be syntactically correct:
ALTER TABLE myTable DROP IF EXISTS PARTITION
(date>='date1' and date<='date2');
ALTER TABLE myTable DROP IF EXISTS PARTITION
(date>='date1' && date<='date2');
ALTER TABLE myTable DROP IF EXISTS PARTITION
(date between 'date1' and 'date2');
I tried this syntax it worked.
ALTER TABLE mytable DROP PARTITION (dates>'2018-04-14',dates<'2018-04-16');
Command output:
Dropped the partition dates=2018-04-15/country_id=107
Dropped the partition dates=2018-04-15/country_id=110
Dropped the partition dates=2018-04-15/country_id=112
Dropped the partition dates=2018-04-15/country_id=14
Dropped the partition dates=2018-04-15/country_id=157
Dropped the partition dates=2018-04-15/country_id=159
Dropped the partition dates=2018-04-15/country_id=177
Dropped the partition dates=2018-04-15/country_id=208
Dropped the partition dates=2018-04-15/country_id=22
Dropped the partition dates=2018-04-15/country_id=233
Dropped the partition dates=2018-04-15/country_id=234
Dropped the partition dates=2018-04-15/country_id=76
Dropped the partition dates=2018-04-15/country_id=83
OK
Time taken: 0.706 seconds
I am using, Hive 1.2.1000.2.5.5.0-157
Solution: alter table myTable drop partition (unix_timestamp('date1','yyyy-MM-dd')>unix_timestamp(myDate,'yyyy-MM-dd'),unix_timestamp('date2','yyyy-MM-dd')<unix_timestamp(myDate,'yyyy-MM-dd'));
Related
1- I'm trying to delete multiple partitions at once, but struggling to do it with either Impala or Hive. I tried the following query, with and without ':
ALTER TABLE cz_prd_corrti_st.s1mme_transstats_info DROP IF EXISTS
PARTITION (pr_load_time='20170701000317')
PARTITION (pr_load_time='20170701000831')
The error I'm getting is as follow:
AnalysisException:
Syntax error in line 3: PARTITION (pr_load_time='20170701000831') ^
Encountered: PARTITION Expected: CACHED, LOCATION, PURGE, SET,
UNCACHED CAUSED BY: Exception: Syntax error
The partition column is bigint type, query for deleting only one partition works as expected:
ALTER TABLE cz_prd_corrti_st.s1mme_transstats_info DROP IF EXISTS
PARTITION (pr_load_time='20170701000317')
2- Is it a good practice delete the hdfs data first and then drop the partitions in Impala/Hive, or is it supposed to be done vice versa?
1.
Your syntax is wrong.
In the DROP command the partitions should be separated by commas.
Demo
hive> create table t (i int) partitioned by (p int);
OK
hive> alter table t add partition (p=1) partition(p=2) partition(p=3) partition(p=4) partition(p=5);
OK
hive> show partitions t;
OK
partition
p=1
p=2
p=3
p=4
p=5
hive> alter table t drop if exists partition (p=1),partition (p=2),partition(p=3);
Dropped the partition p=1
Dropped the partition p=2
Dropped the partition p=3
OK
hive> show partitions t;
OK
partition
p=4
p=5
2.
You can drop a range.
Demo
hive> create table t (i int) partitioned by (p int);
OK
hive> alter table t add partition (p=1) partition(p=2) partition(p=3) partition(p=4) partition(p=5);
OK
hive> show partitions t;
OK
partition
p=1
p=2
p=3
p=4
p=5
hive> alter table t drop if exists partition (p<=3);
Dropped the partition p=1
Dropped the partition p=2
Dropped the partition p=3
OK
hive> show partitions t;
OK
partition
p=4
p=5
I have an HIVE table with daily partitions day wise, something like below (which includes future date's partition as well)
20160901
20160902
........
........
........
20160931
20161001
20161002
I want to pass one date say for example yesterday's date 20160922 and want to drop all partitions dynamically which are >= 20160922 (though today is 20160923, but I want to drop from 20160922 date).
How can I can drop all these partitions dynamically.
You can not do in hive directly as it does not support dynamic sql.
There can be work around using shell script/or any script create file having drop partition script like below.
alter table partition_t drop if exists partition (y=20160922 );
alter table partition_t drop if exists partition (y=20160921 );
alter table partition_t drop if exists partition (y=20160920 );
...
then run hive -v -f ./file.sh
alter table partition_t drop if exists partition
Before Inserting Data Into Table Perform the below steps.
1) Go to Hdfs Folder of that table and delete all the folders Inside
Table Directory using Shell Commands. hadoop fs -rm r <>
2) Run MSCK repair Table to update the metadata about partitions.
above two steps will delete all the available partitions based on pattern.
Now Insert your new data.
You can drop partitions giving a range filter. For reference see that answer : https://stackoverflow.com/a/48422251/3132181
So your code could be like that:
Alter table mytable drop partition (datehour >= '20160922')
I have a drop alter query , i want to set limit for it.
Query is :
alter table dim_known_hosts drop if exists PARTITION (dimensional_partition_folder<'2016_01_04_00_30');
It drops all partition less than 2016_01_04_00_30.
I want to delete just first 10.
I use hive-0.10.0-cdh-4.7.0 in my environment.
I have a table named test store as sequence file and some partitions by date_dim like below:
game=Test/date_dim=2014-07-01
game=Test/date_dim=2014-07-11
game=Test/date_dim=2014-07-21
game=Test/date_dim=2014-07-31
I want to drop partitions between 2014-07-21 and 2014-07-30 in SQL command:
alter table test drop partition (date_dim>='2014-07-11',date_dim<='2014-07-30')
I hope these 2 partitions be deleted:
game=Test/date_dim=2014-07-11
game=Test/date_dim=2014-07-21
But actually, these 3 partitions be deleted:
game=Test/date_dim=2014-07-01
game=Test/date_dim=2014-07-11
game=Test/date_dim=2014-07-21
It seems hive drop partition only use the date_dim<='2014-07-30' condition.
Is there anyway to make hive drop partition as I wish?
You should convert the string to the date type, for that purpose you can use unix_timestamp function:
alter table test drop partition (unix_timestamp(date_dim,'yyyy-MM-dd')>=unix_timestamp('2014-07-11','yyyy-MM-dd'),unix_timestamp(date_dim,'yyyy-MM-dd')<=unix_timestamp('2014-07-30','yyyy-MM-dd'))
After adding a partition to an external table in Hive, how can I update/drop it?
You can update a Hive partition by, for example:
ALTER TABLE logs PARTITION(year = 2012, month = 12, day = 18)
SET LOCATION 'hdfs://user/darcy/logs/2012/12/18';
This command does not move the old data, nor does it delete the old data. It simply sets the partition to the new location.
To drop a partition, you can do
ALTER TABLE logs DROP IF EXISTS PARTITION(year = 2012, month = 12, day = 18);
in addition, you can drop multiple partitions from one statement (Dropping multiple partitions in Impala/Hive).
Extract from above link:
hive> alter table t drop if exists partition (p=1),partition (p=2),partition(p=3);
Dropped the partition p=1
Dropped the partition p=2
Dropped the partition p=3
OK
EDIT 1:
Also, you can drop bulk using a condition sign (>,<,<>), for example:
Alter table t
drop partition (PART_COL>1);
Alter table table_name drop partition (partition_name);
You can either copy files into the folder where external partition is located or use
INSERT OVERWRITE TABLE tablename1 PARTITION (partcol1=val1, partcol2=val2...)...
statement.
You may also need to make database containing table active
use [dbname]
otherwise you may get error (even if you specify database i.e. dbname.table )
FAILED Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter partition. Unable to alter partitions because table or database does not exist.