I want to launch this command on my hive table
Delete from customer where id=3;
And i had this error
FAILED: SemanticException: [Error 10294]: Attempt to do update or delete using transactiob manager that does not support these operations.
Who can help me please??
You can not delete from any table in hive just like this. Thetable should be transactional.
Couple of steps has to be performed to enable this.
Set hive params
# this must be set to true. Hive uses txn manager to do DML.
SET hive.support.concurrency=true;
SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
# The follwoing are not required if you are using Hive 2.0
SET hive.enforce.bucketing=true;
SET hive.exec.dynamic.partition.mode=nostrict;
# required for standalone hive metastore
SET hive.compactor.initiator.on=true;
SET hive.compactor.worker.threads=1
Create table in orc format with table properties= true and having buckets(Hive 2.0).
Sample create table is -
CREATE TABLE db.emptrans (
id int,
name string)
STORED AS ORC
TBLPROPERTIES ('transactional'='true');
Pls refer to below answer for more details -
How to do update/delete operations on non-transactional table
Related
I am creating a managed table via Impala as follows:
CREATE TABLE IF NOT EXISTS table_name
STORED AS parquet
TBLPROPERTIES ('transactional'='false', 'insert_only'='false')
AS ...
This should result in a managed table which does not support HIVE-ACID.
However, when I run the command I still end up with an external table.
Why is this?
I found out in the Cloudera documentation that neglecting the EXTERNAL-keyword when creating the table does not mean that the table definetly will be managed:
When you use EXTERNAL keyword in the CREATE TABLE statement, HMS stores the table as an external table. When you omit the EXTERNAL keyword and create a managed table, or ingest a managed table, HMS might translate the table into an external table or the table creation can fail, depending on the table properties.
Thus, setting transactional=false and insert_only=false leads to an External Table in the interpretation of the Hive Metastore.
Interestingly, only setting TBLPROPERTIES ('transactional'='false') is completly ignored and will still result in a managed table having transactional=true).
We are thinking in create a housekeeping program to delete old partitions from HIVE tables. After reading the documentation, I found that you can establish a retention period using SET TBLPROPERTIES.
ALTER TABLE employees SET TBLPROPERTIES ('discover.partitions'='true');
ALTER TABLE employees SET TBLPROPERTIES ('partition.retention.period'='7d');
My questions are the following:
After executing this command HIVE will delete old partitions ? In
other systems, normally setting properties don't affect to existing
elements.
When this clean-up takes place ? I did not see a way to configure it ? What happens if someone is accessing the partition in the moment the process is triggered ?
Or it's just best to just create an script to read the metadata dictionary
in HIVE and delete old partitions ?
Thanks for your help!
I want to create HIVE table with the transactional table with TBLPROPERTIES ("transactional"="true") set using create table statement. instead of setting for all tables can I set TBLPROPERTIES using hive-site.xml.
Unfortunately, we can't set it in hive-site.xml since transactional is a per table property. And we should not do it that way beacause 'transactional table' comes with some prerequisites and limitations.
having problem with impala update statement, when I used code below
update john_estares_db.tempdbhue set QU=concat(account_id,"Q",quarter(mrs_change_date)," ",year(mrs_change_date));
it return error message:
AnalysisException: Impala does not support modifying a non-Kudu table: john_estares_db.tempdbhue
I would like to know if I can either change my non-Kudu table into a Kudu table or is there an alternate for update statement for non-Kudu in Impala. TIA
Apache Kudu is a data store (think of it as alternative to HDFS/S3 but stores only structured data) which allows updates based on primary key. This has good integration with Impala. A kudu table on Imapla is a way to query data stored on Kudu.
In short if you do not already have Kudu installed and setup already you cannot create a kudu table on Impala.
If you have it Kudu installed and setup, then you cannot simply convert a table kudu table. You have to create a new kudu table with similar structure with some primary-key columns (Kudu requires primary key for all tables) and insert data into this from old non-kudu table using sql query insert into .. select * from ....
What is the type of table john_estares_db.tempdbhue?
Hive or other table type, update or upsert is not supported.
You can use show create table to check your table type.
show create table
If you have kudu installed you can create a kudu table, and move your data into kudu table,then you can use your update code.
I have set all the parameters that needs to be set in hive for using transactions.
set hive.support.concurrency=true;
set hive.enforce.bucketing=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.compactor.initiator.on=true;
set hive.compactor.worker.threads=0;
Created table using below command
CREATE TABLE Employee(Emp_id int,name string,company string,Desg string) clustered by (Emp_id) into 5 buckets stored as orc TBLPROPERTIES(‘transactional’=’true’);
Inserted Data in hive table by using below command
INSERT INTO table Employee values(1,’Jigyasa’,’Infosys’,’Senior System Engineer’), (2,’Pooja’,’HCL’,’Consultant’), (3,’Ayush’,’Asia Tours an travels’,’Manager’), (4,’Tarun’,’Dell’,’Architect’), (5,’Namrata’,’Apolo’,’Doctor’);
But while Updating the data
Update Employee set Company=’Ganga Ram’ where Emp_id=5;
I am getting below error message
FAILED:SemanticException [Error 10294]:Attempt to do Update or delete unsingtransaction manager thatdoes not support these operations.
Older versions of Hive have a bug where
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; at CLI doesn't take effect.
You can check this by running "set hive.txn.manager" which will print the current value.
The safest way is set this in hive-site.xml.