I have already an internal table in hive. Now I want to create an external table with partitions based on date to it. But it throws error, when I try to create it.
Sample code:
create external table db_1.T_DATA1 partitioned by (date string) as select * from db_2.temp
LOCATION 'file path';
Error:
ParseException line 2:0 cannot recognize input near 'LOCATION' ''file
path'' '' in table source
As per the answer provided at https://stackoverflow.com/a/26722320/4326922 you should be able to create external table with CTAS.
Related
I want to copy the structure of full load temp table and add the addition table properties like partitioned by (partition_col), Format='ORC'
Temp table :
Create table if not exists tmp.temp_table( id int,
name string,
datestr string )
temp table got created.
Final table :
CREATE TABLE IF NOT EXISTS tmp.{final_table_name} (
LIKE tmp.temp_table
)
WITH (
FORMAT = 'ORC'
partitioned by('datestr')
)
But I am getting the error as "Error: Error while compiling statement: FAILED: ParseException line 1:63 missing EOF at 'WITH' near 'temp_table' (state=42000,code=40000)"
Any solution to achieve this functionality.
You should not use like and instead use create table as (CTAS) select * from mytab where 1=2.
CREATE TABLE IF NOT EXISTS tmp.{final_table_name}
As select * from tmp.temp_table where 1=2
WITH (
FORMAT = 'ORC'
partitioned by('datestr')
)
Like will create an empty table with exact same definition. CTAS will use same column sequence, data type/length, the sql, and your definition to create new empty table because we are using 1=2.
I'm trying to create a delta lake table in hive 3.
I moved the delta-hive-assembly_2.11-0.3.0.jar to the hive aux directory and set into the hive cli
SET hive.input.format=io.delta.hive.HiveInputFormat;
SET hive.tez.input.format=io.delta.hive.HiveInputFormat;
but when I try to create the table it throws the following error:
[2b8497c1-b4d3-492e-80a5-ec4db4119018 HiveServer2-Handler-Pool: Thread-133]: Exception occured while getting the URI from storage handler: Expected authority at index 22: deltastoragehandler://: Unsupported ex
java.net.URISyntaxException: Expected authority at index 22: deltastoragehandler://: Unsupported ex
at org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler.getURIForAuth(DefaultStorageHandler.java:76) ~[hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
at org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:210) [hive-exec-3.1.3000.7.1.7.0-551.jar:3.1.3000.7.1.7.0-551]
The create statement:
CREATE EXTERNAL TABLE default.test_delta
( id INT, activ INT)
STORED BY 'io.delta.hive.DeltaStorageHandler'
LOCATION '/dev/delta/tessttt';
It's anyone that know why this error occurs ?
You need to add in the properties of the table the following: TBLPROPERTIES('DO_NOT_UPDATE_STATS'='true');
Example
CREATE EXTERNAL TABLE default.test_delta
( id INT, activ INT)
STORED BY 'io.delta.hive.DeltaStorageHandler'
LOCATION '/dev/delta/tessttt'
TBLPROPERTIES('DO_NOT_UPDATE_STATS'='true';
The solution also works for iceberg format.
Inspired by: https://github.com/delta-io/connectors/issues/279
For example, when using Parquet format, I'd like to be able to specify the compression scheme (("parquet.compression"="SNAPPY")). Running this query:
CREATE TABLE table_a_copy
STORED AS PARQUET
TBLPROPERTIES("parquet.compression"="SNAPPY")
AS
SELECT * FROM table_a
returns an error:
Error: Error while compiling statement: FAILED: ParseException line 1:69 cannot recognize input near 'parquet' '.' 'compression' in table properties list (state=42000,code=40000)
The same query without the TBLPROPERTIES works just fine.
This is similar to this question: Create hive table using "as select" or "like" and also specify delimiter. But I can't figure out how to make TBLPROPERTIES work with that approach. I'm using Hive 1.1.
I was able to run same exact statement in Hive 2.1.1 version.
Try with this workaround:
CREATE TABLE table_a_copy like table_a STORED AS PARQUET;
alter table set TBLPROPERTIES("parquet.compression"="SNAPPY");
insert into table table_a_copy select * from table_a ;
I would like to create externel table using like option.
CREATE EXTERNAL TABLE IF NOT EXISTS test1 (rec string)
LIKE 'EPCTR_201804'
LOCATION '/hdfs/t1/tt1/PR/34/1xx/E1ERPSE/201801/PR/20180202-000758/*';
But this error was shown saying :
FAILED: ParseException line 2:0 missing EOF at 'LIKE' near ')'
How can I resolve it please ?
You don't have to give schema for new table. When you use CREATE TABLE LIKE, new table keeps the same schema as old one.
Use following:
CREATE EXTERNAL TABLE IF NOT EXISTS test1
LIKE 'EPCTR_201804'
LOCATION '/hdfs/t1/tt1/PR/34/1xx/E1ERPSE/201801/PR/20180202-000758/*';
Create Table Like should be without columns specification, because LIKE means create table with the same exactly schema like the other table.
Also table location is a folder where the data files are being stored, there should be no /* at the end, like this:
CREATE EXTERNAL TABLE IF NOT EXISTS test1 LIKE 'EPCTR_201804'
LOCATION '/hdfs/t1/tt1/PR/34/1xx/E1ERPSE/201801/PR/20180202-000758';
I create a external table with a wrong(non-exists) path :
create external table IF NOT EXISTS ds_user_id_csv
(
type string,
imei string,
imsi string,
idfa string,
msisdn string,
mac string
)
PARTITIONED BY(prov string,day string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
stored as textfile
LOCATION 'hdfs://cdh0:8020/user/hive/warehouse/test.db/ds_user_id';
And I can not drop the table:
[cdh1:21000] > drop table ds_user_id_csv
> ;
Query: drop table ds_user_id_csv
ERROR:
ImpalaRuntimeException: Error making 'dropTable' RPC to Hive Metastore:
CAUSED BY: MetaException: java.lang.IllegalArgumentException: Wrong FS: hdfs://cdh0:8020/user/hive/warehouse/test.db/ds_user_id, expected: hdfs://nameservice1
So how to solve this? Thank you.
Use the following command to change the location
ALTER TABLE name ds_user_id_csv SET LOCATION '{new location}';