I am looking to create a table from an existing table in Amazon Athena. The existing table is partitioned on partition_0, partition_1, and partition_2 (all strings) and I would like this partition to carry over. Here is my code:
CREATE TABLE IF NOT EXISTS newTable
AS
Select x, partition_0, partition_1, partition_2
FROM existingTable T
PARTITIONED BY (partition_0 string, partition_1 string, partition_2 string)
Trying to run this gives me an error at the FROM line, saying "mismatched input 'by'. expecting: '(', ',',".... Status code: 400; error code:invalidrequestexception
Not sure what syntax I am missing here.
This is the syntax for creating new tables:
CREATE TABLE new_table
WITH (
format = 'parquet',
external_location = 's3://example-bucket/output/',
partitioned_by = ARRAY['partition_0', 'partition_1', 'partition_2'])
AS SELECT * FROM existing_table
See the documentation for more examples: https://docs.aws.amazon.com/athena/latest/ug/ctas-examples.html#ctas-example-partitioned
Related
I want to copy the structure of full load temp table and add the addition table properties like partitioned by (partition_col), Format='ORC'
Temp table :
Create table if not exists tmp.temp_table( id int,
name string,
datestr string )
temp table got created.
Final table :
CREATE TABLE IF NOT EXISTS tmp.{final_table_name} (
LIKE tmp.temp_table
)
WITH (
FORMAT = 'ORC'
partitioned by('datestr')
)
But I am getting the error as "Error: Error while compiling statement: FAILED: ParseException line 1:63 missing EOF at 'WITH' near 'temp_table' (state=42000,code=40000)"
Any solution to achieve this functionality.
You should not use like and instead use create table as (CTAS) select * from mytab where 1=2.
CREATE TABLE IF NOT EXISTS tmp.{final_table_name}
As select * from tmp.temp_table where 1=2
WITH (
FORMAT = 'ORC'
partitioned by('datestr')
)
Like will create an empty table with exact same definition. CTAS will use same column sequence, data type/length, the sql, and your definition to create new empty table because we are using 1=2.
I have already an internal table in hive. Now I want to create an external table with partitions based on date to it. But it throws error, when I try to create it.
Sample code:
create external table db_1.T_DATA1 partitioned by (date string) as select * from db_2.temp
LOCATION 'file path';
Error:
ParseException line 2:0 cannot recognize input near 'LOCATION' ''file
path'' '' in table source
As per the answer provided at https://stackoverflow.com/a/26722320/4326922 you should be able to create external table with CTAS.
While trying to insert data into the following table, I get the following error message.
--create table mydataset.struct_1(x struct<course string,id int64>)
insert into `mydataset.struct_1` (course,id) values("B.A",12)
Error: Column course is not present in table mydataset.struct_1 at [2:35]
-- CREATE TABLE mydataset.struct_1(x STRUCT<course STRING,id INT64>)
INSERT INTO `mydataset.struct_1` (x) VALUES(STRUCT("B.A",12))
If you want to create STRUCT with a nested STRUCT named x with two fields y and z you should do this:
STRUCT<x STRUCT<y INT64, z INT64>>
So in your example:
create table mydataset.struct_1(STRUCT<x STRUCT<course string,id int64>>)
CREATE TABLE STRUCT_1 (x STRUCT<course: STRING,id: int>)
comment 'demonstrating how to work-around to insert complex
datatype unnested structs into a complex table '
Stored as parquet
location '/user/me/mestruct'
tblproperties ('created date: '=' 2019/01','done by: '='me');
Now lets do the insert.
insert into table STRUCT_1 select named_struct("course","B.A","id",12) from (select 't') s;
I am trying to load data from normal table to Hive partitioned table.
Here is my normal table syntax:
create table x(name string, date1 string);
Here is my new partitioned table syntax:
create table y (name string, date1 string) partitioned by (timestamp1 string);
Here is how I am how to load data to y:
insert into table y PARTITION(SUBSTR(date1,0,2)) select name, date1 from x;
Here is my Exception:
FAILED: ParseException line 1:39 missing ) at '(' near ',' in column name
line 1:51 cannot recognize input near '0' ',' '2' in column name
Use dynamic partitioning:
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
insert into table y PARTITION(timestamp1)
select name, date1, SUBSTR(date1,0,2) as timestamp1 from x;
I create a external table with a wrong(non-exists) path :
create external table IF NOT EXISTS ds_user_id_csv
(
type string,
imei string,
imsi string,
idfa string,
msisdn string,
mac string
)
PARTITIONED BY(prov string,day string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
stored as textfile
LOCATION 'hdfs://cdh0:8020/user/hive/warehouse/test.db/ds_user_id';
And I can not drop the table:
[cdh1:21000] > drop table ds_user_id_csv
> ;
Query: drop table ds_user_id_csv
ERROR:
ImpalaRuntimeException: Error making 'dropTable' RPC to Hive Metastore:
CAUSED BY: MetaException: java.lang.IllegalArgumentException: Wrong FS: hdfs://cdh0:8020/user/hive/warehouse/test.db/ds_user_id, expected: hdfs://nameservice1
So how to solve this? Thank you.
Use the following command to change the location
ALTER TABLE name ds_user_id_csv SET LOCATION '{new location}';