I am creating a temp table from another table using AS clause where I am including the partition column of another table also be part of temp table and then I am getting the below error. Below is the table create statement where col4 is the partition column of table xyz.
And while running the create statement i am getting the below error. And when I am removing the col4 from the create statement its running fine.
Error:
Error while compiling statement: FAILED: NumberFormatException For
input string: "HIVE_DEFAULT_PARTITION" (state=42000,code=40000)
Please help.
Example:
CREATE TEMPORARY TABLE abc STORED AS PARQUET AS SELECT
col1 AS col1,
col2 AS col2,
col3 AS col3,
col4 AS col4
FROM xyz;
This is a problem with source table xyz because it contains partition __HIVE_DEFAULT_PARTITION__
Hive creates a partition with value __HIVE_DEFAULT_PARTITION__ when in dynamic partition mode inserted partition value is NULL.
Partition __HIVE_DEFAULT_PARTITION__ is not compatible with numeric type and this causing error because it cannot be cast to numeric type.
To remove or query this partition, you need to change the column type to string first:
ALTER TABLE xyz PARTITION COLUMN (col4 string);
Of course you may want to backup table and check the data before removing and decide what to do with this data.
To remove partition:
ALTER TABLE xyz DROP PARTITION (col4 = '__HIVE_DEFAULT_PARTITION__');
After removing partition you can change the type of partition column back to numeric type.
Related
I have a table with approximately 90 columns and want to delete all after the 5th. How to delete all columns except first five?
Given that you only want to keep a few columns, the option with least code would be to make a new table with those columns only. You should do this in a transaction to avoid losing data.
Example with two columns:
ALTER TABLE TableName RENAME TO TmpTableName;
CREATE TABLE TableName(Col1 INTEGER, Col2 INTEGER);
INSERT INTO TableName(Col1, Col2) SELECT Col1, Col2 FROM TmpTableName;
DROP TABLE TmpTableName;
Before version 3.35.0, SQLite did not support removing columns, so this was the only possible option.
SQL Server supports syntax that allows to remove more than one column at a time. How to DROP multiple columns with a single ALTER TABLE statement in SQL Server?
Though it seems to be impossible to force it to work with IF EXISTS clasue for each occurence.
ALTER TABLE
DROP
| COLUMN [ IF EXISTS ]
{
column_name
} [ ,...n ]
Conditionally drops the column or constraint only if it already exists.
Example:
CREATE TABLE t(i INT, col1 INT, col2 INT);
ALTER TABLE t DROP COLUMN IF EXISTS col1, col2;
-- col1, col2 were successfully removed
ALTER TABLE t DROP COLUMN IF EXISTS col1, col2;
-- Msg 4924 Level 16 State 1 Line 1
-- ALTER TABLE DROP COLUMN failed because column 'col2' does not exist in table 't'.
Based on error message IF EXISTS takes effect only for first column.
db<>fiddle demo
Is there a limitation of this clause when combined with multiple columns at once?
The syntax is a bit cumbersome but
ALTER TABLE t DROP COLUMN IF EXISTS col1,
COLUMN IF EXISTS col2;
works fine
Sorry for all the setup. This is a hive datatype and comment question.
I have a single file in HDFS which combines 4 sets of table data. Breaking the data out ahead of time is not my preferred option. The first 4 rows specify the column headers:
*1 col1, col2, col3
*2 cola, colb, colc, cold, col5e
etc....
data rows begin with matching number at position 1 of the header.
1 data, data, data,
2 data, data, data, data, data,
etc...
The base hive table is just col0 - col60 for the raw file. I've tried creating a CTAS table to hold all of the "1" columns and one for the "2" columns where I can specify data type, and comments. Since the column names vary, I cannot give the columns names on the base table nor can I comment them with column based metadata.
This DDL didn't work but giving an example of what I'm hoping to do. Any thoughts ?
CREATE TABLE foo (
col1 as meaningful_name string comment 'meaningful comment')
as
SELECT col1
FROM base_hive table
WHERE col1 = 1;
CREATE TABLE foo
as
SELECT col1 string comment 'meaningful comment'
FROM base_hive table
WHERE col1 = 1;
thanks TD
I dont understand much what you are trying to achieve here, but looking at your DDL, I can see some errors. For the correct CREATE TABLE AS SELECT implementation, pl use the below DDL:
CREATE TABLE foo (
col1 STRING COMMENT 'meaningful comment')
AS
SELECT col1 AS meaningful_name
FROM base_hive table
WHERE col1 = 1;
i have one table test it has 10 column with 20 rows.
I need to move this data to archive_test table which has 11 column (10 same as test table plus one column is archive date).
when i tried to insert like below its shows error because number of column mismatch.
insert into archive_test
select * from test;
Please suggest the better way to do this.Thanks!
Well, obviously you need to supply values for all the columns, and although you can avoid doing so you should also explicitly state whic value is going to be inserted into which column. If you have an extra column in the target table you either:
Do not mention it
Specify a default value as part of its column definition in the table
Have a trigger to populate it
Specify a value for that column.
eg.
insert into table archive_test (col1, col2, col3 ... col11)
select col1,
col2,
col3,
...
sysdate
from test;
assuming that archive_date is the last column:
INSERT INTO archive_test
SELECT test.*, sysdate
FROM test
I have the following table in postgres:
create table1 (col1 character varying, col2 character varying);
My table has the following data:
col1 col2
Questions Tags Users
Value1 Value2 Val
I want find the length of col1 and col2 and when the length of values of column 1 and column2 exceeds 6, I want to truncate it and discard the remaining values. i.e. I want my final table to look like the following:
col1 col2
Questi Tags U
Value1 Value2
Actually the reason why I want to do this is, when I create index on table1 then I am getting the following error:
ERROR: index row size 2744 exceeds maximum 2712 for index "allstrings_string_key"
HINT: Values larger than 1/3 of a buffer page cannot be indexed.
Consider a function index of an MD5 hash of the value, or use full text indexing.
I know I can do this by importing the values to some programming language and then truncating the value. Is there some way by which I may achieve the same using an sql query in postgres.
Couldn't you just update them to contain only strings of length 6 at max?
I am no postrgres pro, so this is probably not the best method, but should do the job anyways:
UPDATE table1 SET col1 = SUBSTRING(col1, 1, 6) WHERE LEN(col1) > 6
UPDATE table1 SET col2 = SUBSTRING(col2, 1, 6) WHERE LEN(col2) > 6
I'd suggest that you actually follow the advice from Postgres, rather than changing your data. Clearly, that column with a 2k character long string shouldn't be indexed -- or not with a btree index anyway.
If the idea behind the index is searching, use full text search instead:
http://www.postgresql.org/docs/current/static/textsearch.html
If the idea behind the need is for sorting, use a functional index instead. For instance:
create index tbl_sort on (substring(col from 1 for 20));
Then, instead of ordering by col, order by substring(col from 1 for 20).
Have you tried changing the type of the column to CHAR instead of VARCHAR?
ALTER TABLE table1
ALTER COLUMN col1 SET DATA TYPE CHAR(6),
ALTER COLUMN col2 SET DATA TYPE CHAR(6)
If you need the column to be variable length, you can specify a limit (note that this is a PostgreSQL extension):
ALTER TABLE table1
ALTER COLUMN col1 SET DATA TYPE CHARACTER VARYING(6),
ALTER COLUMN col2 SET DATA TYPE CHARACTER VARYING(6)