In GCP BigQuery, Can we add a column to an existing table and change his order in the table ?
for the moment I found only one way which is to delete the table modify the JSON of the table and recreate it with the new column in the right place
Do you have a solution please?
Add column using SQL:
ALTER TABLE mydataset.mytable
ADD COLUMN new_column STRING;
Source: https://cloud.google.com/bigquery/docs/managing-table-schemas#sql
Change order of columns:
create or replace table <SCHEMA.NEW_TABLE_NAME> as
select col1,col2,col3 from <SCHEMA.OLD_TABLE_NAME>;
Source: Changing order of columns for a table
Related
I am writing the following query to add a column at a specified position but getting the below error:
alter table quantum_raw_dev.rpt_backup_allocation
change upt_type upt_type STRING after tray_size;
You can add one or more columns to the end of the column list using:
ALTER TABLE <table_name> ADD COLUMNS (col_name col_type, ...);
[note: there is NO comma between column name and type]
Adding or Removing Columns
You can add one or more columns to the end of the column list using ADD COLUMNS,
or (with Impala only) you can delete columns using DROP COLUMN.
The general syntax is
ALTER TABLE tablename ADD COLUMNS (col1 TYPE1,col2 TYPE2,… );
ALTER TABLE tablename DROP COLUMN colname;
For example, you can add a bonus integer column to the employees table:
ALTER TABLE employees ADD COLUMNS (bonus INT);
Or you can drop the office_id column from the employees table:
ALTER TABLE employees DROP COLUMN office_id;
Notes
DROP COLUMN is not available in Hive, only in Impala. However, see “Replacing All Columns” below.
You can only drop one column at a time.
To drop multiple columns, use multiple statements or use the method to replace columns (see below).
You cannot add a column in the middle of the list rather than at the end.
You can, however, add the column then change the order (see above) or use the method to replace columns (see below).
As with changing the column order, these do not change the data files.
If the table definition agrees with the data files before you drop any column other than the last one,
you will need to recreate the data files without the dropped column's values.
If you drop the last column, the data will still exist but it will be ignored when a query is issued.
If you add columns for which no data exists, those columns will be NULL in each row.
Replacing All Columns
You can also completely replace all the columns with a new column list.
This is helpful for dropping multiple columns,
<h1>or if you need to add columns in the middle of the list<h1>
<h2>(like your use case)<h2>
The general syntax is
ALTER TABLE tablename REPLACE COLUMNS (col1 TYPE1,col2 TYPE2,… );
This completely removes the existing list of columns and replaces it with the new list.
Only the columns you specify in the ALTER TABLE statement will exist, and they will be in the order you provide.
Note
Again, this does not change the data files, only the metadata for the table,
so you'll either want the new list to match the data files or need to recreate the data files to match the new list.
I do not think you can add columns in between columns in Impala like above.
You can backup the data, drop the and recreate with new structure, and load the table from backup. Also if you have HIVE in your system you can try to do below steps -
Add column first and then use below commands to move columns around.
ALTER TABLE tab ADD COLUMNS (id BIGINT);
This moves id column to the beginning.
ALTER TABLE tab CHANGE COLUMN id id BIGINT first;
This moves existing_col after id.
ALTER TABLE tab CHANGE COLUMN existing_col existing_col string AFTER id;
Please refresh/invalidate metadata after applying all DDLs.
You cannot add column in between. Best way is to archive the data in another table. Drop the impala old table and create a fresh table with new columns as per the desired location and then reinsert the data.
Actually, I want to move one table to another database.
But spark don't permit this.
Then, how to copy table by spark-sql?
I already tried this.
SELECT *
INTO table1 IN new_database
FROM old_database.table1
But it was not working.
maybe try:
CREATE TABLE new_db.new_table AS
SELECT *
FROM old_db.old_table;
To preserve partitioning and storage format do the following-
Get the complete schema of the existing table by running-
show create table db.old_table
The above query will output the table schema which you can just execute after changing the path name and table name.
Then insert all the rows into the new blank table using-
insert into db.new_table select * from db.old_table
The following snippet will create a new table while preserving the definition of the "old" table.
CREATE TABLE db.new_table LIKE db.old_table;
For more info, check the doc's CREATE TABLE.
How can I change DATA TYPE of a column from number to varchar2 without deleting the table data?
You can't.
You can, however, create a new column with the new data type, migrate the data, drop the old column, and rename the new column. Something like
ALTER TABLE table_name
ADD( new_column_name varchar2(10) );
UPDATE table_name
SET new_column_name = to_char(old_column_name, <<some format>>);
ALTER TABLE table_name
DROP COLUMN old_column_name;
ALTER TABLE table_name
RENAME COLUMN new_column_name TO old_coulumn_name;
If you have code that depends on the position of the column in the table (which you really shouldn't have), you could rename the table and create a view on the table with the original name of the table that exposes the columns in the order your code expects until you can fix that buggy code.
You have to first deal with the existing rows before you modify the column DATA TYPE.
You could do the following steps:
Add the new column with a new name.
Update the new column from old column.
Drop the old column.
Rename the new column with the old column name.
For example,
alter table t add (col_new varchar2(50));
update t set col_new = to_char(col_old);
alter table t drop column col_old cascade constraints;
alter table t rename column col_new to col_old;
Make sure you re-create any required indexes which you had.
You could also try the CTAS approach, i.e. create table as select. But, the above is safe and preferrable.
The most efficient way is probably to do a CREATE TABLE ... AS SELECT
(CTAS)
alter table table_name modify (column_name VARCHAR2(255));
Since we can't change data type of a column with values, the approach that I was followed as below,
Say the column name you want to change type is 'A' and this can be achieved with SQL developer.
First sort table data by other column (ex: datetime).
Next copy the values of column 'A' and paste to excel file.
Delete values of the column 'A' an commit.
Change the data type and commit.
Again sort table data by previously used column (ex: datetime).
Then paste copied data from excel and commit.
I want to update an empty table , which has a column of type timestamp to varbinary(8)
I used the following command
ALTER TABLE Notification ALTER COLUMN RowRevisionID varbinary(8)
and I get and an error
Cannot alter column 'RowRevisionID' because it is 'timestamp'.
How can I change a timestamp column type?
I do not wish to drop the column an add a new one , because that will create a column at the end , and I wish to preserve column order to use this table in an INSERT INTO
You unfortunately cannot make a change to a timestamp column, as the error implies; you are stuck with what you have. Also, each table can only have one timestamp column, so you cannot duplicate the column in any solution.
Your best bet (depending on the size of the table) might be to copy the data into a staging table (using SELECT * INTO MyTempTable FROM OriginalTable syntax to preserve the timestamp values), then drop and recreate the original table with the desired columns in the desired order and reinsert the data, or you could add a new VARBINARY(8) column to the existing table and drop the timestamp column, preserving the original table. There are probably other solutions along the same lines as these, but all will require a new column, rather than an ALTER COLUMN script.
You are looking for:
ALTER TABLE Notification DROP RowRevisionID;
and
ALTER TABLE Notification ADD RowRevisionID varbinary(8) AFTER myOtherColumn;
Is there an SQL command on the AS400/iSeries/System-i/whatever to add a column to a table in a specific ordinal position, or moving an existing column to a different position?
IBM i 7.1 now allows you to add a column in front of another.
ALTER TABLE table ADD COLUMN colname ... BEFORE othercolumn
No. The ALTER TABLE statement will allow you add a column to a table, but, according to the documentation:
The new column is the last column of the table; that is, if initially there are n columns, the added column is column n+1.
If you'd like to change the order of columns in your table, your best bet is to:
Use the RENAME statement to rename the table.
Recreate the table, with its original name, with the columns in the order that you want.
Use an INSERT SELECT to populate the new table with the data from the renamed table.
When you're sure the data is intact, you can drop the renamed version of the table.