I have the following table:
CREATE EXTERNAL TABLE aggregate_status(
m_point VARCHAR(50),
territory VARCHAR(50),
reading_meter VARCHAR(50),
meter_type VARCHAR(500)
)
PARTITIONED BY(
insert_date VARCHAR(10))
STORED AS PARQUET
LOCATION '<the s3 route>/aggregate_status'
TBLPROPERTIES(
'parquet.compression'='SNAPPY'
)
I wish to change the reading_meter column to reading_mode, without losing data.
ALTER TABLE works, but the field now shows null.
I'm not the owner of the Hadoop enviroment I'm working on so changing properties such as set parquet.column.index.access = true is discarded.
Any help would be appreciated. Thanks.
Managed to find a solution, at least for short amounts of data.
Create a backup of the table, with the column name already changed.
CREATE TABLE aggregate_status_bkp AS
SELECT
m_point,
territory,
reading_meter AS reading_mode,
meter_type,
insert_date
FROM aggregate_status
Perform the ALTER TABLE
ALTER TABLE aggregate_status CHANGE COLUMN reading_meter reading_mode VARCHAR (50)
INSERT OVERWRITE from the backup to the original.
--You might need to temporarily disable strict partition mode depending on your case, this is safe since it's only a lock.
--set hive.exec.dynamic.partition.mode=nonstrict;
INSERT OVERWRITE TABLE aggregate_status PARTITION(insert_date)
SELECT
m_point,
territory,
reading_mode,
meter_type,
insert_date
FROM aggregate_status_bkp;
--set hive.exec.dynamic.partition.mode=strict;
Another situation we want to protect against dynamic partition insert is that the user may accidentally specify all partitions to be dynamic partitions without specifying one static partition, while the original intention is to just overwrite the sub-partitions of one root partition. We define another parameter hive.exec.dynamic.partition.mode=strict to prevent the all-dynamic partition case.
See https://cwiki.apache.org/confluence/display/Hive/Tutorial#Tutorial-QueryingandInsertingData
Optional Delete the backup table after you're finished.
DROP TABLE aggregate_status_bkp;
I have an given financial application that has very large tables, which are—more or less—of the following form:
CREATE TABLE ProjectAccounts (
RecordId BIGINT PRIMARY KEY,
AccountId GUID NOT NULL,
Version BIGINT,
-- some data
)
Because it is a business requirement that old versions of the records need to be preserved, changes in this table are done by creating a new version of the record and increment the field Version.
What I want to do is tho partition the table in such a way, that only the newest version of an record (highest version number) stays in the main partition, while obsolete records should be moved into a shadow-partition.
What would the best approach be to do so? Is there a better approach?
Alternative ideas from my colleagues:
the record with the newest version should be a magic value (ie. -1, 0, or 1)
move old records into a archive-table (ie. using a trigger)
However, I am worried about the possible performance impact of triggers (almost always bad), updates (searching and locking), and the added complexity created by searching multiple tables.
As you want to create two partitions , You can add one extra column "IsLatestVersion" use int datatype for this column.
Set "IsLatestVersion" = 0 , for most recent record and insert new record with "IsLatestVersion" = 1 , You might need to modify your code file to maintain this or you can create small trigger to do this task.
Now Table structure is ready to use. Lets create partition on this table. To do this you need to follow following steps :
1. Add two file groups to current database. You can add filegroup by following query :
ALTER DATABASE CurrentDB ADD FILEGROUP [Filegroup_2001]
GO
ALTER DATABASE CurrentDB ADD FILEGROUP [Filegroup_2002]
GO
Attach files with created file groups. You can use following query to do that:
ALTER DATABASE CurrentDB ADD FILE(NAME = N'data_2001',
FILENAME = N'C:\data_2001.ndf',
SIZE = 5000MB,
MAXSIZE = 10000MB,
FILEGROWTH = 500MB)
TO FILEGROUP [Filegroup_2001]
GO
Create partition function :
USE [YourTableName]
GO
CREATE PARTITION FUNCTION [Newpf](int) AS RANGE RIGHT FOR VALUES (0, 1)
GO
Create partition scheme
USE [YourTableName]
GO
CREATE PARTITION SCHEME [PFScheme] AS PARTITION [Newpf] TO ([PRIMARY], [Secondary], [Secondary])
GO
Now attach partition function and scheme to existing table. Following images will describe how to do that.
You are done.
Lets check number of records in each partition by following query :
SELECT $PARTITION.Newpf(IsLatestVersion) AS PARTITIONID,
COUNT(* ) AS ROW_COUNT
FROM DBO.ProjectAccounts
GROUP BY $PARTITION.Newpf(IsLatestVersion)
ORDER BY PARTITIONID
I got http://prntscr.com/5wuvu8
I'm currently working on a project that uses a Redshift table with 51 columns. However, the person who made the table forgot to add a sortkey to our time column which will hurt performance for our use case if we don't add it.
How can I make a version of the table with our time column as the sortkey? I'm aware that you can't make a column a sortkey if its a member of an existing table, but I was hoping there's a way to do it that doesn't involve writing out the CREATE TABLE syntax by hand; for example, something like this would be nice:
timecube=# CREATE TABLE foo (like bar) sortkey(time);
ERROR: CREATE TABLE LIKE is not supported with DISTSTYLE, DISTKEY(), or SORTKEY() clauses
but as you can see its not supported. Is there another way? As we're still developing we don't need any of existing data.
Using traditional tools like pgdump didn't work well because they don't include any of the Redshift extras like encoding.
Redshift supports specifying the DIST and SORT keys as part of CREATE TABLE AS statements, as per the docs.
CREATE TABLE table_name
DISTSTYLE KEY
DISTKEY ( column )
SORTKEY ( column )
AS
(SELECT *
FROM source_table)
;
First step you need to do use get create table statement for existing table. Then create new table this time add sort key to new table.
Check encoding for old table ( when you load data using copy command it automatically adds compression encodings)
select "column", type, encoding
from pg_table_def where tablename = 'old_table'
When creating new table add encoding type for each column. Create table with Sort key .
Once new table is created use below command
insert into new table ( select * from old table order by time asc)
I'm trying to setup temporary tables for unit-testing purposes. So far I managed to create a temporary table which copies the structure of an existing table:
CREATE TEMP TABLE t_mytable (LIKE mytable INCLUDING DEFAULTS);
But this lacks the data from the original table. I can copy the data into the temporary table by using a CREATE TABLE AS statement instead:
CREATE TEMP TABLE t_mytable AS SELECT * FROM mytable;
But then the structure of t_mytable will not be identical, e.g. column sizes and default values are different. Is there a single statement which copies everything?
Another problem with the first query using LIKE is that the key column still references the SEQUENCE of the original table, and thus increments it on insertion. Is there an easy way to create the new table with its own sequence, or will I have to set up a new sequence by hand?
I'm using the following code to do it:
CREATE TABLE t_mytable (LIKE mytable INCLUDING ALL);
ALTER TABLE t_mytable ALTER id DROP DEFAULT;
CREATE SEQUENCE t_mytable_id_seq;
INSERT INTO t_mytable SELECT * FROM mytable;
SELECT setval('t_mytable_id_seq', (SELECT max(id) FROM t_mytable), true);
ALTER TABLE t_mytable ALTER id SET DEFAULT nextval('t_my_table_id_seq');
ALTER SEQUENCE t_mytable_id_seq OWNED BY t_mytable.id;
Postgres 10 or later
Postgres 10 introduced IDENTITY columns conforming to the SQL standard (with minor extensions). The ID column of your table would look something like:
id integer PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY
Syntax in the manual.
Using this instead of a traditional serial column avoids your problem with sequences. IDENTITY columns use exclusive, dedicated sequences automatically, even when the specification is copied with LIKE. The manual:
Any identity specifications of copied column definitions will only be
copied if INCLUDING IDENTITY is specified. A new sequence is created
for each identity column of the new table, separate from the sequences
associated with the old table.
And:
INCLUDING ALL is an abbreviated form of INCLUDING DEFAULTS INCLUDING IDENTITY INCLUDING CONSTRAINTS INCLUDING INDEXES INCLUDING STORAGE INCLUDING COMMENTS.
The solution is simpler now:
CREATE TEMP TABLE t_mytable (LIKE mytable INCLUDING ALL);
INSERT INTO t_mytable TABLE mytable;
SELECT setval(pg_get_serial_sequence('t_mytable', 'id'), max(id)) FROM tbl;
As demonstrated, you can still use setval() to set the sequence's current value. A single SELECT does the trick. pg_get_serial_sequence()]6 gets the name of the sequence.
db<>fiddle here
Related:
How to reset postgres' primary key sequence when it falls out of sync?
Is there a shortcut for SELECT * FROM?
Creating a PostgreSQL sequence to a field (which is not the ID of the record)
Original (old) answer
You can take the create script from a database dump or a GUI like pgAdmin (which reverse-engineers database object creation scripts), create an identical copy (with separate sequence for the serial column), and then run:
INSERT INTO new_tbl
SELECT * FROM old_tbl;
The copy cannot be 100% identical if both tables reside in the same schema. Obviously, the table name has to be different. Index names would conflict, too. Retrieving serial numbers from the same sequence would probably not be in your best interest, either. So you have to (at least) adjust the names.
Placing the copy in a different schema avoids all of these conflicts. While you create a temporary table from a regular table like you demonstrated, that's automatically the case since temp tables reside in their own temporary schema.
Or look at Francisco's answer for DDL code to copy directly.
How to change column order in a table using SQL query in SQL Server 2005?
I want to rearrange column order in a table using SQL query.
You cannot. The column order is just a "cosmetic" thing we humans care about - to SQL Server, it's almost always absolutely irrelevant.
What SQL Server Management Studio does in the background when you change column order there is recreating the table from scratch with a new CREATE TABLE command, copying over the data from the old table, and then dropping it.
There is no SQL command to define the column ordering.
You have to explicitly list the fields in the order you want them to be returned instead of using * for the 'default' order.
original query:
select * from foobar
returns
foo bar
--- ---
1 2
now write
select bar, foo from foobar
bar foo
--- ---
2 1
according to https://learn.microsoft.com/en-us/sql/relational-databases/tables/change-column-order-in-a-table
This task is not supported using Transact-SQL statements.
Well, it can be done, using create/ copy / drop/ rename, as answered by komma8.komma1
Or you can use SQL Server Management Studio
In Object Explorer, right-click the table with columns you want to reorder and click Design (Modify in ver. 2005 SP1 or earlier)
Select the box to the left of the column name that you want to reorder. (You can select multiple columns by holding the [shift] or
the [ctrl] keys on your keyboard.)
Drag the column(s) to another location within the table.
Then click save. This method actually drops and recreates the table, so some errors might occur.
If Change Tracking option is enabled for the database and the table, you shouldn't use this method.
If it is disabled, the Prevent saving changes that require the table re-creation option should be cleared in Tools menu > Options > Designers, otherwise "Saving changes is not permitted" error will occur.
Disabling the Prevent saving changes that require the table re-creation option is strongly advised against by Microsoft, as it leads to the existing change tracking information being deleted when the table is re-created, so you should never disable this option if Change Tracking is enabled!
Problems may also arise during primary and foreign key creation.
If any of the above errors occurs, saving fails which leaves you with the original column order.
In SQLServer Management Studio:
Tools -> Options -> Designers -> Table and Database Designers
Unselect 'Prevent saving changes that require table re-creation'.
Then:
right click the table you want to re-order the columns for.
click 'Design'.
Drag the columns to the order you want.
finally, click save.
SQLServer Management studio will drop the table and recreate it using the data.
This is similar to the question on ordering the records in the result of a query .. and typically no one likes the formally correct answer ;-)
So here it goes:
as per SQL standard, the columns in a table are not "ordered"
as a result, a select * does not force the columns to be returned in a particular order
typically, each RDBMS has a kind of "default" order (usually the order that the columns were added to the table, either in the create table' or in thealter table add ` statements
therefore, if you rely on the order of columns (because you are using the results of a query to poulate some other datastructure from the position of the columns), explicitly list the columns in the order you want them.
You can of course change the order of the columns in a sql statement. However if you want to abstract tables' physical column order, you can create a view. i.e
CREATE TABLE myTable(
a int NULL,
b varchar(50) NULL,
c datetime NULL
);
CREATE VIEW vw_myTable
AS
SELECT c, a, b
FROM myTable;
select * from myTable;
a b c
- - -
select * from vw_myTable
c a b
- - -
You can do it by creating a new table, copy all the data over, drop the old table, then renaming the new one to replace the old one.
You could also add new columns to the table, copy the column by column data over, drop the old columns, then rename new columns to match the old ones. A simple example below:
http://sqlfiddle.com/#!3/67af4/1
CREATE TABLE TestTable (
Column1 INT,
Column2 VARCHAR(255)
);
GO
insert into TestTable values(1, 'Test1');
insert into TestTable values(2, 'Test2');
GO
select * from TestTable;
GO
ALTER TABLE TestTable ADD Column2_NEW VARCHAR(255);
ALTER TABLE TestTable ADD Column1_NEW INT;
GO
update TestTable
set Column1_NEW = Column1,
Column2_NEW = Column2;
GO
ALTER TABLE TestTable DROP COLUMN Column1;
ALTER TABLE TestTable DROP COLUMN Column2;
GO
sp_rename 'TestTable.Column1_NEW', 'Column1', 'COLUMN';
GO
sp_rename 'TestTable.Column2_NEW', 'Column2', 'COLUMN';
GO
select * from TestTable;
GO
In SQLServer Management Studio:
Tools -> Options -> Designers -> Table and Database Designers
Unselect Prevent saving changes that require table re-creation.
Now you can reorder the table.
Sql server internally build the script. It create a temporary table with new changes and copy the data and drop current table then recreate the table insert from temp table. I find it from "Generate Change script" option ssms 2014. Script like this. From Here: How to change column order in a table using sql query
BEGIN TRANSACTION
SET QUOTED_IDENTIFIER ON
SET ARITHABORT ON
SET NUMERIC_ROUNDABORT OFF
SET CONCAT_NULL_YIELDS_NULL ON
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
COMMIT
BEGIN TRANSACTION
GO
CREATE TABLE dbo.Tmp_emps
(
id int NULL,
ename varchar(20) NULL
) ON [PRIMARY]
GO
ALTER TABLE dbo.Tmp_emps SET (LOCK_ESCALATION = TABLE)
GO
IF EXISTS(SELECT * FROM dbo.emps)
EXEC('INSERT INTO dbo.Tmp_emps (id, ename)
SELECT id, ename FROM dbo.emps WITH (HOLDLOCK TABLOCKX)')
GO
DROP TABLE dbo.emps
GO
EXECUTE sp_rename N'dbo.Tmp_emps', N'emps', 'OBJECT'
GO
COMMIT
If your table has enough columns then you can try this. First create a new table with preferred order of columns.
create table new as select column1,column2,column3,....columnN from table_name;
Now drop the table using drop command
drop table table_name;
now rename the newly created table to your old table name.
rename new to table_name;
now select the table, you have your columns rearranged as you preferred before.
select * from table_name;
Not sure if still relevant, but SSMS can generate a change scripts for this.
Re-order (drag the column) the table in Designer View
Click on 'Generate Change Script'
The generated script contains the script which does the following:
Create a temporary table
Adds the constraints, relationships and triggers from original table to temporary table
Drop original table
Rename temporary table to original table name
If you have not yet added any data into your table yet, there is one way to move the columns around.
Try this:
In SSMS, click Tools > Options > Designers > Table and Database Designers > Uncheck the box next to Prevent saving changes that require table re-creation > Click OK.
In the object tree, right-click on your table and select Design > in the thin column to the left of the Column Name column, you can click and drag the columns around to wherever you want them. When you're done, just go to close the Design tab and SSMS will ask you if you want to save your changes, click OK.
Optional:
3. Re-enable the checkbox for the option from Step 1 to re-secure your table.
Hope this helps someone!
Credit goes to Microsoft:
https://learn.microsoft.com/en-us/troubleshoot/sql/ssms/error-when-you-save-table#more-information
At the end of the day, you simply cannot do this in MS SQL. I recently created tables on the go (application startup) using a stored Procedure that reads from a lookup table. When I created a view that combined these with another table I had manually created earlier one (same schema, with data), It failed - simply because I was using ''Select * UNION Select * ' for the view. At the same time, if I use only those created through the stored procedure, I am successful.
In conclusion: If there is any application which depends on the order of column it is really not good programming and will for sure create problems in the future. Columns should 'feel' free to be anywhere and be used for any data process (INSERT, UPDATE, SELECT).
You can achieve it with these steps:
remove all foreign keys and primary key of the original table.
rename the original table.
using CTAS create the original table in the order you want.
drop the old table.
apply all constraints back to the original table
If the columns to be reordered have recently been created and are empty, then the columns can be deleted and re-added in the correct order.
This happened to me, extending a database manually to add new functionality, and I had missed a column out, and when I added it, the sequence was incorrect.
After finding no adequate solution here I simply corrected the table using the following kind of commands.
ALTER TABLE tablename DROP COLUMN columnname;
ALTER TABLE tablename ADD columnname columntype;
Note: only do this if you don't have data in the columns you are dropping.
People have said that column order does not matter. I regularly use SQL Server Management Studio "generate scripts" to create a text version of a database's schema. To effectively version control these scripts (git) and to compare them (WinMerge), it is imperative that the output from compatible databases is the same, and the differences highlighted are genuine database differences.
Column order does matter; but just to some people, not to everyone!
Use
SELECT * FROM TABLE1
which displays the default column order of the table.
If you want to change the order of the columns.
Specify the column name to display correspondingly
SELECT COLUMN1, COLUMN5, COLUMN4, COLUMN3, COULMN2 FROM TABLE1
you can use indexing.. After indexing, if select * from XXXX results should be as per the index, But only result set.. not structrue of Table
In order to have a specific column order You need to select column by column in the order You wish.
Selection order dictates how columns will be ordered in output.
Try this command:
alter table students modify age int(5) first;
This will change the position of age to the first position.
You can change this using SQL query. Here is sql query to change the sequence of column.
ALTER TABLE table name
CHANGE COLUMN `column1` `column1` INT(11) NOT NULL COMMENT '' AFTER `column2`;
alter table name modify columnname int(5) first;
will bring the column to first
alter table name modify columnname int(5) after (tablename);
This worked for me on Oracle DB:
select column1, column2, t.* from table t
Example: Change position of field_priority after field_price in table status.
ALTER TABLE `status` CHANGE `priority` `priority` INT(11) NULL DEFAULT NULL AFTER `price`;