I have a table structure I want to partition by timestamp, in ranges of day.
It consists of two tables which has common fields, to make it simple I will use an example of tables A,B as follows:
create table A (
id serial not null,
ts timestamp without time zone not null,
name text,
some_custom_A_field_1 numeric,
some_custom_A_field_2 numeric
);
create table B (
id serial not null,
ts timestamp without time zone not null,
name text,
some_custom_B_field_1 numeric,
some_custom_B_field_2 numeric,
some_custom_B_field_3 numeric,
some_custom_B_field_4 numeric
);
So I have two options, and I'm not sure which one is preferred.
Option #1 - create base table, inherit, partition childs:
create table Base (
id serial not null,
ts timestamp without time zone not null,
name text
)
create table ABase (
some_custom_A_field_1 numeric,
some_custom_A_field_2 numeric
)INHERITS (Base );
create table BBase (
some_custom_B_field_1 numeric,
some_custom_B_field_2 numeric,
some_custom_B_field_3 numeric,
some_custom_B_field_4 numeric
)INHERITS (Base );
CREATE TABLE a_2018_06_24 (CHECK (ts >= '2018-06-24 00:00:00' AND ts < '2018-06-25 00:00:00')) INHERITS (ABase);
CREATE TABLE b_2018_06_24 (CHECK (ts >= '2018-06-24 00:00:00' AND ts < '2018-06-25 00:00:00')) INHERITS (BBase);
Option #2 - two base tables partitioned:
create table ABase (
id serial not null,
ts timestamp without time zone not null,
name text,
some_custom_A_field_1 numeric,
some_custom_A_field_2 numeric
);
create table BBase (
id serial not null,
ts timestamp without time zone not null,
name text,
some_custom_B_field_1 numeric,
some_custom_B_field_2 numeric,
some_custom_B_field_3 numeric,
some_custom_B_field_4 numeric
);
CREATE TABLE a_2018_06_24 (CHECK (ts >= '2018-06-24 00:00:00' AND ts < '2018-06-25 00:00:00')) INHERITS (ABase);
CREATE TABLE b_2018_06_24 (CHECK (ts >= '2018-06-24 00:00:00' AND ts < '2018-06-25 00:00:00')) INHERITS (BBase);
Additional info:
I'm using version 9.6
Table A approx 2M records per day , Table B approx 120M records per day.
Use cases - ingest data to tables once a day, run select queries (based on time-series).
What will be the better practice, pros / cons?
Thanks in advance.
Related
I have the following table:
CREATE TABLE duration
(
departure_time TIMESTAMPTZ,
arrival_time TIMESTAMPTZ,
duration TIME NOT NULL, -- Not sure about the datatype..
flight_id INT UNIQUE NOT NULL,
CHECK (scheduled_duration > 0),
CHECK (scheduled_arrival_time > scheduled_departure_time),
FOREIGN KEY (flight_id) REFERENCES flight(flight_id),
PRIMARY KEY (scheduled_departure_time, scheduled_arrival_time)
);
I want to calculate arrival_time - departure_time and then insert the result into the column duration. Preferably, the result of the duration subtraction would be 6h 30m. I am new to databases and PostgreSQL and I can't find a way to calculate a subtraction of two timestamps, taking into consideration their timezones at the same time.
Use a generated column
CREATE TABLE duration
(
departure_time TIME WITH TIME ZONE,
arrival_time TIME WITH TIME ZONE,
scheduled_duration INT,
flight_id INT,
duration2 TIME GENERATED ALWAYS AS ("arrival_time"::time - "departure_time"::time) STORED,
CHECK (scheduled_duration > 0),
CHECK (arrival_time > departure_time),
FOREIGN KEY (flight_id) REFERENCES flight(flight_id),
PRIMARY KEY (departure_time, arrival_time)
);
SELECT
EXTRACT(EPOCH FROM '2022-07-07 15:00:00.00000'::TIMESTAMP - '2022-07-07 15:00:00.00000'::TIMESTAMP)
how to create date format yyyy-mm with postgresql11
CREATE TABLE public."ASSOL"
(
id integer NOT NULL,
"ind" character(50) ,
"s_R" character(50) ,
"R" character(50) ,
"th" character(50),
"C_O" character(50) ,
"ASSOL" numeric(11,3),
date date,
CONSTRAINT "ASSOL_pkey" PRIMARY KEY (id)
This is a variation of Kaushik's answer.
You should just use the date data type. There is no need to create another type for this. However, I would implement this use a check constraint:
CREATE TABLE public.ASSOL (
id serial primary key,
ind varchar(50) ,
s_R varchar(50) ,
R varchar(50) ,
th varchar(50),
C_O varchar(50) ,
ASSOL numeric(11,3),
yyyymm date,
constraint chk_assol_date check (date = date_trunc('month', date))
);
This only allows you to insert values that are the first day of the month. Other inserts will fail.
Additional notes:
Don't use double quotes when creating tables. You then have to refer to the columns/tables using double quotes, which just clutters queries. Your identifiers should be case-insensitive.
An integer primary key would normally be a serial column.
NOT NULL is redundant for a PRIMARY KEY column.
Use reasonable names for columns. If you want a column to represent a month, then yyyymm is more informative than date.
Postgres stores varchar() and char() in the same way, but for most databases, varchar() is preferred because trailing spaces actually occupy bytes on the data pages.
for year and month you can try like below
SELECT to_char(now(),'YYYY-MM') as year_month
year_month
2019-05
You cannot create a date datatype that stores only the year and month component. There's no such option available at the data type level.
If you want to to truncate the day component to default it to start of month, you may do it. This is as good as having only the month and year component as all the dates will have day = 1 and only the month and year would change as per the time of running insert.
For Eg:
create table t ( id int, col1 text,
"date" date default date_trunc('month',current_date) );
insert into t(id,col1) values ( 1, 'TEXT1');
select * from t
d col1 date
1 TEXT1 2019-05-01
If you do not want to store a default date, simply use the date_trunc('month,date) expression wherever needed, it could either be in group by or in a select query.
Is there a way to create a table with a timestamp column defaulting to now?
Any of the following succeeds on creation but fails on insertion.
CREATE TABLE MyTable
(
device_id VARCHAR(64) NOT NULL encode lzo,
app_id VARCHAR(150) NOT NULL distkey encode lzo,
retrieval_date TIMESTAMP without time zone default (now() at time zone 'utc') encode lzo
)
diststyle key
compound sortkey(app_id,device_id);
Then on insert:
insert into MyTable (device_id,app_id) values('a','b');
INFO: Function "timezone(text,timestamp with time zone)" not supported.
INFO: Function "now()" not supported.
ERROR: Specified types or functions (one per INFO message) not supported on Redshift tables.
I tried other flavors as below but all fail.
a) Tried with now in single quotes 'now' , create succeeded but failed with another error
b) Tried without the timezone, create succeeded, insert failed.
You can use SYSDATE or GETDATE() to put a current timestamp value. Here is an example.
dev=> create table my_table (id int, created_at datetime default sysdate);
CREATE TABLE
dev=> insert into my_table (id) values (1);
INSERT 0 1
dev=> select * from my_table;
id | created_at
----+---------------------------
1 | 2016-01-04 19:07:14.18745
(1 row)
SYSDATE (Oracle Compatibility Function)
http://docs.aws.amazon.com/redshift/latest/dg/r_SYSDATE.html
GETDATE()
http://docs.aws.amazon.com/redshift/latest/dg/r_GETDATE.html
I am making a database with postgresql 9.1
Given tables:
CREATE TABLE rooms(
room_number int,
property_id int,
type character varying,
PRIMARY KEY (room_number, property_id)
);
Insert into rooms values (1,1,double),(2,1,double),(3,1,triple)
CREATE TABLE reservations(
reservation_ID int,
property_id int,
arrival date,
departure date,
room_num int,
PRIMARY KEY(reservation_ID,property_id)
FOREIGN KEY (room_number, property_id)
);
INSERT INTO orders VALUES (1,1,2013-9-27,2013-9-30,1),
(2,1,2013-9-27,2013-9-28,2),
(3,1,2013-9-29,2013-9-30,3);
I want to give 2 dates and check availability in between. So at the 1st column should apear:
all the dates between the given and
additional one column for every type of the room displaying the availability.
So my result, given 2013-9-27 & 2013-9-30 as input, must be sth like this:
I think the best solution would be use both generate_series() and crosstab() to create a dynamic table. Moreover you can use a left join from a CTE to your data tables so you get better information. Something like:
WITH daterange as (
SELECT s::date as day FROM generate_series(?, ?, '1 day')
)
SELECT dr.day, sum(case when r.type = 'double' then r.qty else 0) as room_double,
sum(case when r.type = 'triple' then r.qty else 0) as room_triple....
);
But note that crosstab would make the second query a little easier.
I have a table like this:
date_start date_end account_id product_id
2001-01-01 2001-01-31 1 1
2001-02-01 2001-02-20 1 1
2001-04-01 2001-05-20 1 1
I want to disallow overlapping intervals a given (account_id, product_id)
EDIT: I found something:
CREATE TABLE test (
from_ts TIMESTAMPTZ,
to_ts TIMESTAMPTZ,
account_id INTEGER,
product_id INTEGER,
CHECK ( from_ts < to_ts ),
CONSTRAINT overlapping_times EXCLUDE USING GIST (
account_id WITH =,
product_id WITH =,
box(
point( extract(epoch FROM from_ts at time zone 'UTC'), extract(epoch FROM from_ts at time zone 'UTC') ),
point( extract(epoch FROM to_ts at time zone 'UTC') , extract(epoch FROM to_ts at time zone 'UTC') )
) WITH &&
)
);
If you want to know more about this http://www.depesz.com/2010/01/03/waiting-for-8-5-exclusion-constraints/
My only problem is that it doesn't work with null values as a ending timestamp, I thought of replace it with infinite values but does not work as well.
Ok i ended up doing this :
CREATE TABLE test (
from_ts TIMESTAMPTZ,
to_ts TIMESTAMPTZ,
account_id INTEGER DEFAULT 1,
product_id INTEGER DEFAULT 1,
CHECK ( from_ts < to_ts ),
CONSTRAINT overlapping_times EXCLUDE USING GIST (
account_id WITH =,
product_id WITH =,
period(from_ts, CASE WHEN to_ts IS NULL THEN 'infinity' ELSE to_ts END) WITH &&
)
);
Works perfectly with infinity, transaction proof.
I just had to install temporal extension which is going to be native in postgres 9.2 and btree_gist available as an extension in 9.1 CREATE EXTENSION btree_gist;
nb : if you don't have null timestamp there is no need to use the temporal extension you could go with the box method as specified in my question.
In up to date postgres versions (I tested it in 9.6 but I assume it's working in >=9.2) you can use the build in function tstzrange() as mentioned in some other comments. Null values will be treated as positive or negative infinity by default and the CHECK contraint is then not explicitly needed anymore (if you are fine that the check is only <= and a range can start and end with the same date). Only the extension btree_gist is still needed:
CREATE EXTENSION btree_gist;
CREATE TABLE test (
from_ts TIMESTAMPTZ,
to_ts TIMESTAMPTZ,
account_id INTEGER DEFAULT 1,
product_id INTEGER DEFAULT 1,
CONSTRAINT overlapping_times EXCLUDE USING GIST (
account_id WITH =,
product_id WITH =,
TSTZRANGE(from_ts, to_ts) WITH &&
)
);
This is a difficult problem because constraints can only reference the "current row", and may not contain subqueries. (otherwise the trivial solution would be to add some NOT EXISTS() subquery in the check)
A check constraint specified as a column constraint should reference that column's value only, while an expression appearing in a table constraint can reference multiple columns.
Currently, CHECK expressions cannot contain subqueries nor refer to variables other than columns of the current row.
Popular work-arounds are: use a trigger function which does the dirty work (or use the rule system, which is deprecated by most people)
Because most people favor triggers, I'll repost a rule-system hack here... (it does not have the extra "id" key element, but that's a minor detail)
-- Implementation of A CONSTRAINT on non-overlapping datetime ranges
-- , using the Postgres rulesystem.
-- We need a shadow-table for the ranges only to avoid recursion in the rulesystem.
-- This shadow table has a canary variable with a CONSTRAINT (value=0) on it
-- , and on changes to the basetable (that overlap with an existing interval)
-- an attempt is made to modify this variable. (which of course fails)
-- CREATE SCHEMA tmp;
DROP table tmp.dates_shadow CASCADE;
CREATE table tmp.dates_shadow
( time_begin timestamp with time zone
, time_end timestamp with time zone
, overlap_canary INTEGER NOT NULL DEFAULT '0' CHECK (overlap_canary=0)
)
;
ALTER table tmp.dates_shadow
ADD PRIMARY KEY (time_begin,time_end)
;
DROP table tmp.dates CASCADE;
CREATE table tmp.dates
( time_begin timestamp with time zone
, time_end timestamp with time zone
, payload varchar
)
;
ALTER table tmp.dates
ADD PRIMARY KEY (time_begin,time_end)
;
CREATE RULE dates_i AS
ON INSERT TO tmp.dates
DO ALSO (
-- verify shadow
UPDATE tmp.dates_shadow ds
SET overlap_canary= 1
WHERE (ds.time_begin, ds.time_end) OVERLAPS ( NEW.time_begin, NEW.time_end)
;
-- insert shadow
INSERT INTO tmp.dates_shadow (time_begin,time_end)
VALUES (NEW.time_begin, NEW.time_end)
;
);
CREATE RULE dates_d AS
ON DELETE TO tmp.dates
DO ALSO (
DELETE FROM tmp.dates_shadow ds
WHERE ds.time_begin = OLD.time_begin
AND ds.time_end = OLD.time_end
;
);
CREATE RULE dates_u AS
ON UPDATE TO tmp.dates
WHERE NEW.time_begin <> OLD.time_begin
AND NEW.time_end <> OLD.time_end
DO ALSO (
-- delete shadow
DELETE FROM tmp.dates_shadow ds
WHERE ds.time_begin = OLD.time_begin
AND ds.time_end = OLD.time_end
;
-- verify shadow
UPDATE tmp.dates_shadow ds
SET overlap_canary= 1
WHERE (ds.time_begin, ds.time_end) OVERLAPS ( NEW.time_begin, NEW.time_end)
;
-- insert shadow
INSERT INTO tmp.dates_shadow (time_begin,time_end)
VALUES (NEW.time_begin, NEW.time_end)
;
);
INSERT INTO tmp.dates(time_begin,time_end) VALUES
('2011-09-01', '2011-09-10')
, ('2011-09-10', '2011-09-20')
, ('2011-09-20', '2011-09-30')
;
SELECT * FROM tmp.dates;
EXPLAIN ANALYZE
INSERT INTO tmp.dates(time_begin,time_end) VALUES ('2011-09-30', '2011-10-04')
;
INSERT INTO tmp.dates(time_begin,time_end) VALUES ('2011-09-02', '2011-09-04')
;
SELECT * FROM tmp.dates;
SELECT * FROM tmp.dates_shadow;
How to create unique constraint to a group of columns :
CREATE TABLE table (
date_start date,
date_end date,
account_id integer,
UNIQUE (account_id , date_start ,date_end) );
in your case you will need to ALTER TABLE if the table already exists, check the documentation it would be helpful for you :
- DDL Constraints
- ALTER Table